# TranSmart: A Practical Interactive Machine Translation System

Guoping Huang, Lemao Liu, Xing Wang, Longyue Wang,
Huayang Li, Zhaopeng Tu, Chengyan Huang and Shuming Shi

Tencent AI Lab
<EMAIL_ADDRESS>
###### Abstract

Automatic machine translation is super efficient to produce translations yet
their quality is not guaranteed. This technique report introduces TranSmart, a
practical human-machine interactive translation system that is able to trade
off translation quality and efficiency. Compared to existing publicly
available interactive translation systems, TranSmart supports three key
features, word-level autocompletion, sentence-level autocompletion and
translation memory. By word-level and sentence-level autocompletion, TranSmart
allows users to interactively translate words in their own manners rather than
the strict manner from left to right. In addition, TranSmart has the potential
to avoid similar translation mistakes by using translated sentences in history
as its memory. This report presents major functions of TranSmart, algorithms
for achieving these functions, how to use the TranSmart APIs, and evaluation
results of some key functions. TranSmart is publicly available at its homepage
111 https://transmart.qq.com.

_K_ eywords Neural Machine Translation $\cdot$ Interactive Machine Translation
$\cdot$ Autocompletion $\cdot$ Constrained Decoding $\cdot$ Translation Memory

##  1 Introduction

Recent years have witnessed a breakthrough in automatic machine translation,
thanks to the advances in neural machine translation (NMT) [1, 2]. The key
idea in NMT is an encoder-decoder framework where a source sentence is
represented by an encoder network and a target sentence is generate by an
decoder network [3, 4]. By using a large-scale bilingual corpus, both encoder
and decoder networks consisting of massive parameters can be trained
sufficiently enough to yield excellent generalization ability on unseen
sentences. As a result, NMT delivers state of the art performance in many
machine translation benchmarks [5, 6, 7].

Figure 1: Autocompletion comparison between conventional IMT (a & c) and
TranSmart (b & d). For conventional IMT, autocompletion at sentence level (a)
or word level (c) performs with the left-to-right direction; while TranSmart
enables any directions as users enjoy for both sentence level (b) and word
level autocompletion (d). Red words denotes those validated by users while
gray words are automatically completed.

To the date, however, even a state-of-the-art NMT system is incapable of
meeting the strict requirements in some translation scenarios, where human
translation is still intensively involved even if it is inefficient and costly
[8, 9, 10]. Therefore, interactive machine translation (IMT) [11, 12], which
is believed to better trade off translation quality and efficiency, has drawn
increasing attention [13, 14, 15]: At each time a user corrects a translation,
the machine automatically generates a new translation based on the human
corrections until the translation process is finished [16]. Compared to
automatic machine translation, an interactive machine translation system
includes more components and it is more complicated in essence. During the
past decade, there were a few IMT systems available, for instance, TransType
[17], CASMACAT [18], and LILT 222www.lilt.com. which are either on top of
statistical machine translation [19, 20, 21] or the advanced NMT. All these
systems share a common characteristic: they perform translation in an
incremental manner from left to right and naturally require human translators
to follow this strict manner, as shown in Figure 1 (a & c), no matter what
translation manner of translators is.

In this report, we describe a new human-machine interactive translation
system, TranSmart, which was first made publicly accessible in 2018. Compared
with other IMT systems, one advantage of TranSmart is its flexibility in the
sense that users can employ any translation manners to interact with the
machine rather than the incremental manner from left to right. Therefore, if
some users prefer to translate difficult words first and then easy words,
TranSmart is able to conduct interactive translation in their own manners, as
shown in Figure 1 (b & d). Moreover, TranSmart includes an NMT engine
augmented with translation memory, and it is very helpful to translate a
document where similar translation errors may occur for an automatic machine
translation engine. To the best of our knowledge, TranSmart is the first
interactive neural machine translation (INMT) system which makes use of
translation memory.

Specifically, TranSmart provides three key features as follows:

  * •

Word-level autocompletion: At word level, in order to input a correct word,
users do not need to type all characters of this word from scratch but a few
characters instead, and then the system can automatically complete this word.
Unlike other INMT systems, the corrected word is not necessary to be adjacent
to the translation prefix.

  * •

Sentence-level autocompletion: At sentence level, users do not need to
translate all words from scratch but provide some of them (for instance, some
difficult words), and then the system can automatically complete the
translation based on user provided words. Different from other INMT systems,
user provided words can be discontinuous.

  * •

Translation-memory-augmented NMT: The system is able to re-use translation
results from users through translation memory and offers better translation.
As there may be several similar sentences in a document to be translated, the
system can efficiently avoid the occurrence of similar translation errors
thanks to the knowledge from translation memory.

By repeatedly leveraging these key features, TranSmart is able to
interactively collaborate human and a machine to generate a high-quality
translation in an efficient way. In addition, TranSmart provides some extended
features, including terminology translation, bilingual sentence examples,
document translation, tag-preserving translation, and image translation. In
the remaining part of this report, we first present our implementation of the
key modules of TranSmart. Then the TranSmart API is briefly introduced.
Finally, the effectiveness of some functions is demonstrated through empirical
experiments.

##  2 System Features

At high level, TranSmart contains three key features as well as several
extended features as shown in the left side of Figure 2. This section
describes the basic ideas of these features, whose detailed implementations
will be given in the next section.

Figure 2: Overview of the TranSmart system.

###  2.1 Key Features

#### Word Level Autocompletion

Given a source sentence, a translation context consisting of translation
pieces, and a human-typed character sequence, word level autocompletion aims
to predict a target word which is compatible with the typed sequence. With the
help of this feature, if a user is expected to type a target word to correct a
translation, it is not necessary to manually type all the characters but only
some of them. This feature is inspired from recent advances [22] in input
methods from monolingual scenario which are designed to improve the efficiency
of human input.

#### Sentence Level Autocompletion

Given a source sentence and a translation context, sentence-level
autocompletion aims to generate a complete translation for the source sentence
on top of the context. The translation context can be human typed words (with
or without word level autocompletion), or human-edited translation pieces from
a translation generated by the system. With this feature, a user does not need
to manually translate all words from scratch but some of them which are
usually difficult to the system, and then the system tries to complete the
translation by generating the remaining words automatically.

#### Memory-Aware Machine Translation

Memory-Aware Machine Translation aims to generate high-quality translation by
making use of translation memory. Users can provide their own domain-specific
translation data to our system as the memory or we can use our bilingual
training corpus as the memory. In addition, after users finish translating a
sentence, we have a mechanism to accumulate their translation history into the
memory. In this way, our system has the ability to avoid the same translation
errors occurring multiple times, and this feature is useful in translating a
document where there are similar sentences.

The above three key features can be used as atomic operations, which can be
applied multiple times to interact between the user and machine for a
translation task. For example, by repeatedly leveraging word level
autocompletion and sentence level autocompletion, TranSmart can generate a
complete translation for a source sentence with high-quality, by using the
memory-aware translation engine where translation memory is accumulated from
translation history of users. It is worth noting that the translation pieces
in the translation context may be discontinuous, and thus our word level and
sentence level autocompletion is more general and flexible than those in
existing INMT systems.

###  2.2 Extended Features

#### Document Translation

This feature is used to translate a formatted document in a source language to
a corresponding formatted document in a target language. It supports many
popular formats including TXT, HTML, XML, MARKDOWN, PDF, DOCX, PPTX and XLSX.
To this end, it generally performs two steps as follows. First, it parses the
input formatted document into a text document consisting of sentences with
several tags. Each tag may indicate some structural information such as a
paragraph or a font. For example, a sentence in the formatted document may be
”<style text-fill="red">Forrest Gump</style> is a 1994 American drama film”,
where ”<style text-fill="red">Forrest Gump</style>” indicates the phrase
Forrest Gump is with red color as its format. Second, it translates each
tagged sentence by the automatic translation engine with a specially designed
technique, which is named tag translation. Tag translation is crucial to
document translation and we will present its challenges and our solution in
the next section.

#### Image Translation

This feature aims to translate text contained in an image file in a source
language to a text document in a target language. It supports many popular
image formats such as JPG and PDF. Generally, it is implemented by a pipeline
procedure consisting of two steps as follows. First, it employs an external
OCR toolkit to extract a text document where any content beyond text is
ignored. Second, it uses our translation engine to translate all sentences in
the text document one by one. The first step, which is called text extraction
from image, is critical to image translation and we will present its challenge
and the technique to address in next section.

#### Terminology Translation

We collect more than 3 million of Chinese-English terminologies from websites.
However, this corpus contains a large amount of noises, including non-
terminology, unaligned, inconsistent format. We filter non-terminology words
by considering the word frequency due to the long-tail property of
terminology. More specifically, we built a phrase table from a large-scale
parallel data and then extract high-frequency phrase pairs as a non-
terminology list. Finally, we filter noises by comparing the stop list and
collected data [23]. Furthermore, we employ our in-house filtering scripts
[24] to filter unaligned terms according to various features such as length
ratio and language identification. As a result, we obtained a clean version of
terminology corpus that contains around 2 million terms.

#### Bilingual Examples

The input sentence is used to retrieve bilingual examples from the
corresponding retrieval repository. We selected and used more than 200M
bilingual sentences to build the retrieval repository. The three most similar
bilingual examples are displayed to help users to translate the input
sentence.

##  3 Implemented Techniques

###  3.1 Generic Translation Model

#### Translation Model

We implemented the generic translation model on top of the Transformer
architecture [1]. To balance the translation performance and inference
efficiency, we used a 24-layer encoder and a 6-layer decoder, whose hidden
size is 1024. We trained the translation model on our in-house data after the
following data manipulation methods, which consists of 200 millions of
Chinese-English sentence pairs. We followed [25] to train models with batches
of approximately 460k tokens, using Adam [26] with $\beta_{1}=0.9$,
$\beta_{2}=0.98$ and $\epsilon=10^{-8}$.

Figure 3: The framework of data rejuvenation, which consists of two models.
The identification model identifies inactive examples from the original
training data, which is then rejuvenated by the rejuvenation model. The
rejuvenated examples along with the active examples are used together to train
the NMT model.

#### Data Rejuvenation

Large-scale parallel datasets lie at the core of the recent success of NMT
models. However, the complex patterns and potential noises in the large-scale
data make training NMT models difficult. We introduce _data rejuvenation_ to
improve the training of NMT models on large-scale datasets by exploiting
inactive examples [27]. The proposed framework consists of three phases, as
shown in Figure 3. First, we train an identification model on the original
training data to distinguish inactive examples and active examples by their
sentence-level output probabilities. Then, we train a rejuvenation model on
the active examples to re-label the inactive examples with forward-
translation. Finally, we combined the rejuvenated examples and the active
examples as the final bilingual data.

#### Data Augmentation

Although we have large-scale parallel data, there are limited amount parallel
data for some specific domains. Data augmentation methods (e.g. self-training
and back-translation) are a promising way to alleviate this problem by
augmenting model training with synthetic parallel data. The common practice is
to construct synthetic data based on a randomly sampled subset of large-scale
monolingual data, which we empirically show is sub-optimal. In response to
this problem, we improve the sampling procedure by selecting the most
informative monolingual sentences to complement the parallel data. To this
end, we compute the uncertainty of monolingual sentences using the bilingual
dictionary extracted from the parallel data. Intuitively, monolingual
sentences with lower uncertainty generally correspond to easy-to-translate
patterns which may not provide additional gains. Accordingly, we design an
uncertainty-based sampling strategy [28] to efficiently exploit the
monolingual data for self-training, in which monolingual sentences with higher
uncertainty would be sampled with higher probability.

###  3.2 General Word-level Autocompletion

Figure 4: The input representation of our model and architecture of
Bidirectional Masked Attention. The input embeddings are the sum of the token
embeddings and position embeddings. [MASK] represents the potenial target word
in this translation context.

#### Task Definition

Word-level autocompletion aims to complete the target word based on human
typed characters for a given a source sentence and translation context.
Previous studies have explored word-level autocompletion task, but they either
do not take into account translation context [29] or they require the target
word to be the next word of the translation prefix [17, 30], which limit its
applications in real-world scenarios such as post-editing [31]. To this end,
we propose a general word-level autocompletion task which can be applied to
more general scenarios.

Suppose $\boldsymbol{x}$=$(x_{1},x_{2},\dots,x_{m})$ is a source sequence,
$\boldsymbol{s}$=$(s_{1},s_{2},\dots,s_{k})$ is a sequence of human typed
characters, and translation context is denoted by
$\boldsymbol{c}$=$(\boldsymbol{c}_{l},\boldsymbol{c}_{r})$, where
$\boldsymbol{c}_{l}$=$(c_{l,1},c_{l,2},\dots,c_{l,i})$ and
$\boldsymbol{c}_{r}$=$(c_{r,1},c_{r,2},\dots,c_{r,j})$. The translation pieces
$\boldsymbol{c}_{l}$ and $\boldsymbol{c}_{r}$ are on the left and right hand
side of $\boldsymbol{s}$, respectively. Formally, given a source sequence
$\boldsymbol{x}$, typed character sequence $\boldsymbol{s}$ and a context
$\boldsymbol{c}$, the general word-level autocompletion (GWLAN) task aims to
predict a target word $w$ which is to be placed in the middle between
$\boldsymbol{c}_{l}$ and $\boldsymbol{c}_{r}$ to constitute a partial
translation. Note that $\boldsymbol{c}_{l}$ or $\boldsymbol{c}_{r}$ may be
empty in some scenarios.

#### Methodology

Given a tuple $(\boldsymbol{x},\boldsymbol{c},\boldsymbol{s})$, our approach
decomposes the whole word autocompletion process into two parts: model the
distribution of the target word $w$ based on the source sequence
$\boldsymbol{x}$ and the translation context $\boldsymbol{c}$, and find the
most possible word $w$ based on the distribution and human typed sequence
$\boldsymbol{s}$.

In the first part, we propose a word prediction model (WPM) to define the
distribution $p(w|\boldsymbol{x},\boldsymbol{c})$ of the target word $w$. We
use a single placeholder [MASK] to represent the unknown target word $w$, and
use the representation of [MASK] learned from WPM to predict it. Formally,
given the source sequence $\boldsymbol{x}$, and the translation context
$\boldsymbol{c}=(\boldsymbol{c}_{l}$, $\boldsymbol{c}_{r})$, the possibility
of the target word $w$ is:

$P(w|\boldsymbol{x},\boldsymbol{c}_{l},\boldsymbol{c}_{r};\theta)=\mathrm{softmax}\left(\phi(h)\right)[w]$
(1)

where $h$ is the hidden representation of decoding state with respect to
[MASK], $\phi$ is a linear network that projects the hidden representation $h$
to a vector with dimension of target vocabulary size, and
$\mathrm{softmax}(d)[w]$ takes the component regarding to $w$ after the
softmax operation. Our model has a source encoder and a cross-lingual encoder.
The source encoder of WPM is the same as the Transformer encoder, which is
used to encode the source sequence $\boldsymbol{x}$. The output of source
encoder is passed to the cross-lingual encoder later. The cross-lingual
encoder is similar to the Transformer decoder, while the only difference is
that we replace the auto-regressive attention layer by a bidirectional masked
attention (BMA) module, which is shown in Figure 4.

Suppose $\boldsymbol{s}$ denotes a human typed sequence of characters, in the
second part, we predict the best word according to the constrained
optimization:

${\operatorname*{argmax}}_{w\in\mathcal{V}(\boldsymbol{s})}P(w\mid\boldsymbol{x},\boldsymbol{c};\theta)$
(2)

where $\mathcal{V}(\boldsymbol{s})$ denotes a set of target words, whose
element satisfies the sequences of $\boldsymbol{s}$, for example,
$\boldsymbol{c}$ is a prefix of $w$. More details can be found in [32].

#### Data Generation

For training and evaluating GWLAN models above, firstly we should create a
large scale dataset including tuples of
$(\boldsymbol{x},\boldsymbol{s},\boldsymbol{c},w)$. Ideally, we may hire
professional translators to manually annotate such a dataset, but it is too
costly in practice. We instead propose to automatically construct the dataset
from parallel datasets.

Assume we are given a parallel dataset
$\\{(\boldsymbol{x}^{i},\boldsymbol{y}^{i})\\}$, where $\boldsymbol{y}^{i}$ is
the reference translation of $\boldsymbol{x}^{i}$. We automatically construct
the data $\boldsymbol{c}^{i}$ and $\boldsymbol{s}^{i}$ by randomly sampling
from $\boldsymbol{y}^{i}$. Specifically, we first sample a word
$w=\boldsymbol{y}^{i}_{k}$, and sample two spans $[a_{l},b_{l}]$ and
$[a_{r},b_{r}]$ such that $0\leq a_{l}\leq b_{l}\leq k$ and $k+1\leq a_{r}\leq
b_{r}\leq|\boldsymbol{y}^{i}|$, leading to $\boldsymbol{c}_{l}=\langle
y^{i}_{a_{l}},\cdots,y^{i}_{b_{l}-1}\rangle$ and $\boldsymbol{c}_{r}=\langle
y^{i}_{a_{r}},\cdots,y^{i}_{b_{r}-1}\rangle$. It is worth mentioning that
$\boldsymbol{c}_{l}$ or $\boldsymbol{c}_{r}$ may not be adjacent to
$y^{i}_{k}$. In addition, we randomly sample a character sequence as
$\boldsymbol{s}^{i}_{k}$ for $y^{i}_{k}$ as follows: for languages like
English and German, $\boldsymbol{s}^{i}_{k}$ is a character prefix of
$y^{i}_{k}$; for languages like Chinese, $\boldsymbol{s}^{i}_{k}$ consists of
the first phonetic symbol of each character in $y^{i}_{k}$. In this way, we
can obtain a collection of
$\\{(\boldsymbol{x}^{i},\boldsymbol{c}_{l},\boldsymbol{c}_{r},\boldsymbol{s}^{i}_{k},y^{i}_{k})\mid\forall
i,\forall k\\}$, which can be divided into training, validation and test
datasets for the GWLAN task.

###  3.3 Sentence-level Autocompletion by Lexical Constraints

In our human-machine interactive translation scenario, human translators may
pre-specify some constraint words (or lexical constraints) and our system
requires to output a high-quality translation with knowledge from these pre-
specified constraints. For example, the constraints can be typed by human
translators through word level autocompletion in Section 3.2, and they can be
a translation prefix corrected by translators, or a post-edited partial
translation. It is worth noting that the constraints are not necessary to be
continuous [33], unlike [34, 13]. In TranSmart, we implement two different
approaches to incorporate these constraints into NMT. The first one relies on
constrained decoding which requires the output translation to include
constraints in a hard manner; whereas the second one makes use of them in a
soft manner, i.e., the output translation may not include some of constraints.

#### Constrained Decoding

Constrained decoding is essentially a constrained optimization problem, and it
aims to search the translation which satisfies some constraints and is with
the best model score. Formally, suppose $P(\boldsymbol{y}\mid\boldsymbol{x})$
is a translation model, and $\boldsymbol{c}$ denotes a set of constraint
words.

$\operatorname*{argmax}_{\boldsymbol{y}\in\mathcal{Y}(\boldsymbol{c})}P(\boldsymbol{y}\mid\boldsymbol{x})$

where $\mathcal{Y}(\boldsymbol{c})$ denotes a set of translation hypotheses
which include all constraint words in $\boldsymbol{c}$. To address this
constrained optimization problem, [35] proposed a grid beam search algorithm
(GBS). This algorithm maintains a beam along two dimensions, where one is the
length of the hypotheses and the other is the number words in all constraints.
Thus, its complexity is linear to the number of the constraint words, which is
inefficient in our interactive scenario. [36] presented an improved algorithm
by dynamic beam allocation (DBA) which allows hypotheses in a beam to contain
different number of constraint words. In our experiments, this improved
algorithm is indeed more efficient in decoding speed but its translation
quality is worse than grid beam search. To this end, we propose a variant of
grid beam search algorithm to achieve a trade-off between quality and
efficiency.

In our interactive scenario, we observe that constraint words in
$\boldsymbol{c}$ exhibit two characteristics: some of constraints in
$\boldsymbol{c}$ are consecutive to form a translation piece, especially when
the size of $\boldsymbol{c}$ is very large; and $\boldsymbol{c}$ provided by
human translators is placed in a fixed order such that the output translation
should contain $\boldsymbol{c}$ in the same order. Based on the observation,
we organize a beam along the length of hypotheses and the number of constraint
pieces rather than constraint words. Thanks to the fixed order of pieces, any
translations with the same number of pieces will naturally contain the same
number of constraint words and thereby they can be fairly pruned in terms of
model scores, similar to the grid beam search algorithm. Consequently, the
complexity of this algorithm is linear to the number of pieces rather than the
number of constraints, which leads to substantial speedup in practice because
constraint pieces usually are long enough.

#### NMT with Soft Constraints

Unlike constrained decoding, we propose a new approach to make use of pre-
specified constraints in a soft manner. The key is to treat the constraints as
an external memory, integrate the memory into the standard Seq2Seq decoder [3,
4] and then train the memory-augmented NMT from a dataset such that it learns
to constrain the output. Specifically, given $\boldsymbol{x}$,
$\boldsymbol{c}$ and a translation prefix $\boldsymbol{y}_{<t}$, we generate a
target word $\boldsymbol{y}_{t}$ according to
$P\left({y}_{t}\mid\boldsymbol{x},\boldsymbol{y}_{<t},\boldsymbol{c};\theta\right)$.
Our model includes three components: the encoder of vanilla NMT model which
encodes $\boldsymbol{x}$ into a sequence of hidden vectors; the Constraint
Memory Encoder (CME) that encodes the constraints $\boldsymbol{c}$ into
another sequence of hidden vectors $E(\boldsymbol{c})$, i.e., the constraint
memory; and the Constraint Memory Integrator (CMI), which integrates the
constraint memories into the decoder network to generate the next token
$y_{t}$. To train the memory-augmented model, we create a dataset from
available bilingual corpus. We randomly sample some rare words from the
reference $\mathbf{r}$ as constraints, since rare words are more difficult to
translate than other words [14].

In inference, our decoding is an unconstrained optimization problem:

$\operatorname*{argmax}_{\boldsymbol{y}}P(\boldsymbol{y}\mid\boldsymbol{x},\boldsymbol{c};\theta)$

To approximately solve the above optimization problem, we employ the standard
beam search algorithm which is similar to the decoding in our baseline model.
Although we need additional overheads to handle the constraint memories, this
is negligible to the time consuming of the decoding in NMT. Thus, its search
is as efficient as the search algorithm of the standard NMT. In addition,
unlike constrained decoding, the method does not force a feasible
$\boldsymbol{y}$ to include all constraints in $\boldsymbol{c}$. Hence, if the
constraints in $\boldsymbol{c}$ include some noises (i.e., spelling mistakes),
it has the potential to avoid copying these noises. More details about this
function can be found in [37].

###  3.4 Graph based Translation Memory

The basic idea of translation memory based MT is to translate an input source
sentence by using the translations of the source sentences which are similar
to the input one, which is related to domain adaptation [38, 39, 40]. Suppose
we are given a translation memory (TM) for a source sentence which is a list
of bilingual sentence pairs. Generally, there are two ways to improve
translation models with translation memory: training model parameters with
augmented data (i.e., memory) [41, 42, 43] and summarizing knowledge from
translation memory to augment MT decoder [44, 45, 46]. For the latter idea, a
typical solution to represent a TM is to encode each word in both the source
and target sides by a neural memory[44]. Unfortunately, since a word even a
phrase may appear repeatedly in a TM, redundant words are encoded multiple
times, leading to a large memory network as well as considerable computation.
To address this issue, we propose an effective approach to representing a TM
with a compact graph structure, which is further used to enhance the
translation model. The model structure is illustrated in Figure 5 and its more
details can be found in [47].

#### Graph Representation of TM

It is observed that most source words in the TM also appear in the input
sentence and have already been represented by the encoder. In addition, we
believe that those words in the TM yet beyond the source side may not be
informative to translate the input sentence itself. Therefore, in our proposed
model, we directly ignore the source sentences of the TM and only represent
the target side. In addition, instead of sequentially encoding target
sentences in a TM, we pack them into a compact graph such that some words in
different sentences may correspond to the same node in the graph, which is
inspired by the notion of lattice or hypergraph in statistical machine
translation [48, 49]. To this end, we convert the target side in a TM into a
confusion network by using the algorithm proposed by [50] and [51].

Figure 5: The architecture of the proposed NMT with graph based TM. 1) Graph
representation - The part in the dashed box is a concrete example of the graph
representation of a TM. 2) Model architecture - The part outside of the dashed
box shows the core components of the model architecture.

#### NMT with Graph based TM

The graph based TM is further used to enhance the Transformer architecture.
Generally, the enhanced Transformer shares the similar architecture as
Transformer but with two major differences in encoding and decoding phases.

In the encoding phrase, besides encoding the input sequence, the proposed
model also encodes the graph by using $L$ layers of networks in a similar
fashion to the encoding of the input. Specifically, we firstly use a multi-
head attention to encode each node in the TM graph where the query is the
corresponding node, and key-value pairs are obtained from its first-order
neighborhood nodes inspired by graph attention [52]. Then we apply other sub-
layers to the resulting vector obtained by the multi-head attention layer,
which contain a residual layer, a feed-forward layer and a layer
normalization. In this way, we can represent the graph into a list of vectors
whose size is the same as the number of nodes in the graph.

In the decoding phase, similar to Transformer, the proposed model employs $L$
layers of networks but each layer includes sub-layers (i.e., multi-head
attention, residual layer and layer normalization) to incorporate the list of
vectors obtained from graph encoding, besides other six sub-layers. We place
the extra three sub-layers nearest to the output of the decoder network in
order to let the graph encoding fully influence the decoding process.

###  3.5 Others

#### Word Alignment

Word alignment plays an important role in document translation for TranSmart.
Since NMT is a blackbox model with massive parameters, previous work has made
numerous efforts to induce word alignment from attention in NMT [53, 54, 55]
or other explanation methods [56, 57, 58]. Other work improves alignment
quality by building a word alignment model whose architecture is similar to
NMT [59]. Despite its success, statistical aligners [60, 61] are still
respectful counterparts because of their training efficiency and alignment
quality. Therefore, we employ statistical aligners to obtain word alignment.
As TranSmart involves billion scale of bilingual sentences as its training
data, the popular aligner GIZA++ can not train successfully due to memory
consumption. Instead, we re-implement an aligner based on HMM [62] with an
adaptive strategy to prune the word translation table: for each high-frequency
word we allow more words to be its translations whereas we allow less for each
low-frequency word.

#### Tag Translation

Tag translation aims to translate a source language with tags into a target
language with tags, and it serves as the key step for document translation.
Since the standard translation engine are trained on top of bilingual
sentences without tags, standard translation engines can not perform well on
translating tagged sentences. In addition, there are no sufficient tagged
bilingual sentences to train a customized tag translation engine. Therefore,
we propose a simple post-processing approach based on word alignment as
follows. First, we delete all tags from a tagged source sentence to obtain a
tag-free sentence and then translate it into a target sentence by our default
translation engine. Then we run our word aligner on both the tag-free sentence
and its translation to obtain word level alignments. Second, we insert the
corresponding tags from the source sentence into its translation. However, the
second step is not straightforward because one tagged piece (or phrase) within
the tagged source sentence may align to multiple pieces in the target side due
to the essence of word alignment. We propose an algorithm based on dynamic
programming to extract a piece-to-piece alignment: the piece-to-piece
alignment is a one-to-one map between pieces in the source and target sides
such that it makes the minimal violations according to the word alignment
results. By using the piece-to-piece alignment, it is trivial to insert the
tags from the source sentence into its target sentence.

#### Text Extraction from Image

Text Extraction from Image aims to extract the text content from an image file
to form a text document in a source language while ignoring other content.
Generally, this task is challenging due to two reasons. First, a sentence in
an image file may not explicitly contain a special symbol to indicate its end
and several such sentences may actually constitute one sentence. More
importantly, OCR may recognize many blocks of text content from an image file
to some extent, but it is difficult to organize these text blocks into a text
document which preserves the same sequential structure of text blocks as in
the original image file. To tackle the first challenge, we develop a language
model to detect the end symbol for a sentence without an end symbol. In
addition, we design another model to detect whether several sentences without
an end symbol should be combined to one sentence. To address the second
challenge, we employ the position information of each block from the OCR
toolkit and uses it as a signal to decide the sequential order of all blocks,
which is critical to form the final text document.

#### Discourse-Aware Translation

Existing translation models usually translate a text by considering isolated
sentences based on a strict assumption that the sentences in a text are
independent of one another. However, disregarding dependencies across
sentences will harm translation quality especially in terms of coherence,
cohesion, and consistency [63]. To response this problem, we adapt document-
level NMT [64] to document-level training. Specifically, we add bilingual
documents and paragraphs to our training data, which can helps models to learn
discourse knowledge from larger contexts [65, 66]. This works well with the
Document Translation Feature as stated in Section 2.2. Chinese is a pro-drop
language, where pronouns are usually omitted when they can be inferable from
the context. This leads to serious problems for Chinese-to-English translation
models in terms of completeness and correctness. Thus, we recover missing
pronouns in the informal domain of training data (e.g. conversations and movie
subtitles) by leveraging our approaches [67, 68, 69].

##  4 System Usage

Two ways are available to use TranSmart. One way is to visit the website of
TranSmart333https://transmart.qq.com/index, which has a friendly interactive
user-interface (UI). Another way is calling the TranSmart HTTP APIs. The HTTP
APIs can be accessed by sending a JSON request to the
service444https://transmart.qq.com/api/imt via POST method. The major fields
of the JSON request of the translation APIs are shown in Table 1. In this
section, we will focus more on the latter way and introduce the usage of some
selected features.

Field | Field Type | Req/Opt |  Description
---|---|---|---
header | JSON | Required |  Information of API calls, including function name, token, and user name555The token and user name have to be registered in our account management system. Please consult with us for the details of registration process..
reference | JSON | Optional |  Information of references, which includes two fields “term_lib” and “sentence_lib”.
type | String | Optional |  The format of the given source sentence. The value should be one of plain, xml, html, and markdown. The default value is “plain”.
model_category | String | Optional |  The model served for translation. The value should be one of slow, normal, and fast. The default value is “normal”. Note that different models may have different translation speeds and qualities.
source | JSON | Required |  Information of the source text, including a “text_list” filed that contains a list of source sentences.
target | JSON | Required |  Information of the target sentence.
limit | JSON | Optional |  Control the returns in the JSON response, such as the number of suggestions.
reference_list | JSON | Optional |  Information of translation memory. The translation memory has two types: “term” and “sent”.
Table 1:  Major fields of the request JSON object when calling TranSmart API.
Req/Opt denotes the field is required or optional.

###  4.1 Word And Sentence Level Autocompletion

Figure 6: Example of the word-level dynamic suggestion. Figure 7: Example of
the sentence-level dynamic suggestion.

One important API is the “dynamic_suggestion” function, aiming to save the
effort of human translators when correcting the translation generated by AI.
This API is capable of providing autocompletion at both word-level and
sentence-level, which will be introduced one by one.

First, the word-level autocompletion is able to complete the unfinished word
input by human translators. As shown in Figure 6, when the a character
sequence “th” is tagged by “editing” in the “segment_list” field, TranSmart
will return its autocompletion in the “ime_suggestion” field in response.
Second, the sentence-level suggestion can complete the whole translation based
on the user-specified spans. As shown in Figure 7, the span with the “prefix”
type is promised to be the prefix of the re-generated translation in response,
and spans with “std” are forced to be included in re-generated translation.
The result of sentence-level autocompletion is in the “sentence_suggestion”
field in response.

###  4.2 Memory-Aware Machine Translation

Figure 8: Example of the automatic translation with translation memory.

As shown in Figure 8, when users provide the information of translation memory
in the “reference_list” field, TranSmart will be able to leverage those
existing and relevant translations in history to improve the translation
quality. The translation results are in the “auto_translation” field in
response. Note that users can also disable this feature by removing the
“reference_list” field, then TranSmart will translate only based on the source
text, i.e., the setting of traditional automatic translation.

###  4.3 Extended Features

Figure 9: Example of the static suggestion. Figure 10: Example of the
selection suggestion.

TranSmart also has two APIs that may be helpful for human translators. First,
it provides a static_suggestion function to retrieve relevant information from
our pre-constructed terminology and bilingual example databases. As shown in
Figure 9, the “static_suggestion” function returns the most relevant
terminology translations and bilingual examples in the “term_list” and
“sentence_example_list” field in response, respectively. Second, our
“selection_suggestion” is designed to translate specified spans in a source
sentence. As shown in Figure 10, only spans with “selection” status in the
“segment_list” field will be translated. The results of selection suggestions
are placed in the “segment_suggestion” field in response.

##  5 System Evaluation

###  5.1 Generic Translation

#### Datasets

We conducted experiments on the widely used WMT14 English$\Rightarrow$German
(En$\Rightarrow$De) and English$\Rightarrow$French (En$\Rightarrow$Fr)
datasets, which consist of about $4.5$M and $35.5$M sentence pairs,
respectively. 666Note that although the datasets used in the experiments are
preprocessed by the standard toolkit in Moses [70] (for a fair comparison
between previous methods), in implementing the online system, we adopt
TexSmart [71] in data preprocessing (such as Chinese word segmentation) and
postprocessing (e.g., restoring case information). We applied BPE [72] with
32K merge operations for both language pairs. The experimental results were
reported in case-sensitive BLEU score [73].

#### Systems

We validated our approach on a couple of representative NMT architectures:

  * •

Lstm [74] that is implemented in the Transformer framework.

  * •

Transformer [75] that is based solely on attention mechanisms.

  * •

DynamicConv [76] that is implemented with lightweight and dynamic
convolutions, which can perform competitively to the best reported Transformer
results.

We adopted the open-source toolkit Fairseq [77] to implement the above NMT
models. We followed the settings in the original works to train the models. In
brief, we trained the Lstm model for 100K steps with 32K ($4096\times 8$)
tokens per batch. For Transformer, we trained 100K and 300K steps with 32K
tokens per batch for the Base and Big models respectively. We trained the
DynamicConv model for 30K steps with 459K ($3584\times 128$) tokens per batch.
We selected the model with the best perplexity on the validation set as the
final model.

System | Architecture | WMT14 En$\Rightarrow$De | WMT14 En$\Rightarrow$Fr
---|---|---|---
BLEU | $\bigtriangleup$ | BLEU | $\bigtriangleup$
Existing NMT Systems
[74] | Lstm | 26.7 | – | – | –
[75] | Transformer-Base | 27.3 | – | 38.1 | –
Transformer-Big | 28.4 | – | 41.0 | –
[78] |  \+ Large Batch | 29.3 | – | 43.2 | –
[76] | DynamicConv | 29.7 | – | 43.2 | –
Our NMT Systems
This work | Lstm | 26.5 | – | 40.6 | –
\+ Data Rejuvenation | 27.0↑ | +0.5 | 41.1↑ | +0.5
Transformer-Base | 27.5 | – | 40.2 | –
\+ Data Rejuvenation | 28.3⇑ | +0.8 | 41.0⇑ | +0.8
2-6 | Transformer-Big | 28.4 | – | 42.4 | –
|  \+ Data Rejuvenation | 29.2⇑ | +0.8 | 43.0↑ | +0.6
2-6 |  \+ Large Batch | 29.6 | – | 43.5 | –
|  \+ Data Rejuvenation | 30.3⇑ | +0.7 | 44.0↑ | +0.5
| DynamicConv | 29.7 | – | 43.3 | –
|  \+ Data Rejuvenation | 30.2↑ | +0.5 | 43.9↑ | +0.6
Table 3: Evaluation of translation performance across model architectures and
language pairs. “$\uparrow/\Uparrow$”: indicate statistically significant
improvement over the corresponding baseline $p<0.05/0.01$ respectively.

#### Results

Table 3 lists the results across model architectures and language pairs. Our
Transformer models achieve better results than that reported in previous work
[75], especially on the large-scale En$\Rightarrow$Fr dataset (e.g., more than
1.0 BLEU points). [78] showed that models of larger capacity benefit from
training with large batches. Analogous to DynamicConv, we trained another
Transformer-Big model with 459K tokens per batch (“+ Large Batch” in Table 3)
as a strong baseline. We tested statistical significance with paired bootstrap
resampling [79] using compare-mt777https://github.com/neulab/compare-mt [80].

Clearly, our data rejuvenation consistently and significantly improves
translation performance in all cases, demonstrating the effectiveness and
universality of the proposed data rejuvenation approach. It’s worth noting
that our approach achieves significant improvements without introducing any
additional data and model modification. It makes the approach robustly
applicable to most existing NMT systems.

###  5.2 Word Level Autocompletion

#### Datasets

We carry out experiments on four GWLAN tasks including bidirectional
Chinese–English tasks and German–English tasks. The training set for two
directional Chinese–English tasks consists of 1.25M bilingual sentence pairs
from LDC corpora. As discussed in §3.2, the training data for GWLAN is
extracted from 1.25M sentence pairs. The validation data for GWLAN is
extracted from NIST02 and the test datasets for GWLAN are constructed from
NIST05 and NIST06. For two German–English tasks, we use the WMT14 dataset
standard preprocessed by Stanford 888https://nlp.stanford.edu/projects/nmt/.
The validation and test sets for our tasks are based on newstest13 and
newstest14 respectively. All the sample operations in the data construction
process are based on the uniform distribution. For each dataset, the models
are tuned and selected based on the validation set.

#### Systems

In the experiments, we evaluate and compare the performance of our proposed
approach (WPM) and a few baselines, which are illustrated below:

  * •

TransTable: We train an alignment model 999https://github.com/clab/fast_align
on the training set and build a word-level translation table. While testing,
we can find the translations of all source words based on this table, and
select out valid translations based on the human input. The word with highest
frequency among all candidates is regarded as the prediction. This baseline is
inspired by [29, 81].

  * •

Trans-PE: We train a vanilla NMT model using the Transformer-base model.
During the inference process, we use the context on the left hand side of
human input as the model input, and return the most possible words based on
the probability of valid words selected out by the human input. This baseline
is inspired by [17, 30].

  * •

Trans-NPE: As another baseline, we also train an NMT model based on
Transformer, but without position encoding on the target side. While testing,
we use the averaged hidden vectors of all the target words outputted by the
last decoder layer to predict the potential candidates.

#### Evaluation Metric

To evaluate the performance of the well-trained models, we choose accuracy as
the evaluation metric:

$\mathrm{Acc}=\frac{N_{match}}{N_{all}},$

where $N_{match}$ is the number of words that are correctly predicted and
$N_{all}$ is the number of testing examples.

Systems | Zh$\Rightarrow$En | En$\Rightarrow$Zh | De$\Rightarrow$En | En$\Rightarrow$De
---|---|---|---|---
NIST05 | NIST06 | NIST05 | NIST06 | NT13 | NT14 | NT13 | NT14
TransTable | 41.40 | 39.78 | 28.00 | 26.99 | 37.43 | 36.64 | 32.99 | 31.12
Trans-PE | 34.51 | 35.50 | 32.23 | 34.88 | 34.45 | 33.02 | 31.51 | 30.65
Trans-NPE | 35.97 | 36.78 | 34.31 | 36.19 | 36.69 | 36.01 | 33.25 | 31.30
WPM | 55.54 | 55.85 | 53.64 | 54.25 | 57.84 | 56.75 | 56.91 | 52.68

Table 4:  The main results measured by word prediction precision for different
systems on Chinese-English and German-English datasets.

#### Results

Table 4 shows the main results of our method and three baselines on the test
sets of Chinese-English and German-English datasets. The method Trans-PE,
which assumes the human input is the next word of the given context, behaves
poorly under the more general setting. As the results of Trans-NPE show, when
we use the same model as Trans-PE and relax the constraint of position by
removing the position encoding, the accuracy of the model improves. One
interesting finding is that the TransTable method, which is only capable of
leveraging the zero-context, achieves good results on the Chinese-English task
when the target language is English. However, when the target language is
Chinese, the performance of TransTable drops significantly. It is clear from
the results that our method WPM significantly outperforms the three baseline
methods.

###  5.3 Sentence Level Autocompletion

#### Datasets

We conduct experiments on the Zh$\Rightarrow$En, Fr$\Rightarrow$En,
De$\Rightarrow$En, and translation tasks. The Zh$\Rightarrow$En bilingual
corpus includes news articles collected from several online news websites.
After the standard preprocessing procedure as in [70], we obtain about 2
million bilingual sentences in total. Then we randomly select 2000 sentences
as the development and test datasets, respectively, and leave other sentences
as the training dataset. The Fr$\Rightarrow$En bilingual corpus is from JRC-
Acquis datasets [82] and it is preprocessed following [44]. This dataset is a
collection of the parallel legislative text of European Union Law applicable
in the EU member states and thus it is a highly related corpus focusing on a
specific domain. We also evaluate our methods on the WMT 2018 English-German
news translation task, which is composed of Europarl and news commentary
data[83]. We use the WMT newstest2013 and newstest2014 as the development and
test set respectively. In addition, in our experiments, we use the subword
technique [72] to ensure that there are no unknown tokens in our input, even
if a constraint word is not included in the training set. Therefore, if a
human translator provides an unknown word as a constraint word, our system
tokenizes it into several BPE tokens which are considered as several
constraints accordingly. Since there are no real constraints available
provided by human for all these datasets, we simulate this effect by randomly
picking some words from the reference side as the constraints following [35,
84].

#### Systems

As presented in section 3.3, we implement two methods: constrained decoding
with ordered constraints denoted by O-Gbs and NMT with soft constraints
denoted by SC. We compare both of our methods against three baselines:

  * •

Transformer: It is the standard Transformer, which is an in-house
implementation using Pytorch following [85].

  * •

Gbs: It is the Grid Beam Search in [35] built on top of the in-house
Transformer.

  * •

Dba: It is the lexically constrained decoding with Dynamic Beam Allocation (an
improved approach for [35]) in [36] .

To ensure the translation quality, we set the default beam size for Gbs and
Dba as suggested by [35] and [36]. The hyper-parameters for all the systems
are following those of Transformer base model[1]. We train all the models with
Adam optimization algorithm [26] and tune the number of iteration based on the
performance of the development set.

Systems | Fr$\Rightarrow$En | De$\Rightarrow$En | Zh$\Rightarrow$En | Runtime
---|---|---|---|---
Transformer | 67.13 | 26.46 | 36.22 | 0.256
\+ Gbs | 70.99† | 30.67† | 42.17† | 5.692
\+ Dba | 67.35† | 29.04† | 41.23† | 1.531
\+ O-Gbs | 70.99† | 30.67† | 42.17† | 1.959
\+ SC | 71.10† | 30.25† | 42.96† | 0.262
Table 5:  BLEU and runtime comparisons on perfect constraints for the
Zh$\Rightarrow$En, Fr$\Rightarrow$En, and De$\Rightarrow$En tasks. The runtime
is measured by the time consumed by decoding one sentence on the
Zh$\Rightarrow$En task. $\dagger$ indicates the comparison over Transformer is
significant with p < 0.05.

#### Results

As shown in Table 5, although the Dba method would reduce the computational
overhead compared to Gbs, its translation quality slightly sacrifices
accordingly, which is similar to the finding in [36]. Thanks to the ordered
constraints, the O-Gbs is faster than Gbs in running speed and delivers the
same performance as Gbs. The advantage of these three lexically decoding
algorithm is that they do not need to retrain a translation model. Moreover,
SC performs better than Gbs on the Fr$\Rightarrow$En and Zh$\Rightarrow$En
tasks and outperforms Dba on all the three tasks, in terms of translation
quality. This observation indicates that our model SC is able to learn how to
integrate the lexical constraints correctly. Moreover, there is an additional
advantage that the decoding speed of our model is comparable with that of the
Transformer model and is much faster than lexically constrained decoding.

###  5.4 Translation Memory

Since there is no translation history provided by human translators, we simply
use the training set as the memory for simulation. For each sentence, we
retrieve 100 translation pairs from the training set by using Apache Lucene
[86]. We score the source side of each retrieved pair against the source
sentence with fuzzy matching score and select top $N=5$ translation sentence
pairs as a translation memory for the sentence to be translated, following
[44, 45, 46]. Sentences from the target side in the translation memory are
used to form a graph, with each word represented as a node and the connection
between adjacent words in a sentence represented as an undirected edge.

#### Datasets

Following the previous works incorporating TM into NMT models, we use the JRC-
Acquis corpus for training and evaluating our proposed model. The JRC-Acquis
corpus is a collection of parallel legislative text of European union Law
applicable in the EU member states. The highly related text in the corpus is
suitable for us to make evaluations. To fully explore the effectiveness of our
proposed model, we conduct translation experiments on three language pair
bidirectionally, namely, en-fr, en-es, and en-de. We manage to obtain
preprocessed datasets from [44]. For each language pair, we randomly select
3000 samples to form a development and a test set respectively. The rest of
the pairs are used as the training set. Sentences longer than 80 and 100 are
removed from the training and development/test set. The technique of Byte-pair
Encoding [72] is applied and the vocabulary size is set to be 20K for all the
experiments.

#### Systems

The proposed graph based TM model is built on transformer [1], and it is
denoted by G-TFM. We compare the proposed model against the following
baselines:101010We notice that there are some recent advances on NMT with
translation memory [87, 88, 89, 90] after the proposed G-TFM was implemented
in TranSmart. We consider to update our TM model in future.

  * •

TFM: It is a natural baseline as the proposed model is directly built upon the
Transformer architecture.

  * •

P-RNN: It is an in-house implementation of [45] on top of RNN-search.

  * •

P-TFM: It similar to P-RNN but it is on top of Transformer rather than RNN-
search as [45].

  * •

SEG-TFM: It implements the idea of [44] on top of Transformer. Due to the
architecture divergence between RNN-based NMT and Transformer, it only differs
from the RNN-based counterpart in that two quantities $c_{t}$ and $z_{t}$ in
[44] are replaced by the hidden units obtained from the multi-head attention
over the encoding units and the decoding hidden state units before the softmax
operator.

  * •

SEQ-TFM: It sequentially encodes all target sentences in a TM as one of the
baseline models. Specifically, each target sentence in TM goes through a
multi-head mechanism and an immediate residual connection plus layer
normalization in $l_{th}$ layer. The derived representations for these
sentences are then concatenated to form the representation of the translation
memory, which can be utilized flexibly in $l_{th}$ decoding layer.

For training all systems, we maintain the same hyper-parameters for fair
comparison. Besides, we adopt the same training algorithm to learn the models
as follows. We use a customized leaning rate decay paradigm following
Tensor2Tensor[91] package. The learning rate increases linearly on early
stages for a certain number of steps, known as warm-up steps, and decay
exponentially later on. We set the warm-up step to be 5 epochs and we early
stop the model after training 20 epochs, typically the time when the
development performance varies insignificantly. Furthermore, since there is a
hyperparameter in the system P-TFM of [45] which is sensitive to the specific
translation task, we tune it carefully on the development set for all
translation tasks. Its optimized value is 0.7 for es and de tasks while it is
0.8 for fr task. 111111We run all 6 tasks with hyperparameters among [0.5,
1.5] with scale of 0.1, and manually pick the optimized value according to its
performance on the development set.

#### Results

| RNN | P-RNN | TFM | P-TFM | SEG-TFM | SEQ-TFM | G-TFM
---|---|---|---|---|---|---|---
Dev | 57.74 | 60.87 | 62.78 | 63.97 | 63.16 | 64.81 | 66.37
Test | 58.06 | 61.52 | 62.68 | 64.30 | 62.94 | 65.16 | 66.21
Table 6: Translation accuracy in terms of BLEU on the es-en task.

Table 6 shows the experiment results of all the systems on the es-en task in
terms of BLEU. Several observations can be made from the results. First, the
baseline TFM achieves substantial gains over RNN and even outperforms P-RNN by
around 1 BLEU point on the test set. Compared with the strongest baseline
P-TFM, the proposed SEQ-TFM and G-TFM are able to obtain some gains up to 1.9
BLEU points on the test set. This result verifies that our compact
representation of TM is able to guide the decoding of the state-of-the-art
model.

| TFM | SEG-TFM | SEQ-TFM | G-TFM
---|---|---|---|---
Train (s) | 4579 | 44238 | 21920 | 8692
Test (s) | 0.20 | 2.68 | 1.25 | 0.36
Words (#) | 68.28 | 374.52 | 214.97 | 129.18
BLEU | 62.68 | 62.94 | 65.16 | 66.21
Table 7: Running time and memory. Training time reports the time in seconds
for training one epoch on average, and testing time reports the time in
seconds for translating one sentence on average. Words (#) denotes the number
of words encoded in the neural models on average.

Second, it is observed that SEG-TFM is only comparable to TFM on this task,
although its RNN based counterpart brought significant gains as reported in
[44]. This fact shows that the transformer architecture may need a
sophisticated way to well define a key-value memory for TM encoding, which can
be significantly different from that on RNN architecture. This is beyond the
scope of this paper. Fortunately, this paper provides an easy yet effective
approach to encode a TM, i.e. G-TFM, which does not rely on a context-based
key-value memory.

Since the retrieval time can be neglected compared with the decoding time as
found in [45], we thereby eliminate the retrieval time and directly compare
running time for neural models as shown in Table 7. From this table, we
observe that the proposed graph based model G-TFM saves significant running
time compared with SEG-TFM and SEQ-TFM while achieving better translation
performance.

Table 7 depicts the total number of source and target words encoded by the
corresponding model for each test sentence on average. It’s observed that SEG-
TFM needs to encode approximately 3 times and SEQ-TFM encodes approximately 2
times the number of words of our proposed model, G-TFM. There’s no wonder that
TFM takes the fewest words to encode because no extra TM is included. These
statistics indicate that under the scenario of incorporating TM in NMT, our
m1odel requires the least memory.

| BLEU | en-fr | fr-en | en-de | de-en | en-es
---|---|---|---|---|---|---
Dev | TFM | 66.33 | 65.95 | 53.32 | 58.54 | 60.43
| P-TFM | 68.90 | 68.61 | 55.54 | 60.10 | 61.50
| G-TFM | 69.69 | 70.65 | 57.43 | 61.85 | 62.50
Test | TFM | 66.36 | 66.96 | 53.29 | 58.86 | 60.52
| P-TFM | 68.73 | 68.70 | 55.14 | 60.26 | 61.56
| G-TFM | 69.59 | 70.87 | 56.88 | 61.72 | 62.76
Table 8: Translation Results on both development and test sets across other 5
translation tasks.

We pick stronger baselines from the es-en task, i.e. TFM and P-TFM, and
compare them with the proposed G-TFM model on other 5 translation tasks. Table
8 summarizes their results on both the development and test sets. From this
table, we can see that on the test set, G-TFM steadily outperforms TFM by up
to 3 BLEU points across all these 5 tasks, In addition, contrast to P-TFM,
G-TFM demonstrates better performance by exceeding at least 1 BLEU point
across all these tasks except the en-fr task. These results are consistent
with the results on es-en task and further validates the effectiveness of
integrating graph-based translation memory into the Transformer model.

##  6 Conclusion

In this technical report we have presented TranSmart, a practical interactive
machine translation (IMT) system. Unlike the conventional IMT systems with the
strict manner from left to right, TranSmart conducts interaction between a
user and the machine in a flexible manner, and it particularly contains a
translation memory technique to avoid similar mistakes occurring during
translation process. We have introduced the main functions of TranSmart and
key methods for implementing the features. Some instructions about how to use
the TranSmart through online APIs have been described. We have also reported
some evaluation results on major modules of TranSmart.

## References

  * [1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [2] Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  Convolutional sequence to sequence learning.  In International Conference on Machine Learning, pages 1243–1252. PMLR, 2017.
  * [3] Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  arXiv preprint arXiv:1409.3215, 2014.
  * [4] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [5] Bojar Ondrej, Rajen Chatterjee, Federmann Christian, Graham Yvette, Haddow Barry, Huck Matthias, Koehn Philipp, Liu Qun, Logacheva Varvara, Monz Christof, et al.  Findings of the 2017 conference on machine translation (wmt17).  In Second Conference onMachine Translation, pages 169–214. The Association for Computational Linguistics, 2017.
  * [6] Loïc Barrault, Ondřej Bojar, Marta R Costa-Jussa, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, et al.  Findings of the 2019 conference on machine translation (wmt19).  In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, 2019.
  * [7] Shuangzhi Wu, Xing Wang, Longyue Wang, Fangxu Liu, Jun Xie, Zhaopeng Tu, Shuming Shi, and Mu Li.  Tencent neural machine translation systems for the wmt20 news translation task.  In Proceedings of the Fifth Conference on Machine Translation, pages 313–319, 2020.
  * [8] Álvaro Peris, Miguel Domingo, and Francisco Casacuberta.  Interactive neural machine translation.  Computer Speech & Language, 45:201–220, 2017.
  * [9] Rongxiang Weng, Hao Zhou, Shujian Huang, Lei Li, Yifan Xia, and Jiajun Chen.  Correct-and-memorize: Learning to translate from interactive revisions.  arXiv preprint arXiv:1907.03468, 2019.
  * [10] Tianxiang Zhao, Lemao Liu, Guoping Huang, Zhaopeng Tu, Huayang Li, Yingling Liu, Guiquan Liu, and Shuming Shi.  Balancing quality and human involvement: An effective approach to interactive neural machine translation.  In AAAI, 2020.
  * [11] Mirko Plitt and François Masselot.  A productivity test of statistical machine translation post-editing in a typical localisation context.  The Prague bulletin of mathematical linguistics, 93(1):7–16, 2010\.
  * [12] Spence Green, Jason Chuang, Jeffrey Heer, and Christopher D Manning.  Predictive translation memory: A mixed-initiative system for human language translation.  In Proceedings of the 27th annual ACM symposium on User interface software and technology, pages 177–187, 2014.
  * [13] Rebecca Knowles and Philipp Koehn.  Neural interactive translation prediction.  In Proceedings of the Association for Machine Translation in the Americas, pages 107–120, 2016.
  * [14] David Grangier and Michael Auli.  QuickEdit: Editing text & translations by crossing words out.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 272–282, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [15] Qian Wang, Jiajun Zhang, Lemao Liu, Guoping Huang, and Chengqing Zong.  Touch editing: A flexible one-time interaction approach for translation.  In Proceedings of AACL-IJCNLP, 2020.
  * [16] George Foster, Pierre Isabelle, and Pierre Plamondon.  Target-text mediated interactive machine translation.  Machine Translation, 12(1):175–194, 1997.
  * [17] Philippe Langlais, George Foster, and Guy Lapalme.  Transtype: a computer-aided translation typing system.  In ANLP-NAACL 2000 Workshop: Embedded Machine Translation Systems, 2000.
  * [18] Vicent Alabau, Christian Buck, Michael Carl, Francisco Casacuberta, Mercedes García-Martínez, Ulrich Germann, Jesús González-Rubio, Robin Hill, Philipp Koehn, Luis A Leiva, et al.  Casmacat: A computer-assisted translation workbench.  In Proceedings of the Demonstrations at the 14th Conference of the European Chapter of the Association for Computational Linguistics, 2014.
  * [19] Philipp Koehn, Franz J. Och, and Daniel Marcu.  Statistical phrase-based translation.  In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 127–133, 2003.
  * [20] David Chiang.  A hierarchical phrase-based model for statistical machine translation.  In Proceedings of the 43rd annual meeting of the association for computational linguistics (acl’05), pages 263–270, 2005.
  * [21] Philipp Koehn.  Statistical machine translation.  Cambridge University Press, 2009.
  * [22] Zheng Chen and Kai-Fu Lee.  A new statistical approach to chinese pinyin input.  In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 241–247, 2000.
  * [23] Longyue Wang, Derek F Wong, Lidia S Chao, Yi Lu, and Junwen Xing.  A systematic comparison of data selection criteria for smt domain adaptation.  The Scientific World Journal, 2014, 2014.
  * [24] Longyue Wang, Yi Lu, Derek F Wong, Lidia S Chao, Yiming Wang, and Francisco Oliveira.  Combining domain adaptation approaches for medical text translation.  In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 254–259, 2014.
  * [25] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli.  Scaling neural machine translation.  In WMT, 2018.
  * [26] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [27] Wenxiang Jiao, Xing Wang, Shilin He, Irwin King, Michael Lyu, and Zhaopeng Tu.  Data Rejuvenation: Exploiting Inactive Training Examples for Neural Machine Translation.  In EMNLP, 2020.
  * [28] Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Shuming Shi, Michael Lyu, and Irwin King.  Self-training sampling with monolingual data uncertainty for neural machine translation.  In ACL, 2021.
  * [29] Guoping Huang, Jiajun Zhang, Yu Zhou, and Chengqing Zong.  A new input method for human translators: integrating machine translation effectively and imperceptibly.  In IJCAI, 2015.
  * [30] Sebastin Santy, Sandipan Dandapat, Monojit Choudhury, and Kalika Bali.  INMT: Interactive neural machine translation prediction.  In EMNLP-IJCNLP: System Demonstrations, November 2019.
  * [31] Muriel Vasconcellos and Marjorie León.  Spanam and engspan: machine translation at the pan american health organization.  Computational Linguistics, 11(2-3):122–136, 1985.
  * [32] Huayang Li, Lemao Liu, Guoping Huang, and Shuming Shi.  Gwlan: General word-level autocompletion for computer-aided translation.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
  * [33] Shanbo Cheng, Shujian Huang, Huadong Chen, Xinyu Dai, and Jiajun Chen.  Primt: A pick-revise framework for interactive machine translation.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1240–1249, 2016.
  * [34] Joern Wuebker, Spence Green, John DeNero, Saša Hasan, and Minh-Thang Luong.  Models and inference for prefix-constrained machine translation.  In Proceedings of ACL, 2016.
  * [35] Chris Hokamp and Qun Liu.  Lexically constrained decoding for sequence generation using grid beam search.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1535–1546, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * [36] Matt Post and David Vilar.  Fast lexically constrained decoding with dynamic beam allocation for neural machine translation.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1314–1324, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [37] Huayang Li, Guoping Huang, Deng Cai, and Lemao Liu.  Neural machine translation with noisy lexical constraints.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:1864–1874, 2020.
  * [38] Chenhui Chu, Raj Dabre, and Sadao Kurohashi.  An empirical comparison of domain adaptation methods for neural machine translation.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385–391, 2017.
  * [39] Rui Wang, Masao Utiyama, Lemao Liu, Kehai Chen, and Eiichiro Sumita.  Instance weighting for neural machine translation domain adaptation.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1482–1488, 2017.
  * [40] Chenhui Chu and Rui Wang.  A survey of domain adaptation for neural machine translation.  In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304–1319, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics.
  * [41] Lemao Liu, Hailong Cao, Taro Watanabe, Tiejun Zhao, Mo Yu, and Conghui Zhu.  Locally training the log-linear model for SMT.  In EMNLP, July 2012.
  * [42] Xiaoqing Li, Jiajun Zhang, and Chengqing Zong.  One sentence one model for neural machine translation.  arXiv preprint arXiv:1609.06490, 2016.
  * [43] M Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico.  Multi-domain neural machine translation through unsupervised adaptation.  In Proceedings of the Second Conference on Machine Translation, pages 127–137, 2017.
  * [44] Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li.  Search engine guided neural machine translation.  In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018, 2018\.
  * [45] Jingyi Zhang, Masao Utiyama, Eiichiro Sumita, Graham Neubig, and Satoshi Nakamura.  Guiding neural machine translation with retrieved translation pieces.  In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL), June 2018.
  * [46] Qiuxiang He, Guoping Huang, Lemao Liu, and Li Li.  Word position aware translation memory for neural machine translation.  In NLPCC, 2019.
  * [47] Mengzhou Xia, Guoping Huang, Lemao Liu, and Shuming Shi.  Graph based translation memory for neural machine translation.  In The Thirty-Third AAAI Conference on Artificial Intelligence, pages 7297–7304, 2019.
  * [48] Philipp Koehn.  Statistical machine translation.  Cambridge University Press, 2009.
  * [49] Chris Dyer, Adam Lopez, Juri Ganitkevitch, Jonathan Weese, Ferhan Türe, Phil Blunsom, Hendra Setiawan, Vladimir Eidelman, and Philip Resnik.  cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models.  In Proceedings of the ACL 2010 System Demonstrations, pages 7–12, 2010.
  * [50] Lidia Mangu, Eric Brill, and Andreas Stolcke.  Finding consensus among words: Lattice-based word error minimization.  In Sixth European Conference on Speech Communication and Technology, 1999.
  * [51] Lidia Mangu, Eric Brill, and Andreas Stolcke.  Finding consensus in speech recognition: word error minimization and other applications of confusion networks.  Computer Speech & Language, 14(4):373–400, 2000.
  * [52] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  In Proceedings of ICLR, 2018.
  * [53] Lemao Liu, Masao Utiyama, Andrew M. Finch, and Eiichiro Sumita.  Neural machine translation with supervised attention.  In COLING, pages 3093–3102, 2016.
  * [54] Xintong Li, Lemao Liu, Zhaopeng Tu, Shuming Shi, and Max Meng.  Target foresight based attention for neural machine translation.  In NAACL-HLT, pages 1380–1390, 2018.
  * [55] Yun Chen, Yang Liu, Guanhua Chen, Xin Jiang, and Qun Liu.  Accurate word alignment induction from neural machine translation.  In EMNLP, November 2020.
  * [56] Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi.  On the word alignment from neural machine translation.  In ACL, pages 1293–1303, 2019.
  * [57] Ding Shuoyang, Xu Hainan, and Koehn Philipp.  Saliency-driven word alignment interpretation for neural machine translation.  In WMT 2019, page 1.
  * [58] Jierui Li, Lemao Liu, Huayang Li, Guanlin Li, Guoping Huang, and Shuming Shi.  Evaluating explanation methods for neural machine translation.  In ACL, pages 365–375, July 2020.
  * [59] Chi Chen, Maosong Sun, and Yang Liu.  Mask-align: Self-supervised neural word alignment.  In arXiv, 2020.
  * [60] Franz Josef Och and Hermann Ney.  A systematic comparison of various statistical alignment models.  Computational linguistics, 29(1):19–51, 2003.
  * [61] Chris Dyer, Victor Chahuneau, and Noah A Smith.  A simple, fast, and effective reparameterization of ibm model 2.  In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, 2013.
  * [62] Stephan Vogel, Hermann Ney, and Christoph Tillmann.  Hmm-based word alignment in statistical translation.  In COLING, 1996.
  * [63] Longyue Wang.  Discourse-aware neural machine translation.  PhD thesis, Dublin City University, 2019.
  * [64] Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu.  Exploiting cross-sentence context for neural machine translation.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2826–2831, 2017.
  * [65] Longyue Wang, Xiaojun Zhang, Zhaopeng Tu, Andy Way, and Qun Liu.  Automatic construction of discourse corpora for dialogue translation.  In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2748–2754, 2016.
  * [66] Longyue Wang, Zhaopeng Tu, Xing Wang, Li Ding, Liang Ding, and Shuming Shi.  Tencent ai lab machine translation systems for wmt20 chat translation task.  In Proceedings of the Fifth Conference on Machine Translation, pages 483–491, 2020.
  * [67] Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, Hang Li, Andy Way, and Qun Liu.  A novel approach to dropped pronoun translation.  In Proceedings of NAACL-HLT, pages 983–993, 2016.
  * [68] Longyue Wang, Zhaopeng Tu, Shuming Shi, Tong Zhang, Yvette Graham, and Qun Liu.  Translating pro-drop languages with reconstruction models.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [69] Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu.  Learning to jointly translate and predict dropped pronouns with a shared reconstruction mechanism.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2997–3002, 2018.
  * [70] Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al.  Moses: Open source toolkit for statistical machine translation.  In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180. Association for Computational Linguistics, June 2007\.
  * [71] Haisong Zhang, Lemao Liu, Haiyun Jiang, Yangming Li, Enbo Zhao, Kun Xu, Linfeng Song, Suncong Zheng, Botong Zhou, Jianchen Zhu, et al.  Texsmart: A text understanding system for fine-grained ner and enhanced semantic analysis.  arXiv preprint arXiv:2012.15639, 2020.
  * [72] Rico Sennrich, Barry Haddow, and Alexandra Birch.  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany, August 2016. Association for Computational Linguistics.
  * [73] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In ACL, 2002.
  * [74] Tobias Domhan.  How much attention do you need? a granular analysis of neural machine translation architectures.  In ACL, 2018.
  * [75] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [76] Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli.  Pay less attention with lightweight and dynamic convolutions.  In ICLR, 2019.
  * [77] Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli.  Fairseq: A fast, extensible toolkit for sequence modeling.  In NAACL (Demonstrations), 2019.
  * [78] Myle Ott, Sergey Edunov, David Grangier, and Michael Auli.  Scaling neural machine translation.  In WMT, 2018.
  * [79] Philipp Koehn.  Statistical Significance Tests for Machine Translation Evaluation.  In EMNLP, 2004.
  * [80] Graham Neubig, Zi-Yi Dou, Junjie Hu, Paul Michel, Danish Pruthi, and Xinyi Wang.  compare-mt: A tool for holistic comparison of language generation systems.  In NAACL (Demonstrations), 2019.
  * [81] Guoping Huang, Jiajun Zhang, Yu Zhou, and Chengqing Zong.  Input method for human translators: A novel approach to integrate machine translation effectively and imperceptibly.  ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 18(1):1–22, 2018.
  * [82] Ralf Steinberger, Bruno Pouliquen, Anna Widiger, Camelia Ignat, Tomaž Erjavec, Dan Tufiş, and Dániel Varga.  The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages.  In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy, May 2006. European Language Resources Association (ELRA).
  * [83] Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan.  Training neural machine translation to apply terminology constraints.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3063–3068, Florence, Italy, July 2019. Association for Computational Linguistics.
  * [84] Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang.  Code-switching for enhancing NMT with pre-specified translation.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 449–459, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [85] Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush.  OpenNMT: Open-source toolkit for neural machine translation.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 67–72, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * [86] Andrzej Białecki, Robert Muir, Grant Ingersoll, and Lucid Imagination.  Apache lucene 4.  In SIGIR 2012 workshop on open source information retrieval, page 17, 2012.
  * [87] Jitao Xu, Josep Crego, and Jean Senellart.  Boosting neural machine translation with similar translations.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1580–1590, 2020.
  * [88] Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis.  Nearest neighbor machine translation.  arXiv preprint arXiv:2010.00710, 2020.
  * [89] Qiuxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu.  Fast and accurate neural machine translation with translation memory.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
  * [90] Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu.  Neural machine translation with monolingual translation memory.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
  * [91] Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan N Gomez, Stephan Gouws, Llion Jones, Łukasz Kaiser, Nal Kalchbrenner, Niki Parmar, et al.  Tensor2tensor for neural machine translation.  arXiv preprint arXiv:1803.07416, 2018.