# Cross-lingual Contextualized Phrase Retrieval

Huayang Li♣ Deng Cai${}^{{}^{\heartsuit}}$ Zhi Qu♣ Qu
Cui${}^{{}^{\heartsuit}}$
Hidetaka Kamigaito♣ Lemao Liu${}^{{}^{\heartsuit}}$ Taro Watanabe♣
♣Nara Institute of Science and Technology ♡Tencent AI Lab
{li.huayang.lh6, qu.zhi.pv5, kamigaito.h<EMAIL_ADDRESS>
{jcykcai, qucui<EMAIL_ADDRESS>Work done during Huayang’s internship
at Tencent AI Lab. Correspondence to Deng Cai and Lemao Liu.

###### Abstract

Phrase-level dense retrieval has shown many appealing characteristics in
downstream NLP tasks by leveraging the fine-grained information that phrases
offer. In our work, we propose a new task formulation of dense retrieval,
cross-lingual contextualized phrase retrieval, which aims to augment cross-
lingual applications by addressing polysemy using context information.
However, the lack of specific training data and models are the primary
challenges to achieve our goal. As a result, we extract pairs of cross-lingual
phrases using word alignment information automatically induced from parallel
sentences. Subsequently, we train our Cross-lingual Contextualized Phrase
Retriever (CCPR) using contrastive learning, which encourages the hidden
representations of phrases with similar contexts and semantics to align
closely. Comprehensive experiments on both the cross-lingual phrase retrieval
task and a downstream task, i.e, machine translation, demonstrate the
effectiveness of CCPR. On the phrase retrieval task, CCPR surpasses baselines
by a significant margin, achieving a top-1 accuracy that is at least 13 points
higher. When utilizing CCPR to augment the large-language-model-based
translator, it achieves average gains of 0.7 and 1.5 in BERTScore for
translations from X$\Rightarrow$En and vice versa, respectively, on WMT16
dataset. Our code and data are available at
https://github.com/ghrua/ccpr_release.

##  1 Introduction

Compared with the dense retrieval at sentence (or passage) level Karpukhin et
al. (2020); Borgeaud et al. (2022); Asai et al. (2024), learning dense
retrieval at the phrase level has shown more appealing characteristics in
extensive NLP tasks, such as entity linking Gillick et al. (2019), open-domain
question answering Lee et al. (2021a, b), text generation Lan et al. (2023);
Cao et al. (2024), etc. An important reason is that phrases can provide more
fine-grained information than sentences.

In cross-lingual research, the phrase-level dense retrieval also shows the
promise to solve a range of NLP problems Bapna and Firat (2019); Zheng et al.
(2022). The cross-lingual phrase retrieval in Zheng et al. (2022) focuses on
the mapping of cross-lingual wiki entities, where each of them is represented
by averaged hidden representations from 32 different contexts. However, the
constrained phrase type, i.e., the wiki entity, makes the cross-lingual phrase
retriever difficult to augment general NLP tasks. Moreover, unlike wiki
entities, which may have fewer ambiguities, general-type phrases that are
lexically identical can have different meanings depending on their contexts.
Thus, accounting for polysemy Cruse (1986) using the context information
becomes critical. Even when the lexically identical phrases share similar
semantics, a more appropriate context would provide richer information for
utilizing the phrases in downstream tasks Min et al. (2019); Cao et al.
(2024).

Therefore, we advocate for a new task formulation, i.e., cross-lingual
contextualized phrase retrieval, which aims to find the cross-lingual phrase
that is mostly relevant to the provided (general-type) source phrases,
considering their meanings and surrounding contexts. However, achieving this
goal is non-trivial, due to the scarcity of specific training data and models.
Since annotating cross-lingual contextualized phrase pairs of general type is
very difficult and expensive, we first introduce a data collection method,
which leverages the automatically induced word alignment information from
parallel sentences to extract suitable cross-lingual phrase pairs for
training. This ensures that the phrases are of general type and both cross-
lingual phrases and contexts are well aligned. Thereafter, we propose a Cross-
lingual Contextualized Phrase Retriever (CCPR) which is trained on the
constructed dataset. The CCPR mainly employs phrase-level contrastive learning
to draw cross-lingual phrases with similar contexts and meanings closer in the
hidden space. At inference time, we can leverage the well-trained CCPR to
build a phrase-level index and use the query phases for search. One remaining
question is how to select meaningful phrases for indexing. To address this
problem, we propose to learn a phrase segmentation module, which can be used
to predict meaningful phrases from sentence- or passage-level retrieval data
at inference time. Both of the modules are critical for employing our CCPR in
real-word applications.

Experiments on both of the cross-lingual contextualized phrase retrieval and a
downstream task, i.e., machine translation, show the effectiveness of CCPR. We
first evaluate CCPR on the cross-lingual contextualized phrase retrieval task,
since a higher accuracy is more beneficial for downstream tasks, such as
machine translation, cross-lingual dictionary induction, etc Zhang and Zong
(2016); Søgaard et al. (2018). For this task, we use the human annotated
cross-lingual phrase pairs as the golden truth and evaluate whether CCPR and
other baselines can retrieve those golden phrases from a large-scale index,
which contains approximately 10M phrases. In the experiments, CCPR outperforms
the baselines for at least 13 points in terms of the top-1 accuracy. We also
conduct evaluation on the machine translation task. The purpose of machine
translation task is to evaluate whether the information retrieved by CCPR is
beneficial for cross-lingual generation. As a result, we simply integrate the
retrieved phrase information to the input of an large language model (LLM),
e.g., LLama-2 Touvron et al. (2023), and compare our method with other
baselines. For both X$\Rightarrow$En and En$\Rightarrow$X translation
directions on the WMT16 dataset, where X is from six languages, our CCPR
achieves averaged gians of 0.7 and 1.5 BERTScore points, respectively.

In summary, our contributions are three-fold:

  * •

We propose a new formulation of dense retrieval, i.e., the cross-lingual
contextualized phrase retrieval, which has substantial potential in augmenting
cross-lingual tasks.

  * •

We propose a Cross-lingual Contextualized Phrase Retriever (CCPR), which
leverages the constructed training data, i.e., cross-lingual phrase pairs
extracted from automatically induced word alignment information, to learn both
the phrase alignment and segmentation modules.

  * •

We conduct extensive experiments on cross-lingual contextualized phrase
retrieval and machine translation. Our method outperforms the baselines by a
large margin.

##  2 Related Work

#### Dense Retrieval

In the literature of text generation, dense retrieval has been widely studied
at both sentence and phrase levels. Particularly after the emergence of large
language models (LLMs), how to use dense retrieval to retrieve related
sentences (or passages) Izacard et al. (2021); Ni et al. (2022) for input to
LLMs has became a popular research topic, namely, retrieval-augmented
generation (RAG) Lewis et al. (2020b); Karpukhin et al. (2020); Guu et al.
(2020); Borgeaud et al. (2022); Asai et al. (2024). However, compared to the
sentence-level retrieval, phrase-level dense retrieval has been shown to not
only enhance retrieval accuracy but also markedly boost performance in
downstream tasks like open-domain question answering (QA) Lee et al. (2021a,
b). By indexing and integrating retrieved phrases into the output of language
models, Lan et al. (2023) and Cao et al. (2024) make the text generation
process more attributable and accurate. These studies underline the
effectiveness of phrase-level dense retrieval. However, our work notably
diverges from those research works, because they focus more on the English-
only tasks.

#### Retrieval in Cross-lingual Tasks

In cross-lingual tasks, the majority of research has concentrated on sentence-
level retrieval. Many research works in retrieval-augmented machine
translation Zhang et al. (2018); Gu et al. (2018); Xia et al. (2019); He et
al. (2021) have explored retrieving bilingual sentences based on similarity of
source sentences, guiding translation models to utilize retrieved target
information. A notable limitation of existing approaches within this research
domain is their reliance on bilingual data for constructing the retrieval
index. This requirement substantially limits the scale and diversity of data
available for retrieval purposes. In contrast, Cai et al. (2021) propose to
directly retrieving relevant target sentences using source sentences to assist
machine translation, facilitating cross-lingual retrieval using a monolingual
index. Different from the focus of Cai et al. (2021), which aims for a task-
specific model for machine translation, numerous studies in dense retrieval
aim to enhance the cross-lingual sentence retrieval on more languages and at a
larger scale Chidambaram et al. (2019); Conneau et al. (2020); Reimers and
Gurevych (2020); Heffernan et al. (2022); Feng et al. (2022); Cai et al.
(2022).

However, cross-lingual phrase retrieval has not been extensively investigated.
Bapna and Firat (2019) utilize the source side of the bilingual data for
building a retrieval index and employ the similarity between source phrases
for retrieval. Therefore, their indexing also suffers from the limitation of
reliance on bilingual data. In addition, it uses $n$-grams as phrases by
default, leading to plenty of meaningless and noisy spans in the index. Zheng
et al. (2022) limit their cross-lingual phrase retrieval task to a specific
phrase type, i.e., the wiki entity. The setting in their work overlooks the
nuanced meanings that lexically identical phrases may convey in different
contexts, making the retriever difficult to augment general NLP tasks. In
contrast, our research aims to develop a cross-lingual contextualized phrase
retriever for general-type phrases.

Additionally, there is research focused on employing nearest neighbor
retrieval to support machine translation Khandelwal et al. (2021); Zheng et
al. (2021); Meng et al. (2022); Deguchi et al. (2023). These efforts differ
from the aforementioned studies as they aim to estimate prediction
distributions through nearest neighbor retrieval at each translation step.

#### Mining of Cross-lingual Phrase Pairs

The extraction of cross-lingual phrase pairs by utilizing the word alignment
information has been a standard of the Phrase-based Statistical Machine
Translation (PBSMT) Koehn et al. (2003); Chiang (2005). However, the phrase
table extracted in PBSMT is static and context-independent, where each source
phrase is mapped to multiple target phrases without considering the contexts.
Similarly, Zheng et al. (2022) extract cross-lingual phrase pairs based on
link information of wiki entities, which overlooks the nuanced meanings that
phrases may convey in different contexts. Given the long-recognized importance
of context in resolving polysemy within NLP Devlin et al. (2019), our training
data and model are crafted to be context-dependent, offering greater utility
for downstream NLP applications.

Figure 1: (a) Cross-lingual phrase alignment takes the cross-lingual phrase
pairs within similar contexts as positives and other in-batch phrases as
negatives. Encoders with different colors employ different dropout mask
$\bm{z}$, following Gao et al. (2021). The inputs of the dual encoders are
parallel sentences. (b) A linear phrase-segmentation head is used to predict
whether a span is a phrase or not. (c) The pipeline of using our learned model
for downstream tasks.

##  3 Task Formulation

Our research aims a cross-lingual phrase retrieval system that can augment
downstream NLP tasks. To ensure its effectiveness, the retrieval system must
adhere to two essential criteria. First, it should ensure that phrases in the
retrieval index are of general type. Second, it must resolve the polysemy of
general-type phrases by considering their contextual differences.

Thus, we introduce a new task formulation, cross-lingual contextualized phrase
retrieval. We are given a large collection of $N$ general-type phrases
$\mathcal{P}_{index}=\\{\bm{p}_{1},\dots,\bm{p}_{N}\\}$ in language $L_{y}$,
and a general-type query phrase $\bm{q}$ in language $L_{x}$. Each phrase, by
default, is associated with the context information $(\bm{c},s,e)$, where
$\bm{c}$ represents the original sentence containing this phrase, and $s$ and
$e$ are its start and end positions in $\bm{c}$. The primary objective is to
identify a cross-lingual phrase $\bm{p}\in\mathcal{P}_{index}$ that is
relevant to $\bm{q}$, considering their contexts and meanings. Our formulation
notably diverges from the one proposed by Zheng et al. (2022), which focuses
on a restricted type of phrases, i.e., wiki entities, ignoring the nuanced
meanings that general-type phrases may convey in different contexts.

This new task formulation also introduces several challenges, with the primary
issue being the lack of training data. Specifically, there is a scarcity of
cross-lingual phrase pairs of general type and accompanied by contextual
information. Such data are crucial for training the model to recognize cross-
lingual phrases that have similar contexts and meanings, allowing it to align
these phrases closely in a hidden space.

##  4 Training Data Collection

Annotating cross-lingual contextualized phrase pairs of general type is very
difficult and expensive, posing a significant obstacle to training effective
cross-lingual contextualized phrase retriever. However, the sentence-level
parallel data in general domains are more readily available Ng et al. (2019).
Additionally, as the lower unit of phrase alignment, the word alignment has
been extensive studied Brown et al. (1993); Och and Ney (2003); Dyer et al.
(2013); Li et al. (2019); Jalili Sabet et al. (2020); Dou and Neubig (2021);
Wu et al. (2023). Therefore, to overcome the data scarcity, we propose to use
a word alignment model, such as GIZA++ Och and Ney (2003) and neural aligner
Dou and Neubig (2021), in order to automatically induce word alignments from
parallel sentences and subsequently extract cross-lingual phrase pairs. This
method allows us to produce phrase pairs that are not only of general type but
also accompanied with contexts, aligning with our task requirements.

Formally, given a pair of parallel sentences
$\bm{x}=\\{x_{1},x_{2},...,x_{|\bm{x}|}\\}$, and
$\bm{y}=\\{y_{1},y_{2},...,y_{|\bm{y}|}\\}$, where $|\bm{x}|$ and $|\bm{y}|$
are the number of words in the sequences, we use a word alignment model to
obtain the word alignment information of $\bm{x}$ and $\bm{y}$ Koehn et al.
(2003); Och et al. (1999). If every word in a consecutive span $\bm{x}_{i:j}$
can be aligned to a consecutive span $\bm{y}_{u:v}$, we will use them as a
pair of aligned phrases $\bm{p}^{x}=\bm{x}_{i:j}$ and
$\bm{p}^{y}=\bm{y}_{u:v}$, where $1\leq i\leq j\leq|\bm{x}|$ and $1\leq u\leq
v\leq|\bm{y}|$. Notably, phrases that meet the constraints may range from a
single word to the whole sequence, and extracted phrases are allowed to have
overlaps. In addition, the context information of $\bm{p}^{x}$ and
$\bm{p}^{y}$ are $(\bm{x},i,j)$ and $(\bm{y},u,v)$, respectively.

As shown in Figure 1 (a), The Germany phrase (“einer”, “Schweigeminute”) is
aligned with the the English phrase (“a”, “minute”, “’s”, “silence”) within
similar contexts, i.e., parallel sentences. More implementation details about
the extraction of cross-lingual contextualized phrase pairs are discussed in
Section 6.1.

##  5 Methodology

###  5.1 Model Architecture

We introduce a Cross-lingual Contextualized Phrase Retriever (CCPR) for our
new formulated task. The main target of CCPR is to make the representations of
cross-lingual phrases with similar contexts and meanings to be close in the
hidden space. To this end, we propose a cross-lingual phrase alignment module
based on contrastive learning, utilizing the data collected in Section 4 for
training. In addition, one remaining problem is how to select phrases for
indexing at inference time. To address this, we propose a phrase segmentation
module that uses phrase representations to predict meaningful phrases from
sentences or paragraphs. At inference time, we can leverage the learned phrase
segmentation module to select phrases for indexing, ensuring the train-test
consistency.

#### Phrase Encoding

Our phrase encoder is based on a context encoder and an $\mathrm{MLP}$ layer.
The encoding of a phrase $\bm{p}=\bm{x}_{s:e}$, is defined as:

$\displaystyle\mathbf{H}^{z}$
$\displaystyle=\mathrm{ContextEncoder}(\bm{x},\bm{z})$
$\displaystyle\bm{h}^{z}_{p}$
$\displaystyle=\mathrm{MLP}_{align}\big{(}[\mathbf{H}^{z}_{s};\mathbf{H}^{z}_{e}]\big{)}$
(1)

where $\mathrm{ContextEncoder}(\bm{x},\bm{z})$ is a Transformer model, e.g.,
BERT Devlin et al. (2019); Feng et al. (2022) and RoBERTa Conneau et al.
(2020), that encodes the context $\bm{x}$ to a matrix
$\mathbf{H}^{z}\in\mathbb{R}^{N\times d}$ with a dropout mask $\bm{z}$. The
$\mathbf{H}^{z}_{s}$ and $\mathbf{H}^{z}_{e}$ are the hidden representations
of word $x_{s}$ and $x_{e}$, respectively, i.e., the start and end token of
the phrase. For example, in the phrase (“a”, “minute”, “’s”, “silence”),
$x_{s}=\text{``a''}$ and $x_{e}=\text{``silence''}$.
$\mathrm{MLP}_{align}(\cdot)$ is a function that maps the concatenation of two
hidden states $\mathbf{H}^{z}_{s}$ and $\mathbf{H}^{z}_{e}$ from
$\mathbb{R}^{2d}$ to $\mathbb{R}^{o}$, where $o$ is the output hidden size.
The encoding for phrases in context $\bm{y}$ is similar.

#### Cross-lingual Phrase Alignment

For a batch of parallel sentences, we can extract a collection of cross-
lingual phrase pairs
$\mathcal{P}_{pair}=\\{(\bm{p}^{x}_{i},\bm{p}^{y}_{i})\\}_{i=1}^{K}$, where
$\bm{p}^{x}_{i}$ and $\bm{p}^{y}_{i}$ are the positive examples of each other.
Since each positive phrase pair is from parallel sentences that share similar
semantics, directly learning on them may cause the model to learn a trivial
shortcut. Therefore, inspired by SimCSE Gao et al. (2021), we apply two
independently sampled dropout mask $\bm{z}$ and $\bm{z}^{\prime}$, to encode
phrases $\bm{p}^{x}_{i}$ and $\bm{p}^{y}_{i}$, respectively, following
Equation 1. The training objective for the $x\rightarrow y$ direction is:

$\displaystyle\mathcal{L}_{x\rightarrow y}$
$\displaystyle=-\frac{1}{K}\sum_{i=1}^{K}\log\frac{\exp(\bm{h}^{z}_{p^{x}_{i}}\cdot\bm{h}^{z^{\prime}}_{p^{y}_{i}})}{Z_{x}(i)},$
(2) $\displaystyle Z_{x}(i)$
$\displaystyle=\sum_{(p^{x}_{j},p^{y}_{j})\in\mathcal{P}_{pair}}\exp(\bm{h}^{z}_{p^{x}_{i}}\cdot\bm{h}^{z}_{p^{y}_{j}})$
(3)

where we use all the non-paired phrases in $\mathcal{P}_{pair}$ as in-batch
negatives in Equation 3. For bidirectional symmetry, our final loss is:

$\mathcal{L}_{align}=\mathcal{L}_{x\rightarrow y}+\mathcal{L}_{y\rightarrow
x}.$ (4)

The illustration of the contrastive learning is shown in Figure 1 (a). In
practice, we may use parallel data from multiple languages during training,
i.e., the source contexts $\bm{x}$ in a batch may come from multiple
languages, e.g., Germany, Czech, etc, which is inspired by the success of
multi-lingual training in Liu et al. (2020).

#### Phrase Segmentation

This module aims to learn how to select phrases from sentences or passages,
since most retrieval data at inference time comes in sentences or passages,
rather than phrases. To ensure the train-test consistency, we take the phrases
extracted in Section 4 as positive data, and all the other spans in the
corresponding contexts as negative data. Our phrase segmentation module is a
binary classifier defined as follows:

$\displaystyle
P(T=1)=\sigma\Big{(}\mathrm{MLP}_{seg}\big{(}[\mathbf{H}^{z}_{i};\mathbf{H}^{z}_{j}]\big{)}\Big{)}$
(5)

where $T$ is the label for the span $\bm{x}_{i:j}$, where $1\leq i\leq
j\leq|\bm{x}|$, $\sigma(\cdot)$ is an activation function,
$\mathrm{MLP}_{seg}$ is linear layer that maps the hidden representation from
$\mathbb{R}^{2d}$ to $\mathbb{R}$, and the definitions for
$\mathbf{H}^{z}_{i}$ and $\mathbf{H}^{z}_{j}$ are the same as in Equation 1.
We use the label $T=1$ for phrases and $T=0$ for non-phrase spans. The phrase
segmentation for sequence $\bm{y}$ is similar to Equation 5. In practice, the
number of non-phrase spans is significantly more than the number of phrases.
To mitigate data imbalance, we employ a strategy of randomly sampling an equal
number of non-phrase spans and phrases within a sentence during training Li et
al. (2020). The illustration of the phrase segmentation is shown in Figure 1
(b).

The training loss for the phrase segmentation is

$\displaystyle\mathcal{L}_{seg}=-\frac{1}{|\mathcal{S}|}$
$\displaystyle\sum_{\bm{p}\in\mathcal{S}}\Big{(}T_{p}\log P(T_{p}=1)+$ (6)
$\displaystyle(1-T_{p})\log\big{(}1-P(T_{p}=1)\big{)}\Big{)}$

where $\mathcal{S}$ is a set of spans, which are extracted from sequence
$\bm{x}$ or $\bm{y}$. The learning objective of the whole model becomes:

$\mathcal{L}=\mathcal{L}_{align}+\beta\mathcal{L}_{seg}$ (7)

where $\beta$ is a hyper-parameter. It is worth noting that the parameters of
$\mathrm{ContextEncoder}$ used for phrase alignment and segmentation are
shared.

… Explanations about the phrase information …
————————————
Germany Phrase: Premierminister
Potential Translation: prime ministers
Context: … leaders stayed away, including the [[prime ministers]] of Canada
and India …

Germany Phrase: Indiens und Japans
Potential Translation: India and Japan
Context: … right governments in [[India and Japan]], the weakening clout of
Arab oil …

Germany Phrase: trafen
Potential Translation: met
Context: … Obama and Abe [[met]] with Japanese university students …

Germany Phrase: Tokio
Potential Translation: Tokyo
Context: … Canadian officials privately point fingers at [[Tokyo]] …
…
————————————
Based on the provided information of phrase translation, please faithfully
translate the following sentence from Germany into English:

Germany: Die Premierminister Indiens und Japans trafen sich in Tokio.

English: India and Japan prime ministers meet in Tokyo

Figure 2: An example instruction for a large-language-model based translator
augmented by our method. The phrases in the source sentence are in blue text.
The retrieved translation of the phrases are in green, and their appearances
in the contexts are marked by “[[]]”. The text after “English:” is the
reference, which is for illustration and will not appear in the instruction.

###  5.2 Inference Pipeline

The pipeline of employing our cross-lingual contextualized phrase retriever
for downstream NLP tasks involves two main steps: index building and
searching. An illustration of our inference pipeline is shown in Figure 1 (c).

A significant benefit of our retriever is its ability to leverage vast
monolingual data in the target language, eliminating the need for bilingual
data to construct the index at inference time Zhang et al. (2018); Gu et al.
(2018); Bapna and Firat (2019). Given the retrieval data in target language,
our retriever begins by segmenting sentences (or passages) into phrases. When
segmenting a sentence (or passage), we will enumerate all the possible spans
within it and calculate their probabilities of being meaningful phrases,
according to Equation 5. We select all the spans whose probabilities are
larger than a threshold for indexing. It is worth noting that overlapping
phrases are allowed at the phrase segmentation time. The selected phrases are
encoded into hidden representations according to Equation 1, which will be
used to construct a dense-retrieval index.

For querying, we first process the query sentence (or paragraph) in a source
language similarly, i.e., segmenting and encoding. Subsequently, we use the
encoded query phrases to directly search for the related cross-lingual phrases
using Maximum Inner Product Search (MIPS). Because of the advanced data
structure and search algorithm Malkov and Yashunin (2018); Douze et al.
(2024); Johnson (2022), the search step is highly efficient. The retrieved
information, including the target phrase, its accompanied context, and its
positions in the context, can then be integrated to downstream tasks to
improve the task performance. As shown in Figure 2, the retrieved phrases
along with their surrounding contexts are integrated into the instructions
given to an LLM for tasks such as machine translation.

###  5.3 Discussion

In our preliminary studies, we also investigated various phrase segmentation
strategies, including $n$-gram segmentation and learning a Byte Pair Encoding
(BPE) model Sennrich et al. (2016); Kudo (2018) across word boundaries.
However, we found these methods often struggle to identify meaningful phrases,
leading to significant discrepancies and performance declines between training
and testing phases. Consequently, we decide to develop a trainable phrase
segmentation model. In Section 6.3, we evaluate the effect of the learned
phrase segmentation module on machine translation task.

##  6 Experiments

We propose to evaluate our model on two tasks: cross-lingual contextualized
phrase retrieval and machine translation .

###  6.1 Implementation Details

#### Training Data Collection

We use all the bilingual training datasets of WMT16 on
Huggingface111https://huggingface.co/datasets/wmt16 to train our Cross-lingual
Contextualized Phrase Retriever (CCPR). The WMT16 dataset has six language
pairs, inclduing De-En, Cs-En, Fi-En, Ru-En, and Tr-En. Because of the
efficiency and the satisfactory performance of GIZA++ Och and Ney (2003); Dou
and Neubig (2021), we first use the GIZA++
software222https://github.com/moses-smt/mgiza, i.e., IBM-4 model Brown et al.
(1993), to induce the word alignment for each pair of parallel sentences, and
then extract the cross-lingual phrase pairs. Notably, we did not induce the
phrase table as in Phrase-based Statistical Machine Translation (PBSMT) Koehn
et al. (2003). Instead, in our setting, we extract the cross-lingual
consecutive spans that are aligned in a pair of parallel sentence as cross-
lingual phrase pairs. Each of the cross-lingual phrase pairs is associated
with the surrounding contexts. We filter out the phrases which begin or end
with words whose frequency is more than 30k in the training dataset. We also
filter out the phrases that contain only numbers and punctuations. The final
training dataset contains 1.3 billion cross-lingual phrase pairs extracted
from 10 million parallel sentences.

#### Model

In our work, we train two variants of our model, CCPR-XLMR and CCPR-LaBSE,
whose parameters are initialized from XLM-RoBERTa-base (XLMR) Conneau et al.
(2020) and LaBSE Feng et al. (2022), respectively. The hidden size $d$ of both
models is 768. The output $o$ of
$\mathrm{MLP}_{align}:\mathbf{R}^{2d}\rightarrow\mathbf{R}^{o}$ in Equation
(1) is 128, which plays a critical role in reducing the memory cost of our
phrase index while maintaining comparable performance. Each model is trained
on the mixture of all language pairs of WMT16. We train our model on 8 V100
GPUs for 20K steps, where the learning rate is 5e-5, dropout rate is 0.2,
batch size on each device is 64. The $\beta$ for the phrase segmentation loss
in Equation 5 is $1$. Those hyper-parameters are tuned on the validation
dataset. At inference time, we use the FAISS
library333https://github.com/facebookresearch/faiss Douze et al. (2024) to
build our retrieval index. We use the FlatIP as our index type.

Model | De$\Rightarrow$En | Ro$\Rightarrow$En | Cs$\Rightarrow$En | AVG.
---|---|---|---|---
XLMR | 1.5 | 0.0 | 4.0 | 1.8
mBERT | 8.0 | 3.0 | 7.5 | 6.1
mUSE | 34.5 | 30.0 | 45.5 | 36.6
LaBSE | 52.0 | 33.5 | 61.0 | 48.8
1-5 CPR-XLMR | 54.0 | 63.0 | 73.0 | 63.3
CPR-LaBSE | 57.0 | 60.0 | 78.5 | 65.1
1-5 CCPR-XLMR | 74.5 | 73.5 | 82.5 | 76.8
CCPR-LaBSE | 75.0 | 72.5 | 88.0 | 78.5

Table 1: Cross-lingual contextualized phrase retrieval. We use the accuracy@1
as our evaluation metric. The CCPR and CPR denote contextualized and context-
independent cross-lingual phrase retriever, respectively. The Best results are
highlighted in bold text.

Model (X$\Rightarrow$En) | De$\Rightarrow$En | Cs$\Rightarrow$En | Fi$\Rightarrow$En | Ru$\Rightarrow$En | Ro$\Rightarrow$En | Tr$\Rightarrow$En | AVG.
---|---|---|---|---|---|---|---
Llama-2-7B | 75.42 | 69.03 | 66.13 | 71.65 | 72.89 | 55.10 | 68.37
\+ XLMR | 75.32 | 69.15 | 66.06 | 71.74 | 73.07 | 55.49 | 68.47
\+ LaBSE | 75.74 | 69.60 | 66.81 | 72.12 | 73.22 | 56.76 | 69.04
1-8 \+ CCPR-XLMR | 75.84 | 70.11 | 66.70 | 71.49 | 73.53 | 58.44 | 69.35
\+ CCPR-LaBSE | 75.96 | 70.05 | 67.17 | 72.28 | 73.57 | 59.22 | 69.70
Model (En$\Rightarrow$X) | En$\Rightarrow$De | En$\Rightarrow$Cs | En$\Rightarrow$Fi | En$\Rightarrow$Ru | En$\Rightarrow$Ro | En$\Rightarrow$Tr | AVG.
Llama-2-7B | 63.42 | 52.32 | 48.96 | 84.18 | 82.99 | 58.47 | 65.05
\+ XLMR | 64.03 | 52.34 | 49.04 | 84.08 | 82.99 | 58.66 | 65.19
\+ LaBSE | 64.26 | 52.90 | 49.76 | 84.28 | 83.05 | 60.02 | 65.71
1-8 \+ CCPR-XLMR | 64.42 | 54.53 | 51.12 | 84.93 | 83.18 | 63.40 | 66.92
\+ CCPR-LaBSE | 64.29 | 54.93 | 51.54 | 84.97 | 83.36 | 64.07 | 67.19

Table 2: Machine translation on the test sets of WMT16. For all the retrieval-
based method, the index is built on the monolingual newscrawl data of the
target language. We use the BERTScore Zhang et al. (2019) as the evaluation
metric. The best performance of each translation direction is highlighted in
bold text. Results on additional evaluation metrics, e.g., COMET Rei et al.
(2020), are in Appendix B.2

###  6.2 Cross-lingual Contextualized Phrase Retrieval

As outlined in Section 3, given a query phrase and its context in source
language, the objective of this task is to identify the most relevant cross-
lingual phrase from a large-scale index in target language, emphasizing the
importance of relevance not only for the query and retrieved phrases
themselves but also for their respective contexts.

#### Setup

We build the test set based on the human annotated word alignment data Jalili
Sabet et al. (2020). We process the word alignment data on three language
pairs, De$\Rightarrow$En Ghader and Monz (2017), Ro$\Rightarrow$En Mihalcea
and Pedersen (2003), and Cs$\Rightarrow$En Mareček (2011). For the
Cs$\Rightarrow$En dataset, we only leverage the data under the “pcedt” split,
which are data from The Wall Street Journal (WSJ). For each language pair, we
instruct human annotators to identify 200 high-quality phrase pairs. The
details of the human annotation process are discussed in Section A.1. These
annotated source phrases serve as queries, while their aligned target phrases
help construct the retrieval index. To mimic real-world conditions, where a
retrieval index often contains extra data, we supplement the index with
additional phrases. For the uniformity of phrase segmentation across all
methods, we utilize the phrases extracted by GIZA++ in the training dataset as
the extra data. More concretely, we extract 9.6 million English phrases from a
sample of 1 million parallel sentences from the WMT16 training dataset. Thus,
each retrieval index contains all of the human-annotated target phrases and
9.6M extra phrases.

We compare our models, CCPR-XLMR and CCPR-LaBSE, with several baselines:

  * •

XLMR & mBERT: We use the pre-trained XLMR Devlin et al. (2019) and mBERT
Conneau et al. (2020) models on huggingface444https://huggingface.co/models to
encode the phrase contexts. We represent the phrases by concatenating the
hidden representations of their start and end tokens.

  * •

mUSE & LaBSE: Compared with previous baselines, the only difference here is
that we use the v2 versoin of Multilingual Universal Sentence Encoder (mUSE)
Chidambaram et al. (2019) and LaBSE Feng et al. (2022) released by
SBERT555https://www.sbert.net Reimers and Gurevych (2020) to encode the
contexts.

  * •

CPR-X: To evaluate the effect of context-awareness, we propose two context-
independent counterparts, i.e., CPR-XLMR and CPR-LaBSE. Compared with our
contextualized models, the only difference is that the CPR-X methods use
cross-lingual phrase pairs in semantically different contexts as positive
examples, similar to the setup in Zheng et al. (2022).

For each of the models, we build an independent index for retrieval. Each of
the index includes the hidden representations of human-annotated target
phrases and 9.6M extra phrases encoded by the corresponding model. In this
task, we use the top-1 accuracy to evaluate our methods and baselines.

Model | Index | En$\Rightarrow$Ro | En$\Rightarrow$Cs | En$\Rightarrow$Tr | AVG.
---|---|---|---|---|---
CCPR-XLMR | Tgt. of Train | 82.89 | 53.66 | 62.03 | 66.19
CCPR-XLMR | Newscrawl | 83.18 | 54.53 | 63.40 | 67.03
1-6 CCPR-LaBSE | Tgt. of Train | 83.19 | 54.23 | 63.22 | 66.88
CCPR-LaBSE | Newscrawl | 83.36 | 54.93 | 64.07 | 67.45

Table 3: Analysis of the retrieval data for indexing. The “Tgt. of Train” and
“Newscrawl” indicate using the target-side data of WMT16 training dataset and
monolingual newscrawl data to build the retrieval index, respectively. We use
BERTScore as our evaluation metric Zhang et al. (2019)

#### Results

Table 1 demonstrates that our models significantly outperform baselines in
cross-lingual contextualized phrase retrieval, highlighting the capability of
our models to accurately identify relevant cross-lingual phrases while being
context-aware. In contrast, the CPR-LaBSE and CPR-XLMR, i.e., the context-
independent baselines, exhibit notably poorer performance.

Furthermore, despite being trained solely on sentence-level data, LaBSE still
achieves moderate performance, surpassing many baselines. This supports the
finding that LaBSE implicitly learns the word alignment information, as
suggested by Wang et al. (2022). However, the cross-lingual contextualized
phase retrieval is difficult for baselines that are only trained on
multilingual data using auto-regressive loss, such as XLMR and mBERT.

###  6.3 Machine Translation

The evaluation on machine translation task is to assess if the information
retrieved by our models can enhance performance on downstream tasks. Rather
than training a new machine translation model Vaswani et al. (2017), we
propose to integrate the information to the prompt of the large language model
(LLM), e.g., Llama-2 Touvron et al. (2023), following the retrieval-augmented
generation (RAG) methods Lewis et al. (2020a). Thus, we can fairly evaluate
the effectiveness of the retrieved information.

#### Setup

We choose the Llama-2-7B Touvron et al. (2023) model as the backbone model for
the machine translation task. We also consider two cross-lingual sentence
retrieval models as baseline methods, the XLM-RoBERTa-base (XLMR) model
Conneau et al. (2020) and LaBSE Feng et al. (2022). An example of the
instruction of our method is shown in Figure 2, where the retrieved target
phrases with their contexts are presented along with the source phrases in the
input to Llama-2-7B. It is worth noting that the retrieved surrounding context
of each phrase is truncated to no more than 100 characters. The instructions
of other baselines are shown in Appendix B.1. For all the retrieval-based
methods, we build index on the monolingual newscrawl data of 2016
666https://data.statmt.org/news-
crawl/${LANG}/news.2016.${LANG}.shuffled.deduped.gz, and only use the top-1
retrieved result.

The indexing strategies diverge between the baseline methods and our approach.
For sentence-level baselines, i.e., XLMR and LaBSE, the indexing is
straightforward, directly using the sentences. In contrast, our method employs
a learned phrase segmentation module to extract phrases from sentences for
indexing. When applying our model to machine translation, we set the phrase
segmentation thresholds in Equation 5 to $0.7$ for indexing and $0.9$ for
querying. The rationale behind a lower indexing threshold is to populate the
index with a broader array of phrases, enhancing its comprehensiveness.
Conversely, a higher threshold for query documents ensures that the segmented
phrases are more accurate. These hyper-parameters were tuned using the
validation dataset from WMT16.

Since the output format of the Llama-base model is hard to control and the
Llama-chat model, i.e., the model after instruction-tuning, may reject to
translate a sentence that contains negative words, we decide to fine-tune the
Llama base model using LoRA Hu et al. (2022) for 100 steps to align the
translation format. We use Platypus777https://github.com/arielnlee/Platypus
Lee et al. (2023) as our code base. We follow all the hyper-parameters of
Platypus to fine-tune the LoRA module. The data we used for fine-tuning
contain 1,800 parallel sentences randomly sampled from the WMT16 training set.
For each method, we only fine-tune one model for the X$\Rightarrow$En (or
En$\Rightarrow$X) direction, where X is the non-English language.

Segmentation | En$\Rightarrow$Ro | En$\Rightarrow$Cs | En$\Rightarrow$Tr | AVG.
---|---|---|---|---
N-gram | 83.16 | 53.42 | 60.74 | 65.77
PSM | 83.36 | 54.93 | 64.07 | 67.45

Table 4: Analysis of phrase segmentation methods. The $N$-gram method uses all
available 5-grams for indexing and searching, whereas PSM employs the learned
phrase segmentation module. The segmented phrases are encoded by the same
encoder, i.e., CCPR-LaBSE. We use BERTScore as the evaluation metric.

#### Main Results

As shown in Table 2, both of our CCPR-XLMR and CCPR-LaBSE outperform the
baseline methods when assisting the large language model (LLM) with the
information from cross-lingual contextualized phrase retrieval. Especially on
some low-resource directions, such as the Tr$\Leftrightarrow$En,
En$\Rightarrow$Cs, En$\Rightarrow$Fi, CCPR-LaBSE outperforms other baselines
by more or around 2 BERTScore points . This indicates that the the cross-
lingual phrase information is more critical for improving the performance on
low resource languages, because the percentage of Cs, Fi, and Tr data in
Llama-2 training are all less than $0.03\%$ Touvron et al. (2023). We also
evaluate our methods using additional metrics, e.g., COMET Rei et al. (2022),
as shown in Appendix B.2. The results on COMET Rei et al. (2022) are
consistent with those on BERTScore Zhang et al. (2019).

Additionally, we observe a good correlation between the performance of our
cross-lingual contextualized phrase retrieval task and downstream tasks, such
as machine translation. This suggests that models demonstrating higher
retrieval accuracy tend to achieve superior results in downstream
applications. Moreover, CCPR-LaBSE consistently outperforms CCPR-XLMR,
highlighting the advantage of pre-training on cross-lingual sentence retrieval
for enhancing the capability of contextualized phrase retrieval.

#### Analysis

In Table 2, all retrieval-based methods construct their index using the
monolingual newscrawl data from the target language, which is approximately
six times the size of the bilingual training data. It highlights a key benefit
of our cross-lingual phrase retrieval: the ability to utilize extensive
monolingual resources to build the index Cai et al. (2021), moving beyond the
constraints of solely relying on bilingual data for indexing Bapna and Firat
(2019); Zhang et al. (2018); Gu et al. (2018). Therefore, in Table 3, we
evaluate the differences of indexing on monolingual data and the target side
of the bilingual data. We observe that exploring the vast monolingual data
leads to significantly better performance for both of the CCPR-XMLR and CCPR-
LaBSE models.

In Table 4, we also analyze the effect of the learned phrase segmentation
module (PSM). It demonstrates that PSM outperforms the $N$-gram method by 1.8
BERTScore points in average. Although the $N$-gram is widely used for phrase
segmentation in previous works Bapna and Firat (2019); Lee et al. (2021a, b);
Lan et al. (2023), our experiments demonstrate that a learned phase
segmentation is more suitable for augmenting cross-lingual tasks. The
explanation for this observation was discussed in Section 5.3.

##  7 Conclusion & Future Works

This paper presents a new formulation of dense retrieval, cross-lingual
contextualized phrase retrieval, which aims to resolve the polysemy of
general-type phrases by considering the context information. However, the
primary issue is the lack of training data, i.e., cross-lingual phrase pairs
of general type and with contexts. To address this issue, we propose to
leverage the word alignment model to automatically induce word alignment
information from parallel sentences and then extract cross-lingual phrase
pairs for training, ensuring that extracted phrases are in general type and
both phrases pairs and their contexts are aligned. Afterwards, we also
introduce a Cross-lingual Contextualized Phrase Retriver (CCPR), which is
mainly based on the contrastive learning, to learn close representations for
cross-lingual phrases with similar meanings and surrounding contexts. We
conduct extensive experiments on the cross-lingual contextualized phrase
retrieval and machine translation tasks, which demonstrate that CCPR
outperforms baselines by a large margin.

We believe that our cross-lingual contextualized phrase retrieval has big
potential for augmenting cross-lingual tasks. In the future, we plan to focus
on the acquisition of higher quality training data and on expanding our
methodology to a broader range of cross-lingual applications.

##  8 Limitations

While our cross-lingual contextualized phrase retrieval holds substantial
potential, it is not without limitations. Notably, the phrase-level index
required by our approach is considerably larger than that of a sentence-level
index, given that each sentence can encompass numerous phrases. This expansion
necessitates increased disk space for storing the index, requiring additional
engineering techniques to maintain the scalability of our cross-lingual
contextualized phrase retriever (CCPR), e.g., index quantization or sharding.
Furthermore, to improve the performance of large language models (LLMs) on
cross-lingual tasks, it becomes necessary to integrate the information from
multiple retrieved phrases of the query sentence into the LLM input. This
integration process can lead to a rise in the inference costs associated with
LLMs.

##  9 Ethical Statement

We focus on leveraging cross-lingual contextualized phrase retrieval to
augment the performance of downstream NLP tasks. We emphasize that our model
is strictly designed and applied in a manner that avoids the generation of
sensitive information, such as disinformation or content aimed at deceiving
individuals. Furthermore, we assure that all data utilized for the training
and evaluation of our model have been sourced from publicly accessible
datasets. Our commitment to ethical research ensures our work benefits the NLP
field responsibly, without compromising the ethical standards.

##  10 Acknowledgement

We thank Leyang Cui, Patrick Lewis, Akari Asai, and Tianyu Gao for their
invaluable contributions through discussions during the initial phases of this
work.

## References

  * Asai et al. (2024) Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024.  Self-RAG: Learning to retrieve, generate, and critique through self-reflection.  In _The Twelfth International Conference on Learning Representations_.
  * Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019.  Non-parametric adaptation for neural machine translation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1921–1931, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Borgeaud et al. (2022) Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. 2022.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2206–2240. PMLR.
  * Brown et al. (1993) Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993.  The mathematics of statistical machine translation: Parameter estimation.  _Computational Linguistics_ , 19(2):263–311.
  * Cai et al. (2022) Deng Cai, Xin Li, Jackie Chun-Sing Ho, Lidong Bing, and Wai Lam. 2022.  Retrofitting multilingual sentence embeddings with Abstract Meaning Representation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6456–6472, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Cai et al. (2021) Deng Cai, Yan Wang, Huayang Li, Wai Lam, and Lemao Liu. 2021.  Neural machine translation with monolingual translation memory.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7307–7318, Online. Association for Computational Linguistics.
  * Cao et al. (2024) Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi, Yuexian Zou, and Shuming Shi. 2024.  Retrieval is accurate generation.  In _The Twelfth International Conference on Learning Representations_.
  * Chiang (2005) David Chiang. 2005.  A hierarchical phrase-based model for statistical machine translation.  In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_ , pages 263–270, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Chidambaram et al. (2019) Muthu Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yunhsuan Sung, Brian Strope, and Ray Kurzweil. 2019.  Learning cross-lingual sentence representations via a multi-task dual-encoder model.  In _Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)_ , pages 250–259, Florence, Italy. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Cruse (1986) D Alan Cruse. 1986.  _Lexical semantics_.  Cambridge university press.
  * Deguchi et al. (2023) Hiroyuki Deguchi, Taro Watanabe, Yusuke Matsui, Masao Utiyama, Hideki Tanaka, and Eiichiro Sumita. 2023.  Subset retrieval nearest neighbor machine translation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 174–189, Toronto, Canada. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dou and Neubig (2021) Zi-Yi Dou and Graham Neubig. 2021.  Word alignment by fine-tuning embeddings on parallel corpora.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2112–2128, Online. Association for Computational Linguistics.
  * Douze et al. (2024) Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024.  The faiss library.
  * Dyer et al. (2013) Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013.  A simple, fast, and effective reparameterization of IBM model 2.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–648, Atlanta, Georgia. Association for Computational Linguistics.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ghader and Monz (2017) Hamidreza Ghader and Christof Monz. 2017.  What does attention in neural machine translation pay attention to? In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 30–39, Taipei, Taiwan. Asian Federation of Natural Language Processing.
  * Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019.  Learning dense representations for entity retrieval.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 528–537, Hong Kong, China. Association for Computational Linguistics.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2018.  Search engine guided neural machine translation.  In _AAAI Conference on Artificial Intelligence_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International conference on machine learning_ , pages 3929–3938. PMLR.
  * He et al. (2021) Qiuxiang He, Guoping Huang, Qu Cui, Li Li, and Lemao Liu. 2021.  Fast and accurate neural machine translation with translation memory.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3170–3180, Online. Association for Computational Linguistics.
  * Heffernan et al. (2022) Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022.  Bitext mining using distilled sentence representations for low-resource languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Izacard et al. (2021) Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021.  Unsupervised dense information retrieval with contrastive learning.
  * Jalili Sabet et al. (2020) Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020.  SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1627–1643, Online. Association for Computational Linguistics.
  * Johnson (2022) Christian Johnson. 2022.  Binary encoded word mover’s distance.  In _Proceedings of the 7th Workshop on Representation Learning for NLP_ , pages 167–172, Dublin, Ireland. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Khandelwal et al. (2021) Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021.  Nearest neighbor machine translation.  In _International Conference on Learning Representations_.
  * Koehn et al. (2003) Philipp Koehn, Franz J. Och, and Daniel Marcu. 2003.  Statistical phrase-based translation.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 127–133.
  * Kudo (2018) Taku Kudo. 2018.  Subword regularization: Improving neural network translation models with multiple subword candidates.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 66–75, Melbourne, Australia. Association for Computational Linguistics.
  * Lan et al. (2023) Tian Lan, Deng Cai, Yan Wang, Heyan Huang, and Xian-Ling Mao. 2023.  Copy is all you need.  In _The Eleventh International Conference on Learning Representations_.
  * Lee et al. (2023) Ariel N. Lee, Cole J. Hunter, and Nataniel Ruiz. 2023.  Platypus: Quick, cheap, and powerful refinement of llms.
  * Lee et al. (2021a) Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, and Danqi Chen. 2021a.  Learning dense representations of phrases at scale.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6634–6647, Online. Association for Computational Linguistics.
  * Lee et al. (2021b) Jinhyuk Lee, Alexander Wettig, and Danqi Chen. 2021b.  Phrase retrieval learns passage retrieval, too.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3661–3672, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lewis et al. (2020a) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020a.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 9459–9474. Curran Associates, Inc.
  * Lewis et al. (2020b) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020b.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474.
  * Li et al. (2019) Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019.  On the word alignment from neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1293–1303, Florence, Italy. Association for Computational Linguistics.
  * Li et al. (2020) Yangming Li, Shuming Shi, et al. 2020.  Empirical analysis of unlabeled entity problem in named entity recognition.  In _International Conference on Learning Representations_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Malkov and Yashunin (2018) Yu A Malkov and Dmitry A Yashunin. 2018.  Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(4):824–836.
  * Mareček (2011) David Mareček. 2011.  Automatic alignment of tectogrammatical trees from czech-english parallel corpus.
  * Meng et al. (2022) Yuxian Meng, Xiaoya Li, Xiayu Zheng, Fei Wu, Xiaofei Sun, Tianwei Zhang, and Jiwei Li. 2022.  Fast nearest neighbor machine translation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 555–565, Dublin, Ireland. Association for Computational Linguistics.
  * Mihalcea and Pedersen (2003) Rada Mihalcea and Ted Pedersen. 2003.  An evaluation exercise for word alignment.  In _Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts: Data Driven Machine Translation and Beyond_ , pages 1–10.
  * Min et al. (2019) Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019.  A discrete hard EM approach for weakly supervised question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2851–2864, Hong Kong, China. Association for Computational Linguistics.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019.  Facebook FAIR’s WMT19 news translation task submission.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 314–319, Florence, Italy. Association for Computational Linguistics.
  * Ni et al. (2022) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1864–1874, Dublin, Ireland. Association for Computational Linguistics.
  * Och and Ney (2003) Franz Josef Och and Hermann Ney. 2003.  A systematic comparison of various statistical alignment models.  _Computational Linguistics_ , 29(1):19–51.
  * Och et al. (1999) Franz Josef Och, Christoph Tillmann, and Hermann Ney. 1999.  Improved alignment models for statistical machine translation.  In _1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora_.
  * Rei et al. (2022) Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022.  COMET-22: Unbabel-IST 2022 submission for the metrics shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 578–585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Søgaard et al. (2018) Anders Søgaard, Sebastian Ruder, and Ivan Vulić. 2018.  On the limitations of unsupervised bilingual dictionary induction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 778–788, Melbourne, Australia. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2022) Weikang Wang, Guanhua Chen, Hanqing Wang, Yue Han, and Yun Chen. 2022.  Multilingual sentence transformer as a multilingual word aligner.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2952–2963, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wu et al. (2023) Qiyu Wu, Masaaki Nagata, and Yoshimasa Tsuruoka. 2023.  WSPAlign: Word alignment pre-training via large-scale weakly supervised span prediction.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 11084–11099, Toronto, Canada. Association for Computational Linguistics.
  * Xia et al. (2019) Mengzhou Xia, Guoping Huang, Lemao Liu, and Shuming Shi. 2019.  Graph based translation memory for neural machine translation.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 7297–7304.
  * Zhang and Zong (2016) Jiajun Zhang and Chengqing Zong. 2016.  Bridging neural machine translation and bilingual dictionaries.  _arXiv preprint arXiv:1610.07272_.
  * Zhang et al. (2018) Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig, and Satoshi Nakamura. 2018.  Guiding neural machine translation with retrieved translation pieces.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1325–1335, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zheng et al. (2022) Heqi Zheng, Xiao Zhang, Zewen Chi, Heyan Huang, Yan Tan, Tian Lan, Wei Wei, and Xian-Ling Mao. 2022.  Cross-lingual phrase retrieval.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4193–4204, Dublin, Ireland. Association for Computational Linguistics.
  * Zheng et al. (2021) Xin Zheng, Zhirui Zhang, Junliang Guo, Shujian Huang, Boxing Chen, Weihua Luo, and Jiajun Chen. 2021.  Adaptive nearest neighbor machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 368–374, Online. Association for Computational Linguistics.

Model | De-En | Cs-En | Fi-En | Ru-En | Ro-En | Tr-En | AVG.
---|---|---|---|---|---|---|---
Llama-2-7B | 86.019 | 82.509 | 84.955 | 83.616 | 85.005 | 77.551 | 83.27
\+ XLMR | 85.915 | 82.571 | 84.889 | 83.575 | 84.984 | 77.809 | 83.29
\+ LaBSE | 86.079 | 82.840 | 85.136 | 83.750 | 85.059 | 78.552 | 83.56
1-8 \+ CCPR-XLMR | 86.217 | 83.079 | 85.231 | 83.745 | 85.285 | 79.502 | 83.84
\+ CCPR-LaBSE | 86.202 | 83.014 | 85.342 | 83.855 | 85.322 | 80.123 | 83.97
| En-De | En-Cs | En-Fi | En-Ru | En-Ro | En-Tr |
Llama-2-7B | 82.396 | 79.131 | 82.516 | 84.191 | 82.588 | 58.959 | 78.29
\+ XLMR | 82.821 | 78.888 | 82.442 | 84.097 | 82.802 | 59.095 | 78.35
\+ LaBSE | 83.128 | 79.533 | 83.214 | 84.403 | 82.849 | 61.183 | 79.05
1-8 \+ CCPR-XLMR | 83.529 | 80.517 | 83.962 | 85.377 | 83.776 | 65.975 | 80.52
\+ CCPR-LaBSE | 83.042 | 80.860 | 84.095 | 85.308 | 84.031 | 67.060 | 80.73

Table 5: Machine Translation. COMET.

… Explanations about the retrieved sentence …
————————————
Related English Sentence: The Prime Ministers of India and Pakistan recently
met in Pakistan to discuss the question.
…
————————————
Based on the provided related sentence, please faithfully translate the
following sentence from Germany into English, and do not alter its meaning:

Germany: Die Premierminister Indiens und Japans trafen sich in Tokio.

English: India and Japan prime ministers meet in Tokyo

Table 6: An exemplar instruction of the LLM augmented by cross-lingual
sentence retrieval. The text after “English:” is the reference, which will not
appear in the instruction.

Please faithfully translate the following sentence from Germany into English,
and do not alter its meaning:
Germany: Die Premierminister Indiens und Japans trafen sich in Tokio.
English: India and Japan prime ministers meet in Tokyo

Table 7: An exemplar instruction of the vanilla LLM. The text after “English:”
is the reference, which is just for illustration and will not appear in the
instruction.

##  Appendix A Cross-lingual Contextualized Phrase Retrieval

###  A.1 Human Annotation

We asked human annotators to label 200 high-quality phrase pairs as the golden
truth data for each of the language pairs, i.e., De$\Rightarrow$En,
Ro$\Rightarrow$En, and Cs$\Rightarrow$En. Hiring human annotators to annotate
those bilingual data is expensive and time consuming. However, fortunately,
some bilingual datasets already provide the human-annotated word alignment
Ghader and Monz (2017); Mihalcea and Pedersen (2003); Mareček (2011).
Therefore, in our task, we only ask the human annotators, three authors of our
work, to annotate the English side of the data, i.e., whether a English span
is a high-quality and meaningful phrase or not.

More concretely, we first use heuristic rules as discussed in Section 6.1 to
collect an initial set of cross-lingual phrase pairs. For each phrase pair, we
ask the human annotator to answer three questions:

  1. 1.

If the phrase is a single word, whether it is informative in the context?

  2. 2.

If the phrase has multiple words, whether the semantics of this phrase is
complete and informative? For instance, “local authorities and large” is not a
phrase with complete semantics. In addition, “Of course” has complete
semantics but is not informative.

If the answer of any questions is true, then we add this phrase to a pool of
high-quality phrases. For those phrases, overlaps are allowed. We finally
randomly sampled 200 annotated phrases form the pool for each language.

##  Appendix B Machine Translation

###  B.1 Translation Instructions

The instructions for the vanilla large language model (LLM), e.g.,  Llama-2,
and the LLM augmented by cross-lingual sentence retrieval are shown in Table 6
and 7, respectively.

###  B.2 More Results

More results on evaluated by COMET Rei et al. (2020) are shown in Table 5,
which is consistent with the results evaluated by BERTScore Zhang et al.
(2019).