# Semileptonic decays of doubly charmed baryons with
$\Xi_{c}-\Xi_{c}^{\prime}$ mixing

Chao-Qiang Geng, Xian-Nan Jin, Chia-Wei Liu, Xiao Yu and Ao-Wen Zhou
School of Fundamental Physics and Mathematical Sciences, Hangzhou Institute
for Advanced Study, UCAS, Hangzhou 310024, China
University of Chinese Academy of Sciences, 100190 Beijing, China

###### Abstract

We study the $\Xi_{c}-\Xi_{c}^{\prime}$ mixing effects in the semileptonic
decays the doubly charm baryons of $\Xi_{cc}$. We focus on the ratio of ${\cal
R}(\theta_{c})\equiv{\cal B}(\Xi_{cc}\to\Xi_{c}^{\prime}e^{+}\nu_{e})/{\cal
B}(\Xi_{cc}\to\Xi_{c}e^{+}\nu_{e})$ and find that $({\cal R}(\theta_{0}),{\cal
R}(-\theta_{0}))=(0.46\pm 0.01,7.33\pm 0.23)$ with $\theta_{0}=0.137\pi$,
which are in sharp contrast to ${\cal R}(0)=2.15\pm 0.11$ without the mixing.
The ratio is enhanced (suppressed) by a factor of four for a negative
(positive) $\theta_{c}$. In addition, the polarization asymmetries of
$\Xi_{c}^{(\prime)}$ are found to be $\alpha(-\theta_{0})=0.32~{}(-0.76)$ and
$\alpha(\theta_{0})=-0.82~{}(-0.38)$. As ${\cal R}$ and $\alpha$ are highly
sensitive to $\theta_{c}$ and unaffected by the $W$-exchange contributions,
they provide excellent opportunities to determine $\theta_{c}$ in the ongoing
experiments.

The baryonic semileptonic decays provide excellent opportunities to examine
the baryon wave functions, which are decomposed into four parts: color,
spatial, spin and flavor. The color part is trivial in three quark systems,
while the spatial part is usually assumed to be symmetric in the low-lying
baryons. In the low-lying spin-half baryons, the spin and flavor are known to
be entangled, and together they are named as the spin-flavor configurations.
They are independent of the quark model but nontrivial when the three quarks
possess different flavors.

Recently, the mixing of $\Xi_{c}$ and $\Xi_{c}^{\prime}$, given by

$\displaystyle\left(\begin{array}[]{c}\Xi_{c}\\\
\Xi_{c}^{\prime}\end{array}\right)=\left(\begin{array}[]{cc}\cos\theta_{c}&\sin\theta_{c}\\\
-\sin\theta_{c}&\cos\theta_{c}\end{array}\right)\left(\begin{array}[]{c}\Xi_{c}^{\bm{\overline{3}}}\\\
\Xi_{c}^{\bm{6}}\end{array}\right)\,,~{}~{}~{}|\theta_{c}|=\theta_{0}=0.137(5)\pi\,,$
(7)

has been proposed Mixing , where $\theta_{c}$ is determined from the mass
relations Jenkins:1996rr  to the precision of $m_{s}^{2}/(m_{c}N^{2}_{c})$
with $m_{s,c}$ the quark masses and $N_{c}$ the color number. In Eq. (7),
$\Xi_{c}^{(\prime)}$ represent the physical baryons, and $\overline{{\bf 3}}$
and ${\bf 6}$ are the representations under the $SU(3)$ flavor symmetry,
corresponding to the light quark spins are in antisymmetric and symmetric,
respectively. With the mixing, we are able to explain the large $SU(3)_{F}$
violation in the singly charm baryon decays Belle:2021crz ; ALICE:2021bli .
Though the mixing effects are second order of $\theta_{c}$, the effects are
still sizable. However, it forbids us to fix the sign of $\theta_{c}$ from the
singly charmed baryon semileptonic decays. Therefore, the sign of $\theta_{c}$
remains as an open question.

For the nonleptonic decays, attempts Liu:2022igi ; Ke:2022gxm  have been made
in $\Xi_{cc}$, pounding on the ratio measured at LHCb LHCb:2022rpd

${\cal R}^{\text{non}}=\frac{{\cal
B}(\Xi_{cc}^{++}\to\Xi_{c}^{\prime+}\pi^{+})}{{\cal
B}(\Xi_{cc}^{++}\to\Xi_{c}^{+}\pi^{+})}=1.42\pm 0.17\pm 0.10\,,$ (8)

where the first and second uncertainties are statistical and systematic,
respectively. It disagrees with all predictions in the literature before the
measurements Wang:2017mqp ; Cheng:2020wmk ; Sharma:2017txj ; Gerasimov:2019jwp
; Gutsche:2018msz ; Ke:2019lcf ; Shi:2019hbf ; Ivanov:2020xmw , where ideal
configurations were assumed. The puzzle was partly resolved by the mixing,
first founded in Ref. Ke:2022gxm . As both the denominator and numerator are
affected by the mixing, it has a sizable impact on the ratio. However, the
factorization ansatz has been assumed with $\theta_{c}$ treated as a free
parameter in Ref. Ke:2022gxm . To overcome the weak point, two of the authors
of this work (Geng and Liu) have included the nonfactorizable contributions
Liu:2022igi , and found that the mixing angle in Eq. (7) can explain ${\cal
R}^{\text{non}}$ based on the methodology developed in Ref. Cheng:2020wmk .
The mixing effects modify ${\cal R}^{\text{non}}$ largely from 6.74 to 1.45
with a negative angle of $\theta_{c}$. We note that the same methodology has
shown to be suitable in the heavy-flavor-conserving and $\Lambda_{c}^{+}$
nonleptonic decays Cheng:2018hwl ; Cheng:2022jbr .

Nevertheless, due to the hadron uncertainties, the sign of $\theta_{c}$ still
requires a further confirmation, which is the main purpose of this work. To
this end, we revisit our previous study in Ref. Geng:2022uyy , where
$\theta_{c}=0$ was assumed. The formalism we adopt here is the same as Ref.
Geng:2022uyy  except the spin-flavor configurations given in Eq. (7). The
modifications affect exclusively in the spin-flavor overlappings, defined as
Liu:2022pdk

$\displaystyle
N^{(\prime)}_{\text{flip}}=\Big{\langle}\Xi^{(\prime)}_{c},J_{z}=\frac{1}{2}\Big{|}s^{\dagger}\sigma_{z}c\Big{|}\Xi_{cc},J_{z}=\frac{1}{2}\Big{\rangle}\,,$
$\displaystyle
N_{\text{unflip}}^{(\prime)}=\Big{\langle}\Xi^{(\prime)}_{c},J_{z}=\frac{1}{2}\Big{|}s^{\dagger}c\Big{|}\Xi_{cc},J_{z}=\frac{1}{2}\Big{\rangle}\,,$
(9)

where
$s^{\dagger}\sigma_{z}c=s^{\dagger}_{\uparrow}c_{\uparrow}-s^{\dagger}_{\downarrow}c_{\downarrow}$
and
$s^{\dagger}c=s^{\dagger}_{\uparrow}c_{\uparrow}+s^{\dagger}_{\downarrow}c_{\downarrow}\,,$
while the baryon states in Eq. (Semileptonic decays of doubly charmed baryons
with $\Xi_{c}-\Xi_{c}^{\prime}$ mixing) are in the nonrelativistic constituent
quark limit. We adopt the homogeneous bag model as it provides simple spatial
wave functions, leaving clarity to the spin-flavor configurations. Plugging
Eq. (7) into Eq. (Semileptonic decays of doubly charmed baryons with
$\Xi_{c}-\Xi_{c}^{\prime}$ mixing), we find that

$\displaystyle\left(\begin{array}[]{c}N_{\text{(un)flip}}\\\
N_{\text{(un)flip}}^{\prime}\end{array}\right)=\left(\begin{array}[]{cc}\cos\theta_{c}&\sin\theta_{c}\\\
-\sin\theta_{c}&\cos\theta_{c}\end{array}\right)\left(\begin{array}[]{c}N_{\text{(un)flip}}^{\overline{\bf
3}}\\\ N_{\text{(un)flip}}^{\bf 6}\end{array}\right),$ (16)

resulting in that

$\displaystyle N_{\text{unflip}}^{\overline{{\bf 3}},{\bf
6}}=\left(\frac{\sqrt{6}}{2},\frac{\sqrt{6}}{6}\right)\,,~{}N_{\text{flip}}^{\overline{{\bf
3}},{\bf 6}}=\left(\frac{\sqrt{2}}{2},\frac{5\sqrt{6}}{6}\right)\,.$ (17)

In the following, we omit the small uncertainty of $\theta_{0}$ and simply
take $\theta_{0}=0.137\pi$ in Eq. (7).

In analogy to ${\cal R}^{non}$, we define the semileptonic version of the
ratio

$\mathcal{R}(\theta_{c})=\frac{\Gamma^{\prime}}{\Gamma}=\frac{\Gamma(\Xi_{cc}\to\Xi^{\prime}_{c}e^{+}\nu_{e})}{\Gamma(\Xi_{cc}\to\Xi_{c}e^{+}\nu_{e})}\,.$
(18)

Furthermore, the polarization asymmetries of $\Xi_{c}^{(\prime)}$ are defined
as

$\alpha^{(\prime)}=\frac{\Gamma^{(\prime)}(\lambda=1/2)-\Gamma^{(\prime)}(\lambda=-1/2)}{\Gamma^{(\prime)}(\lambda=1/2)+\Gamma^{(\prime)}(\lambda=-1/2)}\,,$
(19)

where $\lambda$ are the helicities of $\Xi_{c}^{(\prime)}$.

The dependencies of $\mathcal{R}$ and $\alpha^{(\prime)}$ on $\theta_{c}$ are
depicted in Fig. 1, and their explicit values for
$\theta_{c}\in\\{\pm\pi/4,\pm\theta_{0},0\\}$ are presented in TABLE 1. The
uncertainties of $\alpha^{(\prime)}$ are tiny and can be neglected in the
considered precision. It shows that the physical quantities depend heavily on
$\theta_{c}$. Explicitly, ${\cal R}$ varies from $0.45$ to $7.56$ in the range
of $\theta_{c}=\pm\theta_{0}$, and the sign of $\alpha$ is flipped for
$\theta_{c}=-\theta_{0}$. The large deviations can be traced back to the
interference between $N_{\text{(un)flip}}^{\overline{{\bf 3}},{\bf 6}}$. In
particular, a positive (negative) $\theta_{c}$ leads to constructive
(destructive) and destructive (constructive) interference in
$N_{\text{(un)flip}}$ and $N_{\text{(un)flip}}^{\prime}$, respectively. With
this feature, we can identify the sign of $\theta_{c}$ once the experimental
value of ${\cal R}$ is given. It is clear that a negative $\alpha$ is a
smoking gun of $\theta_{c}\neq 0$.

Table 1: $\Gamma^{(\prime)}$ and $\alpha^{(\prime)}$ in units of $10^{-5}$ eV and $\%$, respectively. $\theta_{c}$ | $-\pi/4$ | $-\theta_{0}$ | $0$ | $\theta_{0}$ | $\pi/4$
---|---|---|---|---|---
$\Gamma^{\prime}$ | $12.7(13)$ | $13.5(14)$ | $10.9(8)$ | $5.64(58)$ | $2.34(24)$
$\Gamma$ | $3.04(38)$ | $1.85(25)$ | $5.11(64)$ | $11.8(15)$ | $16.2(21)$
${\cal R}$ | $4.19(10)$ | $7.33(23)$ | $2.15(11)$ | $0.46(1)$ | $0.14(0)$
$\alpha$ | $15$ | $32$ | $-68$ | $-82$ | $-81$
$\alpha^{\prime}$ | $-80$ | $-76$ | $-65$ | $-38$ | $-20$

Figure 1:  ${\cal R}$ and $\alpha^{(\prime)}$ versus $\theta_{c}$.

We note that with $\theta_{c}=0$, ${\cal R}$ are found to be $2.39,$ $1.67,$
and $1.64$ from the MIT bag model Perez-Marcial:1989sch , heavy quark spin
symmetry Albertus:2011xz , and light-front quark model Hu:2020mxk ,
respectively. We emphasize that the future experimental value of ${\cal R}>7$
$({\cal R}<0.5)$ will be a clear evidence of the $\Xi_{c}-\Xi_{c}^{\prime}$
mixing with negative (positive) $\theta_{c}$.

In sum, we have studied the $\Xi_{c}-\Xi_{c}^{\prime}$ mixing effects in
$\Xi_{cc}\to\Xi_{c}^{(\prime)}e^{+}\nu_{e}$. We have found that ${\cal
R}=0.46\pm 0.01$ $(7.33\pm 0.23)$ for $\theta_{c}=(-)\theta_{0}$, which is
about four times smaller (larger) than ${\cal R}=2.15\pm 0.11$ without the
mixing of $\theta_{c}=0$. Notably, with $\theta_{c}=-\theta_{0}$ the
polarization asymmetry of $\Xi_{c}$ flips sign, given as $\alpha=0.32$. We
recommend the future experiments to measure ${\cal R}$ and
$\alpha^{(\prime)}$, since $\alpha<0$ and ${\cal R}>7$ are smoking guns of the
$\Xi_{c}-\Xi_{c}^{\prime}$ mixing with $\theta_{c}<0$, whereas ${\cal R}<0.5$
indicates that $\theta_{c}>0$. As the decays are highly sensitive to
$\theta_{c}$ and unaffected by the $W$-exchange contributions, they provide
the most now known reliable ways to probe $\theta_{c}$.

###### Acknowledgements.

This work is supported in part by the National Key Research and Development
Program of China under Grant No. 2020YFC2201501 and the National Natural
Science Foundation of China (NSFC) under Grant No. 12147103.

## References

  * (1) C. Q. Geng, X. N. Jin and C. W. Liu, arXiv:2210.07211 [hep-ph].
  * (2) E. E. Jenkins, Phys. Rev. D 54, 4515 (1996); E. E. Jenkins, Phys. Rev. D 55, 10 (1997); E. E. Jenkins, Phys. Rev. D 77, 034012 (2008).
  * (3) Y. B. Li et al. [Belle], Phys. Rev. Lett. 127, 121803 (2021).
  * (4) S. Acharya et al. [ALICE], Phys. Rev. Lett. 127, 272001 (2021).
  * (5) H. W. Ke and X. Q. Li, Phys. Rev. D 105, 096011 (2022).
  * (6) C. W. Liu and C. Q. Geng, arXiv:2211.12960 [hep-ph].
  * (7) R. Aaij et al. [LHCb], JHEP 05, 038 (2022).
  * (8) W. Wang, F. S. Yu and Z. X. Zhao, Eur. Phys. J. C 77, 781 (2017).
  * (9) N. Sharma and R. Dhir, Phys. Rev. D 96, 113006 (2017).
  * (10) A. S. Gerasimov and A. V. Luchinsky, Phys. Rev. D 100, 073015 (2019).
  * (11) T. Gutsche, M. A. Ivanov, J. G. Körner, V. E. Lyubovitskij and Z. Tyulemissov, Phys. Rev. D 99, 056013 (2019).
  * (12) H. Y. Cheng, G. Meng, F. Xu and J. Zou, Phys. Rev. D 101, 034034 (2020).
  * (13) Y. J. Shi, W. Wang and Z. X. Zhao, Eur. Phys. J. C 80, 568 (2020).
  * (14) M. A. Ivanov, J. G. Körner and V. E. Lyubovitskij, Phys. Part. Nucl. 51, 678-685 (2020).
  * (15) H. W. Ke, F. Lu, X. H. Liu and X. Q. Li, Eur. Phys. J. C 80, 140 (2020).
  * (16) H. Y. Cheng, X. W. Kang and F. Xu, Phys. Rev. D 97, 074028 (2018).
  * (17) H. Y. Cheng, C. W. Liu and F. Xu, Phys. Rev. D 106, 093005 (2022).
  * (18) C. Q. Geng, C. W. Liu, A. Zhou and X. Yu, arXiv:2211.04372 [hep-ph].
  * (19) C. W. Liu and C. Q. Geng, arXiv:2205.08158 [hep-ph].
  * (20) C. Albertus, E. Hernández and J. Nieves, Phys. Lett. B 704, (2011).
  * (21) R. Perez-Marcial, R. Huerta, A. Garcia and M. Avila-Aoki, Phys. Rev. D 40, 2955 (1989) [erratum: Phys. Rev. D 44, 2203 (1991)].
  * (22) C. Albertus, E. Hernández and J. Nieves, Phys. Lett. B 704, 499 (2011).
  * (23) X. H. Hu, R. H. Li and Z. P. Xing, Eur. Phys. J. C 80, 320 (2020).


# Empowering Dual-Encoder with Query Generator for
Cross-Lingual Dense Retrieval

Houxing Ren1 Linjun Shou2 Ning Wu2 Ming Gong2 Daxin Jiang2
1School of Computer Science and Engineering, Beihang University
2Microsoft STC Asia
<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done
during internship at Microsoft STCA. Corresponding author.

###### Abstract

In monolingual dense retrieval, lots of works focus on how to distill
knowledge from cross-encoder re-ranker to dual-encoder retriever and these
methods achieve better performance due to the effectiveness of cross-encoder
re-ranker. However, we find that the performance of the cross-encoder re-
ranker is heavily influenced by the number of training samples and the quality
of negative samples, which is hard to obtain in the cross-lingual setting. In
this paper, we propose to use a query generator as the teacher in the cross-
lingual setting, which is less dependent on enough training samples and high-
quality negative samples. In addition to traditional knowledge distillation,
we further propose a novel enhancement method, which uses the query generator
to help the dual-encoder align queries from different languages, but does not
need any additional parallel sentences. The experimental results show that our
method outperforms the state-of-the-art methods on two benchmark datasets.

##  1 Introduction

Information Retrieval (IR) aims to retrieve pieces of evidence for a given
query. Traditional methods mainly use sparse retrieval systems such as BM25
Robertson and Zaragoza (2009), which depend on keyword matching between
queries and passages. With the development of large-scale pre-trained language
models (PLMs) Vaswani et al. (2017); Devlin et al. (2019) such as BERT, dense
retrieval methods Lee et al. (2019); Karpukhin et al. (2020) show quite
effective performance. These methods usually employed a dual-encoder
architecture to encode both queries and passages into dense embeddings and
then perform approximate nearest neighbor searching Johnson et al. (2021).

Recently, leveraging a cross-encoder re-ranker as the teacher model to distill
knowledge to a dual-encoder has shown quite effective to boost the dual-
encoder performance. Specifically, these methods first train a warm-up dual-
encoder and a warm-up cross-encoder. Then, they perform knowledge distillation
from the cross-encoder to the dual-encoder by KL-Divergence or specially
designed methods. For example, RocketQAv2 Qu et al. (2021) proposed dynamic
distillation, and AR2 Zhang et al. (2021) proposed adversarial training.

(a) BM25. (b) DPR.

Figure 1: The performance of cross-encoder and query generator when varying
the number of training samples and retrievers. We use BM25 and DPR as
retrievers, respectively. For the cross-encoder (BERT-Large), we use retrieved
top-100 passages which do not contain the answer as negative and contrastive
loss for training. For the query generator (T5-Base), we firstly train it with
the query generation task and then fine-tune the model with the same setting
as BERT-Large. The reported performance is the top-5 score of re-ranked top
500 passages on the NQ test set.

However, there are two major problems when scaling the method to the cross-
lingual dense retrieval setting. Firstly, the cross-encoder typically requires
large amounts of training data and high-quality negative samples due to the
gap between pre-training (token-level task) and fine-tuning (sentence-level
task), which are usually not satisfied in the cross-lingual setting Asai et
al. (2021a). Due to expensive labeling and lack of annotators in global
languages, especially low-resource languages, the training data in cross-
lingual are quite limited. Then with the limited training data, the dual-
encoder is not good enough to provide high-quality negative samples to
facilitate the cross-encoder. Secondly, the cross-lingual gaps between
different languages have a detrimental effect on the performance of cross-
lingual models. Although some cross-lingual pre-training methods such as
InfoXLM Chi et al. (2021) and LaBSE Feng et al. (2022) have put lots of effort
into this aspect by leveraging parallel corpus for better alignment between
different languages, these parallel data are usually expensive to obtain and
the language alignment could be damaged in the fine-tuning stage if without
any constraint.

To solve these problems, we propose to employ a query generator in the cross-
lingual setting, which uses the likelihood of a query against a passage to
measure the relevance. On the one hand, the query generator can utilize pre-
training knowledge with small training data in fine-tuning stage, because both
of its pre-training and fine-tuning have a consistent generative objective. On
the other hand, the query generation task is defined over all tokens from the
query rather than just the _[CLS] token_ in the cross-encoder, which has been
demonstrated to be a more efficient training paradigm Clark et al. (2020). As
shown in Figure 1, with the number of training samples dropping, the
performance of BERT-Large drops more sharply than T5-Base. Besides, the query
generator is less sensitive to high-quality negative samples. As we can see,
using BM25 as the retriever to mine negative samples for re-ranker training,
the gap between cross-encoder and query generator is smaller than the gap
using DPR as the retriever. Finally, the query generator can provide more
training data by generation, which is precious in the cross-lingual setting.
To sum up, the query generator is more effective than the cross-encoder in the
cross-lingual setting.

Based on these findings, we propose a novel method, namely QuiCK, which stands
Query generator improved dual-encoder by Cross-lingual Knowledge distillation.
Firstly, at the passage level, we employ a query generator as the teacher to
distill the relevant score between a query and a passage into the dual-
encoder. Secondly, at the language level, we use the query generator to
generate synonymous queries in other languages for each training sample and
align their retrieved results by KL-Divergence. Considering the noise in the
generated queries, we further propose a scheduled sampling method to achieve
better performance.

The contributions of this paper are as follows:

  * •

We propose a cross-lingual query generator as a teacher model to empower the
cross-lingual dense retrieval model and a novel iterative training approach is
leveraged for the joint optimizations of these two models.

  * •

On top of the cross-lingual query generator, a novel cost-effective alignment
method is further designed to boost the dense retrieval performance in low-
resource languages, which does not require any additional expensive parallel
corpus.

  * •

Extensive experiments on two public cross-lingual retrieval datasets
demonstrate the effectiveness of the proposed method.

##  2 Related Work

Retrieval. Retrieval aims to search relevant passages from a large corpus for
a given query. Traditionally, researchers use bag-of-words (BOW) based methods
such as TF-IDF and BM25 Robertson and Zaragoza (2009). These methods use a
sparse vector to represent the text, so we call them sparse retrievers.
Recently, some studies use neural networks to improve the sparse retriever
such as DocTQuery Nogueira et al. (2019) and DeepCT Dai and Callan (2020).

In contrast to sparse retrievers, dense retrievers usually employ a dual-
encoder to encode both queries and passages into dense vectors whose lengths
are much less than sparse vectors. These methods mainly focus on two aspects:
pre-training Lee et al. (2019); Guu et al. (2020); Lu et al. (2021); Gao and
Callan (2021a, b); Zhou et al. (2022) and fine-tuning methods, including
negative sampling Karpukhin et al. (2020); Luan et al. (2021); Xiong et al.
(2021); Zhan et al. (2021) and multi-view representations Khattab and Zaharia
(2020); Humeau et al. (2020); Tang et al. (2021); Zhang et al. (2022). Another
fine-tuning method is jointly training the dual-encoder with a cross-encoder.
For example, RDR Yang and Seo (2020) and FID-KD Izacard and Grave (2021)
distill knowledge from a reader to the dual-encoder; RocketQA Qu et al.
(2021), PAIR Ren et al. (2021a), RocketQAv2 Ren et al. (2021b), and AR2 Zhang
et al. (2021) jointly train the dual-encoder with a cross-encoder to achieve
better performance.

(a) Dual-Encoder. (b) Cross-Encoder. (c) Query Generator.

Figure 2: Overview of different model architectures designed for retrieval or
re-ranking.

Recently, with the development of cross-lingual pre-trained models Conneau et
al. (2020), researchers pay more attention to cross-lingual dense retrieval
Asai et al. (2021a); Longpre et al. (2020). For example, CORA Asai et al.
(2021b) leverages a generator to help mine retrieval training data, Sentri
Sorokin et al. (2022) proposes a single encoder and self-training, and DR.DECR
Li et al. (2021) uses parallel queries and sentences to perform cross-lingual
knowledge distillation.

Re-ranking. Re-ranking aims to reorder the retrieved passages as the relevant
scores. Due to the small number of retrieved passages, re-ranking usually
employs high-latency methods to obtain better performance, _e.g.,_ cross-
encoder. Traditionally, the re-ranking task is heavily driven by manual
feature engineering Guo et al. (2016); Hui et al. (2018). With the development
of pre-trained language models (_e.g.,_ BERT), researchers use the pre-trained
models to perform re-ranking tasks Nogueira and Cho (2019); Li et al. (2020).
In addition to cross-encoder, researchers also try to apply generator to re-
ranking. For example, monoT5 Nogueira et al. (2020) proposes a prompt-based
method to re-rank passages with T5 Raffel et al. (2020) and other studies dos
Santos et al. (2020); Zhuang et al. (2021); Lesota et al. (2021) propose to
use the log-likelihood of the query against the passage as the relevance to
perform the re-ranking task.

Recently, with the size of pre-trained models scaling up, the generative
models show competitive zero-shot and few-shot ability. Researchers start to
apply large generative models to zero-shot and few-shot re-ranking. For
example, SGPT Muennighoff (2022) and UPR Sachan et al. (2022) propose to use
generative models to perform zero-shot re-ranking. P3 Ranker Hu et al. (2022)
demonstrates that generative models achieve better performance in the few-shot
setting. Note that all of these works are concurrent to our work. Instead of
using a query generator as a re-ranker only, we propose to leverage the query
generator as a teacher model to enhance the performance of the cross-lingual
dual-encoder. In addition to the traditional knowledge distillation, we
further propose a novel cost-effective alignment method to boost the dense
retrieval performance in low-resource languages.

##  3 Preliminaries

In this section, we give a brief review of dense retrieval and re-ranking. The
overviews of all methods are presented in Figure 2.

Dual-Encoder. Given a query $q$ and a large corpus $C$, the retrieval task
aims to find the relevant passages for the query from a large corpus. Usually,
a dense retrieval model employs two dense encoders (_e.g.,_ BERT)
$E_{Q}(\cdot)$ and $E_{P}(\cdot)$. They encode queries and passages into dense
embeddings, respectively. Then, the model uses a similarity function, often
dot-product, to perform retrieval:

$f_{de}(q,p)=E_{Q}(q)\cdot E_{P}(p),$ (1)

where $q$ and $p$ denote the query and the passage, respectively. During the
inference stage, we apply the passage encoder $E_{P}(\cdot)$ to all the
passages and index them using FAISS Johnson et al. (2021) which is an
extremely efficient, open-source library for similarity search. Then given a
query $q$, we derive its embedding by $\bm{v}_{q}=E_{Q}(q)$ and retrieve the
top $k$ passages with embeddings closest to $\bm{v}_{q}$.

Cross-Encoder Re-ranker. Given a query $q$ and top $k$ retrieved passages $C$,
the re-ranking task aims to reorder the passages as the relevant scores. Due
to the limited size of the corpus, the re-ranking task usually employs a
cross-encoder to perform interaction between words across queries and passages
at the same time. These methods also introduce a special token _[SEP]_ to
separate q and p, and then the hidden state of the _[CLS] token_ from the
cross-encoder is fed into a fully-connected layer to output the relevant
score:

$f_{ce}(q,p)=\bm{W}\times E_{C}(q||p)+b,$ (2)

where “$||$” denotes concatenation with the _[SEP] token_. During the
inference stage, we apply the cross-encoder $E_{C}(\cdot)$ to all <q, p> pair
and reorder the passages by the scores.

Query Generator Re-ranker. Similar to cross-encoder re-ranker, query generator
re-ranker also aims to reorder the passages as the relevant scores. For the
query generator re-ranker, we use the log-likelyhood of the query against the
passage to measure the relevance:

$f_{qg}(q,p)=\log P(q|p)=\sum_{t=0}\log P(q_{t}|q_{<t},p),$ (3)

where $q_{<t}$ denotes the previous tokens before $q_{t}$. The rest of
settings are the same as the cross-encoder re-ranker and are omitted here.

Training. The goal of retrieval and re-ranking is to enlarge the relevant
score between the query and the relevant passages (_a.k.a.,_ positive
passages) and lessen the relevant score between the query and the irrelevant
passages (_a.k.a.,_ negative passages). Let
$\\{q_{i},p^{+}_{i},p^{-}_{i,0},p^{-}_{i,1},\dots,p^{-}_{i,n}\\}$ be the
$i$-th training sample. It consists of a query, a positive passage, and $n$
negative passages. Then we can employ the contrastive loss function, called
InfoNCE van den Oord et al. (2018), to optimize the model:

$\mathcal{L}_{R}=-\log\frac{e^{f(q_{i},p^{+}_{i})}}{e^{f(q_{i},p^{+}_{i})}+\sum_{j=0}^{n}e^{f(q_{i},p^{-}_{i,j})}},$
(4)

where $f$ denotes the similarity function, _e.g.,_ $f_{de}$ in Eq. (1),
$f_{ce}$ in Eq. (2), or $f_{qg}$ in Eq. (3).

Cross-lingual Retrieval. In the cross-lingual information retrieval task,
passages and queries are in different languages. In this paper, we consider
the passages are in English and the queries are in non-English languages. A
sample consists of three components: a query in a non-English language, a
positive passage in English, and a span answer in English. Given a non-English
query, the task aims to retrieve relevant passages in English to answer the
query. If a retrieved passage contains the given span answer, it is regarded
as a positive passage, otherwise, it is a negative passage.

Figure 3: Overview of the proposed QuiCK.

##  4 Methodology

In this section, we present the proposed QuiCK. The overview of the proposed
method is presented in Figure 3. We start with the training of the query
generator, then present how to perform distillation and alignment training for
the dual-encoder, and we finally discuss the entire training process.

###  4.1 Training of Query Generator

In our method, we employ mT5 Xue et al. (2021) as the query generator. The
query generator has two roles: teacher and generator. As a teacher, it aims to
better re-rank the candidate passages with relevance and distill the knowledge
to the dual-encoder. As a generator, it aims to generate synonymous queries in
different languages.

Input Format. As we employ mT5, we design a prompt template for input
sentences. Considering that most passages are long, we propose introducing the
span answer as input to encourage the generator to focus on the same segment
and generate parallel queries in different languages. As a result, we use
_“generate [language] query: answer: [span answer] content: [content]”_ as the
template. For a specific sample, we fill the three placeholders with the
language of the target query, the span answer, and the passage content,
respectively.

Training. Considering the two roles of the query generator, the entire
training process for the query generator contains two stages: query generation
training and re-ranking training.

Firstly, we train the generator with the generation task, which takes the
positive passage as input and aims to generate the query. The task can be
formulated as maximizing the conditional probability:

$\begin{aligned} \hat{q}&=\mathop{\arg\max}\limits_{q}P(q|p,a)\\\
&=\mathop{\arg\max}\limits_{q}\prod_{t=0}P(q_{t}|p,a,q_{<t})\end{aligned},$
(5)

where $q_{t}$ is the $t$-th token of the generated query, $a$ denotes the span
answer, and $q_{<t}$ represents the previous decoded tokens. Then we can
employ cross-entropy loss to optimize the model:

$\mathcal{L}_{QG}=\frac{1}{T}\sum_{t=0}^{T}-\log P(q_{t}|p,a,q_{<t}),$ (6)

where $T$ denotes the number of the query tokens.

Secondly, we train the generator with the re-ranking task, which takes a query
and a passage as input and outputs the relevant score of the two sentences.
The detailed training process is introduced in Section 3 and is omitted here.

###  4.2 Distillation for Dual-Encoder

We then present how to distill knowledge from query generator to dual-encoder.
Similar to previous methods Ren et al. (2021b); Zhang et al. (2021), we employ
KL-Divergence to perform distillation. Formally, given a query $q$ and a
candidate passage set $C_{q}=\\{p_{i}\\}_{1\leq i\leq n}$ which is retrieved
by the dual-encoder, we compute relevant scores by query generator and dual-
encoder, respectively. After that, we normalize the scores by softmax and
compute the KL-Divergence as the loss:

$\displaystyle s_{qg}(q,p)$
$\displaystyle=\frac{\exp(f_{qg}(q,p))}{\sum_{p^{\prime}\in
C_{q}}\exp(f_{qg}(q,p^{\prime}))},$ (7) $\displaystyle s_{de}(q,p)$
$\displaystyle=\frac{\exp(f_{de}(q,p))}{\sum_{p^{\prime}\in
C_{q}}\exp(f_{de}(q,p^{\prime}))},$ $\displaystyle\mathcal{L}_{D}$
$\displaystyle=\sum_{p\in C_{q}}s_{qg}(q,p)\frac{s_{qg}(q,p)}{s_{de}(q,p)},$

where $f_{qg}$ and $f_{de}$ denote the relevant score given by the query
generator and the dual-encoder which are presented in Section 3.

###  4.3 Alignment for Dual-Encoder

Alignment is a common topic in the cross-lingual setting, which can help the
model better handle sentences in different languages. Previous works Zheng et
al. (2021); Yang et al. (2022) usually use parallel data or translated data to
perform alignment training among different languages. Here, we propose a novel
method to align queries in different languages for cross-lingual retrieval,
which does not need any parallel data. The core idea of our method is to
leverage the query generator to generate synonymous queries in other languages
to form parallel cases.

Generation. For each case in the training set, we generate a query in each
target language (_a.k.a.,_ if there are seven target languages, we generate
seven queries for the case). Then we use the confidence of the generator to
filter the generated queries. Specially, we set filter thresholds to accept
50% of generated queries.

Scheduled Sampling. In this work, we select a generated query to form a pair-
wise case with the source query. Considering the semantics of generated
queries, we carefully design a scheduled sampling method to replace the random
sampling. For a generated query $q^{\prime}$, we first use the dual-encoder to
retrieve passages for the source query $q$ and generated query $q^{\prime}$,
respectively, namely $C_{q}$ and $C_{q}^{\prime}$. Then we calculate a
coefficient for the generated query $q^{\prime}$ as

$\displaystyle c^{\prime}$ $\displaystyle=\frac{|C_{q}\cap
C_{q}^{\prime}|}{\max(|C_{q}|,|C_{q}^{\prime}|)},$ (8) $\displaystyle
c^{\prime}$ $\displaystyle=\begin{cases}c^{\prime}&\text{ if }c^{\prime}\geq
T,\\\ 0&\text{ if }c^{\prime}<T,\end{cases}$

where threshold $T$ is a hyper-parameter and $|\cdot|$ denotes the size of the
set. The basic idea is that the larger the union of retrieved passages, the
more likely the queries are to be synonymous. When sampling the generated
query, we first calculate coefficients
$\\{c_{1}^{\prime},\dots,c_{m}^{\prime}\\}$ for all generated queries
$\\{q_{1}^{\prime},\dots,q_{m}^{\prime}\\}$, then normalize them as the final
sampling probability $p$:

$p_{i}=\frac{c_{i}^{\prime}}{\sum_{j=0}^{m}c_{i}^{\prime}},$ (9)

where $m$ denotes the number of generated queries. During the training stage,
for each training case, we sample a generated query to form the pair-case with
the source query $q$ based on the probabilities.

Alignment Training. After sampling a generated query, we present the how to
align the source query and the generated query. Different to previous works
Zheng et al. (2021), we employ asymmetric KL-Divergence rather than symmetric
KL-Divergence due to the different quality of the source query and the
generated query:

$\mathcal{L}_{A}=\sum_{p\in C_{q}\cup
C_{q}^{\prime}}c^{\prime}s_{de}(q,p)\frac{s_{de}(q,p)}{s_{de}(q^{\prime},p)},$
(10)

where $q$ denotes the query, $C_{q}$ denotes the set of retrieved passages,
superscript “$\prime$” denotes the generated case, and $c^{\prime}$ is the
coefficient of the generated query. Note that $s_{de}$ in Eq. (10) are
normalized across $C_{q}\cup C_{q}^{\prime}$ instead of $C_{q}$ or
$C_{q}^{\prime}$ in Eq. (7).

###  4.4 Training of Dual-Encoder

As shown in Figure 3, we combine the distillation loss and the alignment loss
as final loss:

$\mathcal{L}=\mathcal{L}_{D}+\mathcal{L}_{D}^{\prime}+\alpha\times\mathcal{L}_{A},$
(11)

where $\mathcal{L}_{D}$ denotes the distillation loss for the source queries,
$\mathcal{L}_{D}^{\prime}$ denotes the distillation loss for the generated
queries, $\mathcal{L}_{A}$ denotes the alignment loss, and $\alpha$ is a
hyper-parameter to balance the loss.

Based on the training method of dual-encoder and query generator, we conduct
an iterative procedure to improve the performance. We present the entire
training procedure in Algorithm 1.

Input: Dual-Encoder $R$, Query Generator $G$, Corpus $C$, and Training Set
$D$.

1

2Initialize $R$ and $G$ with pre-trained model;

3 Train the warm-up $R$ with Eq. (4) on $D$;

4 Train the warm-up $G$ with Eq. (6) on $D$;

5 Generate queries for each sample in $D$;

6 Build ANN index for $R$;

7 Retrieve relevant passages on corpus $C$;

8 Fine-tune the $G$ with Eq. (4) on $D$ and retrieved negative passages.

9while  _models has not converged_ do

10 Fine-tune the $R$ with Eq. (11) on $D$ and retrieved passages;

11 Refresh ANN index for $R$;

12 Retrieve relevant passages on corpus $C$;

13 Fine-tune the $G$ with Eq. (4) on $D$ and retrieved negative passages.

14 end while

Algorithm 1 The training algorithm.

##  5 Experiments

In this section, we construct experiments to demonstrate the effectiveness of
our method.

###  5.1 Experimental Setup

Datasets. We evaluate the proposed method on two public cross-lingual
retrieval datasets: XOR-Retrieve Asai et al. (2021a) and MKQA Longpre et al.
(2020). The detailed descriptions of the two datasets are presented in
Appendix A.

Evaluation Metrics. Following previous works Asai et al. (2021a); Sorokin et
al. (2022), we use R@2kt and R@5kt as evaluation metrics for the XOR-Retrieve
dataset and R@2kt as evaluation metrics for the MKQA dataset. The metrics
measure the proportion of queries to which the top k retrieved tokens contain
the span answer, which is fairer with different passage sizes.

Implementation Details. For the warm-up training stage, we follow XOR-Retrieve
to first train the model on NQ Kwiatkowski et al. (2019) data and then fine-
tune the model with XOR-Retrieve data. For the iteratively training stage, we
generate seven queries for each case (because the XOR-Retrieve data contains
seven languages). We set the number of retrieved passages as 100, the number
of iterations as $5$, threshold $T$ in Eq. (8) as 0.3 and coefficient $\alpha$
in Eq. (11) as 0.5. The detailed hyper-parameters are shown in Appendix C. And
we conduct more experiments to analyze the parameter sensitivity in Appendix
D.

All the experiments run on 8 NVIDIA Tesla A100 GPUs. The implementation code
is based on HuggingFace Transformers Wolf et al. (2020). For the dual-encoder,
we use XLM-R Base Conneau et al. (2020) as the pre-trained model and use the
average hidden states of all tokens to represent the sentence. For the query
generator, we leverage mT5 Base Xue et al. (2021) as the pre-trained model,
which has almost the same number of parameters as a large cross-encoder.

Table 1: Comparison results on XOR-Retrieve dev set. The best results are in bold. “$\ast$” denotes the results are copied from the source paper. Results unavailable are left blank. Methods | R@2kt | R@5kt
---|---|---
Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg | Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg
mDPR∗ | 38.8 | 48.4 | 52.5 | 26.6 | 44.2 | 33.3 | 39.9 | 40.5 | 48.9 | 60.2 | 59.2 | 34.9 | 49.8 | 43.0 | 55.5 | 50.2
DPR + MT∗ | 43.4 | 53.9 | 55.1 | 40.2 | 50.5 | 30.8 | 20.2 | 42.0 | 52.4 | 62.8 | 61.8 | 48.1 | 58.6 | 37.8 | 32.4 | 50.6
Sentri∗ | 47.6 | 48.1 | 53.1 | 46.6 | 49.6 | 44.3 | 67.9 | 51.0 | 56.8 | 62.2 | 65.5 | 53.2 | 55.5 | 52.3 | 80.3 | 60.8
w/ Bi-Encoder∗ | 47.8 | 39.1 | 48.9 | 51.2 | 40.2 | 41.2 | 49.4 | 45.4 | 55.1 | 43.3 | 59.5 | 59.4 | 51.2 | 52.0 | 56.9 | 53.9
DR.DECR∗ | - | - | - | - | - | - | - | 66.0 | 70.2 | 85.9 | 69.4 | 65.1 | 68.8 | 68.8 | 83.2 | 73.1
w/o KD${}_{XOR}^{\ast}$ | - | - | - | - | - | - | - | 60.6 | - | - | - | - | - | - | - | 68.6
w/o KD${}_{PC}^{\ast}$ | - | - | - | - | - | - | - | 56.6 | - | - | - | - | - | - | - | 63.6
QuiCK | 52.8 | 70.1 | 62.2 | 54.8 | 62.8 | 57.8 | 70.6 | 61.3 | 63.8 | 78.0 | 65.3 | 63.5 | 69.8 | 67.1 | 74.8 | 68.9
QuiCK w/ LaBSE | 67.3 | 78.9 | 65.9 | 59.8 | 66.3 | 63.7 | 80.7 | 68.9 | 72.2 | 83.2 | 69.7 | 68.0 | 70.9 | 71.7 | 84.9 | 74.4
Table 2: Comparison results on XOR-Retrieve test set.  Methods | R@2kt | R@5kt
---|---|---
GAAMA | 52.8 | 59.9
Sentri | 52.7 | 61.0
CCP | 54.8 | 63.0
Sentri 2.0 | 58.5 | 64.6
DR.DECR | 63.0 | 70.3
QuiCK w/ LaBSE | 65.6 | 72.0

###  5.2 Results

Baselines. We compare the proposed QuiCK with previous state-of-the-art
methods, including mDPR, DPR+MT Asai et al. (2021a), Sentri Sorokin et al.
(2022), DR.DECR Li et al. (2021). Note that Sentri introduces a shared encoder
with large size, DR.DECR introduces parallel queries and parallel corpus, but
our method only utilizes an encoder with base size, XOR-Retrieve and NQ
training data. For more fairly comparison, we also report their ablation
results. Here, “Bi-Encoder” denotes two unshared encoders with base size.
“KDXOR” denotes a distillation method which introduces synonymous English
queries. “KDPC” denotes a distillation method which introduces parallel
corpus. In addition, we also employ LaBSE base Feng et al. (2022) to evaluate
the proposed QuiCK with parallel corpus, which is a state-of-the-art model
pre-trained with parallel corpus.

XOR-Retrieve. Table 1 shows the results on XOR-Retrieve dev set. The proposed
QuiCK outperforms mDPR, DPR+MT, and Sentri with a clear edge in almost all
languages. Although QuiCK does not introduce any parallel corpus, it also
outperforms DR.DECR w/o KDXOR. Finally, QuiCK based on LaBSE outperforms all
baselines, especially DR.DECR w/o KDXOR, and even outperforms DR.DECR which
utilizes both parallel queries and parallel corpus. Note that knowledge
distillation with parallel corpus in DR.DECR is designed for cross-lingual
dense retrieval, but LaBSE is a general pre-trained model for all cross-
lingual tasks. These results show the effectiveness of the proposed QuiCK. Our
method combines two methods in dense retrieval and cross-lingual tasks, namely
distillation and alignment. We further analyze the contribution of each
component in Section 5.3.

In addition, we show the results on XOR-Retrieve test set in Table 2, which is
copied from the leaderboard111https://nlp.cs.washington.edu/xorqa on June 15,
2022. As we can see, our method achieves the top position on the leaderboard
of XOR-Retrieve.

Table 3: Average performance of 20 unseen languages in MKQA test set. “$\ast$” denotes the results are copied from the Sentri paper. Methods | R@2kt
---|---
CORA∗ | 41.1
BM25 + MT∗ | 42.0
Sentri∗ | 53.3
w/ Bi-Encoder∗ | 45.3
QuiCK | 53.4
QuiCK w/ LaBSE | 60.3

MKQA. Furthermore, we evaluate the zero-shot performance of our method on the
MKQA test set. Following previous works Sorokin et al. (2022), we directly
evaluate the dual-encoder training on XOR-Retrieve data and report the
performance of unseen languages on MKQA. As shown in Table 3, our method
outperforms all baselines and even performs better than Sentri. Note that
Sentri uses a shared encoder with large size. The comparison between Sentri
and Sentri w/ Bi-Encoder shows that the large encoder has better transfer
ability. Finally, the proposed QuiCK w/ LaBSE outperforms all baselines with a
clear edge. It shows the better transfer ability of our methods.

Table 4: Ablation results on XOR-Retrieve dev set.  Methods | R@2kt | R@5kt
---|---|---
QuiCK | 61.3 | 68.9
w/o Sampling | 59.5 | 67.5
w/o Alignment | 59.9 | 67.1
w/o Generation | 58.8 | 65.9
w/o All | 41.5 | 53.4

###  5.3 Methods Analysis

Ablation Study. Here, we check how each component contributes to the final
performance. We construct the ablation experiments on XOR-Retrieve data. We
prepare four variants of our method: (1) w/o Sampling denotes without the
scheduled sampling but keep the threshold $T$ for $c^{\prime}$, _a.k.a.,_ if
$c^{\prime}\geq T$, then $c^{\prime}=1$, otherwise $c^{\prime}=0$; (2) w/o
Alignment denotes without $\mathcal{L}_{A}$ in Eq. (11); (3) w/o Generation
denotes without $\mathcal{L}_{D}^{\prime}$ and $\mathcal{L}_{A}$ in Eq. (11);
(4) w/o All denotes without the enhanced training, _a.k.a.,_ the warm-up dual-
encoder.

Table 4 presents all comparison results of the four variants. As we can see,
the performance rank of R@5kt can be given as: w/o All < w/o Generation < w/o
Alignment < w/o Sampling < QuiCK. These results indicate that all components
are essential to improve performance. And we can find the margin between w/o
Alignment and w/o Sampling is small, it denotes that the generated queries are
noisy and demonstrate the effectiveness of our schedule sampling strategy.

Effect of Alignment. As we mentioned in Section 1, the alignment established
in the pre-training stage may be damaged without any constraint in the fine-
tuning stage. Here, we construct experiments on both XLM-R and LaBSE to
analyze the effectiveness of the proposed alignment training. As shown in
Table 5, the proposed alignment training is effective based on the two models.
It indicates that the alignment constraint in the fine-tuning stage is
effective for models which pre-trained with parallel corpus. And we find that
the gains of alignment training based on XLM-R are larger than LaBSE, which
shows that the alignment constraint is more effective for models which do not
pre-trained with parallel corpus.

Table 5: Effect of alignment based on different pre-trained languages models. Methods | XLM-R | LaBSE
---|---|---
R@2kt | R@5kt | R@2kt | R@5kt
QuiCK | 61.3 | 68.9 | 68.9 | 74.4
w/o Alignment | 59.9 | 67.1 | 67.9 | 73.8
Figure 4: Re-ranking performance of cross-encoder and query generator on XOR-
Retrieve dev set with different numbers of candidate passages. Figure 5: The
changes of R@2kt during the iteratively training on XOR-Retrieve dev set.
Here, “QG” denotes Query Generator and “CE” denotes Cross Encoder.

Cross-Encoder versus Query Generator. Here, we analyze the re-ranking ability
of cross-encoder and query generator. Here, we use the warm-up dual-encoder to
retrieve passages, vary the number of candidate passages, and then evaluate
the re-ranked result. As shown in Figure 4, when we use the top-100 candidate
passages, the performance of the cross-encoder and generator is almost the
same. But as the number of candidate passages increases, especially when it
surpasses 500, the gap between the performance of the cross-encoder and query
generator gradually becomes larger. It shows low generalization performance of
the cross-encoder when there are not enough training samples.

Visualization of the Training Procedure. We visualize the performance changes
of R@2kt during the training of both dual-encoder and query generator re-
ranker which re-ranks the retrieved top-100 passages. We also incorporate a
cross-encoder (initialized with XLM-R Large) to perform distillation and re-
ranking for comparison. As shown in Figure 5, the R@2kt of all models
gradually increases as the iteration increases. While the training advances
closer to convergence, the improvement gradually slows down. In the end, the
performance of the dual-encoder is improved by approximately 17%, and the
performance of the query generator is improved by approximately 20%. Finally,
comparing the performance of the cross-encoder and the query generator, we can
find that there are approximately 6% gaps for both teachers and students. It
shows the effectiveness of our method.

##  6 Conclusion

In this paper, we showed that the cross-encoder performs poorly when there are
not sufficient training samples which are hard to obtain in the cross-lingual
setting. Then we proposed a novel method that utilizes the query generator to
improve the dual-encoder. We firstly proposed to use a query generator as the
teacher. After that, we proposed a novel alignment method for cross-lingual
retrieval which does not need any parallel corpus. Extensive experimental
results show that the proposed method outperforms the baselines and
significantly improves the state-of-the-art performance. Currently, our method
depends on training data in all target languages. As future work, we will
investigate how to perform the proposed method for zero-shot cross-lingual
dense retrieval.

##  7 Limitations

The limitations are summarized as follows.

  * •

The method depends on training data in all target languages. Intuitively, the
method can be directly applied to the zero-shot cross-lingual dense retrieval
if we only take the passage as input for the query generator, but the query
generator performs poorly in the zero-shot setting. As future work, novel pre-
training tasks for cross-lingual generation can be considered.

  * •

The method does not investigate how to effectively train the query generator
for the re-ranking task, just directly applies the training method for the
cross-encoder re-ranker. We believe the potential of query generators for re-
ranking is strong and designing a special re-ranking training method for query
generators such as token-level supervision may be interesting for future work.

  * •

The method requires large GPU sources. The final model approximately costs 12
hours on 8 NVIDIA Tesla A100 GPUs. Although researchers who do not have enough
GPU sources can use the “gradient accumulation” technique to reduce GPU memory
consumption, they also need to pay more time.

  * •

This work does not consider the inconsistency between different countries
(_e.g.,_ law and religion), which leads to inconsistent positive passages for
synonymous queries in different languages (_e.g.,_ the legal age of marriage
varies from country to country). Because we find that most of the queries in
XOR-Retrieve contain the target country such as _“Mikä on yleisin kissa laji
Suomessa?” (translation: What is the most common cat breed in Finland?)_.

## References

  * Asai et al. (2021a) Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021a.  XOR QA: cross-lingual open-retrieval question answering.  In _NAACL-HLT_ , pages 547–564. Association for Computational Linguistics.
  * Asai et al. (2021b) Akari Asai, Xinyan Yu, Jungo Kasai, and Hanna Hajishirzi. 2021b.  One question answering model for many languages with cross-lingual dense passage retrieval.  In _NeurIPS_ , pages 7547–7560.
  * Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021.  Infoxlm: An information-theoretic framework for cross-lingual language model pre-training.  In _NAACL-HLT_ , pages 3576–3588. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _ICLR_. OpenReview.net.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _ACL_ , pages 8440–8451. Association for Computational Linguistics.
  * Dai and Callan (2020) Zhuyun Dai and Jamie Callan. 2020.  Context-aware term weighting for first stage passage retrieval.  In _SIGIR_ , pages 1533–1536. ACM.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _NAACL-HLT (1)_ , pages 4171–4186. Association for Computational Linguistics.
  * dos Santos et al. (2020) Cícero Nogueira dos Santos, Xiaofei Ma, Ramesh Nallapati, Zhiheng Huang, and Bing Xiang. 2020.  Beyond [CLS] through ranking by generation.  In _EMNLP (1)_ , pages 1722–1727. Association for Computational Linguistics.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022\.  Language-agnostic BERT sentence embedding.  In _ACL (1)_ , pages 878–891. Association for Computational Linguistics.
  * Gao and Callan (2021a) Luyu Gao and Jamie Callan. 2021a.  Condenser: a pre-training architecture for dense retrieval.  In _EMNLP (1)_ , pages 981–993. Association for Computational Linguistics.
  * Gao and Callan (2021b) Luyu Gao and Jamie Callan. 2021b.  Unsupervised corpus aware language model pre-training for dense passage retrieval.  _CoRR_ , abs/2108.05540.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016.  A deep relevance matching model for ad-hoc retrieval.  In _CIKM_ , pages 55–64. ACM.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020\.  REALM: retrieval-augmented language model pre-training.  _CoRR_ , abs/2002.08909.
  * Hu et al. (2022) Xiaomeng Hu, Shi Yu, Chenyan Xiong, Zhenghao Liu, Zhiyuan Liu, and Ge Yu. 2022.  P${}^{\mbox{3}}$ ranker: Mitigating the gaps between pre-training and ranking fine-tuning with prompt-based learning and pre-finetuning.  _CoRR_ , abs/2205.01886.
  * Hui et al. (2018) Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. 2018.  Co-pacrr: A context-aware neural IR model for ad-hoc retrieval.  In _WSDM_ , pages 279–287. ACM.
  * Humeau et al. (2020) Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020.  Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring.  In _ICLR_. OpenReview.net.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Distilling knowledge from reader to retriever for question answering.  In _ICLR_. OpenReview.net.
  * Johnson et al. (2021) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021.  Billion-scale similarity search with gpus.  _IEEE Trans. Big Data_ , 7(3):535–547.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _EMNLP (1)_ , pages 6769–6781. Association for Computational Linguistics.
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  Colbert: Efficient and effective passage search via contextualized late interaction over BERT.  In _SIGIR_ , pages 39–48. ACM.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019\.  Natural questions: a benchmark for question answering research.  _Trans. Assoc. Comput. Linguistics_ , 7:452–466.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _ACL (1)_ , pages 6086–6096. Association for Computational Linguistics.
  * Lesota et al. (2021) Oleg Lesota, Navid Rekabsaz, Daniel Cohen, Klaus Antonius Grasserbauer, Carsten Eickhoff, and Markus Schedl. 2021.  A modern perspective on query likelihood with deep generative retrieval models.  In _ICTIR_ , pages 185–195. ACM.
  * Li et al. (2020) Canjia Li, Andrew Yates, Sean MacAvaney, Ben He, and Yingfei Sun. 2020.  PARADE: passage representation aggregation for document reranking.  _CoRR_ , abs/2008.09093.
  * Li et al. (2021) Yulong Li, Martin Franz, Md. Arafat Sultan, Bhavani Iyer, Young-Suk Lee, and Avirup Sil. 2021.  Learning cross-lingual IR from an english retriever.  _CoRR_ , abs/2112.08185.
  * Longpre et al. (2020) Shayne Longpre, Yi Lu, and Joachim Daiber. 2020.  MKQA: A linguistically diverse benchmark for multilingual open domain question answering.  _CoRR_ , abs/2007.15207.
  * Lu et al. (2021) Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021.  Less is more: Pretrain a strong siamese encoder for dense text retrieval using a weak decoder.  In _EMNLP (1)_ , pages 2780–2791. Association for Computational Linguistics.
  * Luan et al. (2021) Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2021.  Sparse, dense, and attentional representations for text retrieval.  _Trans. Assoc. Comput. Linguistics_ , 9:329–345.
  * Muennighoff (2022) Niklas Muennighoff. 2022.  SGPT: GPT sentence embeddings for semantic search.  _CoRR_ , abs/2202.08904.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _EMNLP (Findings)_ , volume EMNLP 2020 of _Findings of ACL_ , pages 708–718. Association for Computational Linguistics.
  * Nogueira et al. (2019) Rodrigo Nogueira, Jimmy Lin, and AI Epistemic. 2019.  From doc2query to doctttttquery.  _Online preprint_.
  * Nogueira and Cho (2019) Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with BERT.  _CoRR_ , abs/1901.04085.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering.  In _NAACL-HLT_ , pages 5835–5847. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Ren et al. (2021a) Ruiyang Ren, Shangwen Lv, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021a.  PAIR: leveraging passage-centric similarity relation for improving dense passage retrieval.  In _ACL/IJCNLP (Findings)_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 2173–2183. Association for Computational Linguistics.
  * Ren et al. (2021b) Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021b.  Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking.  In _EMNLP (1)_ , pages 2825–2835. Association for Computational Linguistics.
  * Robertson and Zaragoza (2009) Stephen E. Robertson and Hugo Zaragoza. 2009.  The probabilistic relevance framework: BM25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389.
  * Sachan et al. (2022) Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022.  Improving passage retrieval with zero-shot question generation.  _CoRR_ , abs/2204.07496.
  * Sorokin et al. (2022) Nikita Sorokin, Dmitry Abulkhanov, Irina Piontkovskaya, and Valentin Malykh. 2022\.  Ask me anything in your native language.  In _NAACL-HLT_ , pages 395–406. Association for Computational Linguistics.
  * Tang et al. (2021) Hongyin Tang, Xingwu Sun, Beihong Jin, Jingang Wang, Fuzheng Zhang, and Wei Wu. 2021\.  Improving document representations by generating pseudo query embeddings for dense retrieval.  In _ACL/IJCNLP (1)_ , pages 5054–5064. Association for Computational Linguistics.
  * van den Oord et al. (2018) Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NIPS_ , pages 5998–6008.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _EMNLP (Demos)_ , pages 38–45. Association for Computational Linguistics.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _ICLR_. OpenReview.net.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _NAACL-HLT_ , pages 483–498. Association for Computational Linguistics.
  * Yang et al. (2022) Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022.  Enhancing cross-lingual transfer by manifold mixup.  _CoRR_ , abs/2205.04182.
  * Yang and Seo (2020) Sohee Yang and Minjoon Seo. 2020.  Is retriever merely an approximator of reader?  _CoRR_ , abs/2010.10999.
  * Zhan et al. (2021) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021\.  Optimizing dense retrieval model training with hard negatives.  In _SIGIR_ , pages 1503–1512. ACM.
  * Zhang et al. (2021) Hang Zhang, Yeyun Gong, Yelong Shen, Jiancheng Lv, Nan Duan, and Weizhu Chen. 2021\.  Adversarial retriever-ranker for dense text retrieval.  _CoRR_ , abs/2110.03611.
  * Zhang et al. (2022) Shunyu Zhang, Yaobo Liang, Ming Gong, Daxin Jiang, and Nan Duan. 2022.  Multi-view document representation learning for open-domain dense retrieval.  In _ACL (1)_ , pages 5990–6000. Association for Computational Linguistics.
  * Zheng et al. (2021) Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021.  Consistency regularization for cross-lingual fine-tuning.  In _ACL/IJCNLP (1)_ , pages 3403–3417. Association for Computational Linguistics.
  * Zhou et al. (2022) Jiawei Zhou, Xiaoguang Li, Lifeng Shang, Lan Luo, Ke Zhan, Enrui Hu, Xinyu Zhang, Hao Jiang, Zhao Cao, Fan Yu, Xin Jiang, Qun Liu, and Lei Chen. 2022.  Hyperlink-induced pre-training for passage retrieval in open-domain question answering.  In _ACL (1)_ , pages 7135–7146. Association for Computational Linguistics.
  * Zhuang et al. (2021) Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021.  Deep query likelihood model for information retrieval.  In _ECIR (2)_ , volume 12657 of _Lecture Notes in Computer Science_ , pages 463–470. Springer.

## Appendix

##  Appendix A Dataset

XOR-Retrieve. XOR-Retrieve dataset is a cross-lingual retrieval dataset which
aims to retrieve relevant passages from the English corpus for non-English
queries. XOR-Retrieve data contains queries in seven typologically diverse
languages: Arabic (Ar), Bengali (Bn), Finnish (Fi), Japanese (Ja), Korean
(Ko), Russian (Ru), and Telugu (Te). The statistics are presented in Table 6.

Table 6: Data statistics for XOR-Retrieve. | Train | Dev | Test
---|---|---|---
Ar | 2,574 | 350 | 137
Bn | 2,582 | 312 | 128
Fi | 2,088 | 360 | 530
Ja | 2,288 | 296 | 449
Ko | 2,469 | 299 | 646
Ru | 1,941 | 366 | 235
Te | 1,308 | 238 | 374
Corpus size | 18,003,200

MKQA. MKQA dataset is a translated dataset of 10,000 query-answer pairs from
NQ to 26 different languages, and the dataset is only used for evaluation. In
our experiments, since we measure the R@2kt score, we filter the samples which
do not have span answers. Then, we get 6,620 parallel queries in each
language. Finally, we directly evaluate the dual-encoder trained on XOR-
Retrieve data and use the same corpus with XOR-Retrieve in the experiments.
Note that Arabic (Ar), English (En), Finnish (Fi), Japanese (Ja), Korean (Ko),
and Russian (Ru) are seen in the training stage and we only report the
performance of the rest 20 languages. As a result, it has a total of 132,400
samples.

##  Appendix B Efficiency Report

We list the time cost of training and inference in Table 7 which is made with
8 NVIDIA A100 GPUs.

Table 7: Efficiency Report. Training | Warm-up | 3h
---|---|---
Per Iteration of Dual-Encoder | 1h
Per Iteration of Generator | 0.3h
Index Refresh | 0.35h
Overall | 11.5h
Inference | Build Index | 0.35h
Query Encoding | 40ns
Dense Retrieval | 2ms

##  Appendix C Hyper-parameters

We present all hyper-parameters in Table 8.

Table 8: Hyper-parameters. | Parameters | Value
---|---|---
| Max Query Length | 32
| Max Passage Length | 128
Training Warm-up Dual-Encoder | Learning Rate | 1e-5
Batch Size | 128
Negative Size | 255
Optimizer | AdamW
Scheduler | Linear
Warmup Proportion | 0.1
Training Steps on NQ | 18400
Training Steps on XOR | 2000
Training Warm-up Generator on QG | Learning Rate | 1e-4
Batch Size | 64
Optimizer | AdamW
Scheduler | Linear
Warmup Proportion | 0.1
Training Steps | 5000
Training Warm-up Generator on Re-ranking | Learning Rate | 1e-5
Batch Size | 32
Negative Size | 15
Optimizer | AdamW
Scheduler | Linear
Warmup Proportion | 0.1
Training Steps | 1000
Iteraively Training of Dual-Encoder | Learning Rate | 1e-5
Batch Size | 64
Candidate Size | 32
Optimizer | AdamW
Scheduler | Linear
Warmup Proportion | 0.1
Training Steps | 3000
Threshold $T$ in Eq. (8) | 0.3
Coefficient $\alpha$ in Eq. (11) | 0.5
# of iterations | 5
Iteraively Training of Generator | Learning Rate | 1e-5
Batch Size | 32
Negative Size | 15
Optimizer | AdamW
Scheduler | Linear
Warmup Proportion | 0.1
Training Steps | 500
# of iterations | 5

(a) Threshold ($T$). (b) Coefficient ($\alpha$).

Figure 6: Parameter sensitivity.

##  Appendix D Additional Experiments

###  D.1 Parameter Sensitivity

In this section, we tune the parameters of the proposed method to analyze
parameter sensitivity. We vary both the threshold $T$ (Eq. (8)) and the
coefficient $\alpha$ (Eq. (11)) in the set $\\{0.1,$ $0.3,$ $0.5,$ $0.7,$
$0.9\\}$. We report the tuning results with both R@2kt and R@5kt on the XOR-
Retrieve dev set in Figure 6. As we can see, $T=0.3$ and $\alpha=0.5$ lead to
the optimal R@5kt which is the ordering basis on the leaderboard.

In addition, we find that the optimal R@2kt and R@5kt are led by different
thresholds $T$. Because the two metrics have different sensitivities to data
quality, low-quality data is helpful to R@5kt but harmful to R@2kt. As a
result, a small threshold $T$ leads to more low-quality alignment training
data and further leads to higher R@5kt but lower R@2kt. On the contrary, the
optimal R@2kt and R@5kt are led by the same coefficient $\alpha$.

Overall, our model is relatively stable when varying the two parameters, and
consistently better than Sentri and Dr.DECR w/o KDPC.

###  D.2 Effect of The Number of Candidates

Here, we investigate the effect of the number of candidates which is
demonstrated to have a significant effect on the final performance. As shown
in Figure 7, a large number of candidates leads to better performance. And
when the number surpasses 32, the improvement gradually slows down. The
results indicate that 32 candidates can better represent the whole corpus.

Figure 7: Effect of the number of candidate passages.

###  D.3 Effect of Model Size

Table 9: Effect of model size. | XLM-R Base | XLM-R Large
---|---|---
R@2kt | R@5kt | R@2kt | R@5kt
Ar | 52.8 | 63.8 | 64.4 | 72.2
Bn | 70.1 | 78.0 | 77.0 | 80.9
Fi | 62.2 | 65.3 | 67.0 | 70.1
Ja | 54.8 | 63.5 | 56.4 | 67.2
Ko | 62.8 | 69.8 | 67.4 | 73.3
Ru | 57.8 | 67.1 | 66.7 | 72.2
Te | 70.6 | 74.8 | 79.4 | 84.0
Avg | 61.3 | 68.9 | 68.4 | 74.3

Following Sentri Sorokin et al. (2022), we employ a shared encoder (_i.e.,_
the parameters of the query encoder and the passage encoder are the same) with
large size as the dual-encoder and evaluate our method. As shown in Table 9,
QuiCK with XLM-R Large achieves a significant performance improvement, which
further demonstrates the effectiveness of the proposed QuiCK.

###  D.4 Effect of Scheduled Sampling

Previously, we demonstrate the effectiveness of the scheduled sampling by
ablation study. Here, we present two generated samples to qualitatively
analyze the scheduled sampling. As shown in Table 10, for the first case, the
generated queries in other languages have the same semantics as the query from
the source language. The sample is effective in alignment training and is
helpful to achieve better performance. For the second case, the generated
query in Finnish is relevant to the query from the source language but not
synonymous. The sample is harmful to the model training. These samples
indicate that the scheduled sampling is necessary for alignment training. In
this way, we can reduce the impact of the cases which do not have the same
semantics and further achieve better performance.

Table 10: Two generated examples. The span answers are in bold. Passage:
Johanna Maria Magdalena "Magda" Goebbels (née Ritschel; 11 November 1901 – 1
May 1945) was the wife of Nazi Germanyś Propaganda Minister Joseph Goebbels. A
prominent member of the Nazi Party, she was a close ally, companion and
political supporter of Adolf Hitler. Some historians refer to her as the
unofficial "First Lady" of Nazi Germany, while others give that title to Emmy
Göring.
---
Source Query (Ja):ヨハンナ・マリア・マクダレナ・ゲッベルスは何歳で死去した？
Translation: At what age did Johanna Maria McDalena Goebbels die?
Generated Query (Ru): В каком возрасте умерла Магда Геббельс?
Translation: At what age did Magda Goebbels die?
Generated Query (Fi): Minä vuonna Magda Goebbels kuoli?
Translation: In what year did Magda Goebbels die?
Passage: Charles V (24 February 1500 – 21 September 1558) was ruler of both
the Holy Roman Empire from 1519 and the Spanish Empire (as Charles V of Spain)
from 1516, as well as of the lands of the former Duchy of Burgundy from 1506.
He stepped down from these and other positions by a series of abdications
between 1554 and 1556. Through inheritance, he brought together under his rule
extensive territories in western, central, and southern Europe, and the
Spanish viceroyalties in the Americas and Asia.
Source Query (Ko): 신성 로마 제국 카를 5세 재위기간은 얼마나 되나요?
Translation: How long was the reign of Charles V of the Holy Roman Empire?
Generated Query (Ru): Сколько лет правил Карл V?
Translation: How many years did Charles V rule?
Generated Query (Fi): Minä vuonna Charles V hallitsi Rooman valtakuntaa?
Translation: In what year did Charles V rule the Roman Empire?
Table 11: A generated example with different input templates. The span answer
is in bold. Here, “QG” denotes query generator. Passage: The Higgs boson is an
elementary particle in the Standard Model of particle physics, produced by the
quantum excitation of the Higgs field, one of the fields in particle physics
theory. It is named after physicist Peter Higgs, who in 1964, along with five
other scientists, proposed the mechanism which suggested the existence of such
a particle. Its existence was confirmed in 2012 by the ATLAS and CMS
collaborations based on collisions in the LHC at CERN.
---
Generated Query by QG w/ span answer (Fi): Kuka on kehittänyt Higgs-boson?
Translation: Who developed the Higgs boson?
Generated Query by QG w/ span answer (Ko): 히그슨을 처음 발견한 사람은 누구인가?
Translation: Who first discovered Higson?
Generated Query by QG w/o span answer (Fi): Milloin Higgs on löydetty?
Translation: When was Higgs Found?
Generated Query by QG w/o span answer (Ko): Кто был первым исследователем
физики Higgs?
Translation: Who was Higgs’ first physics researcher?

###  D.5 Effect of Span Answer

In our method, we employ the span answer to encourage the query generator to
generate synonymous queries. Here, we conduct experiments to evaluate the
effect of the span answer. Specially, we use another template: _“generate
[language] query: [content]”_ where we only need to fill two placeholders with
the language of the target query and the passage content. We also incorporate
the cross-encoder for comparison. We use the re-rankers to re-rank the
retrieved results of the warm-up dual-encoder initialized with XLM-R. Note
that introducing the span answer into the cross-encoder makes the re-ranking
task easier, because the cross-encoder only needs to check whether the passage
contains the span answer. The scores of this cross-encoder almost degenerate
into hard labels and it is difficult to effectively train the dual-encoder by
distilling knowledge from this cross-encoder.

We show the results in Table 12. Based on these results, we have the following
findings. On the one hand, the query generator trained with span answers is
better than the query generator without span answers. It shows that taking
span answers as input leads to better performance on re-ranking tasks for the
query generator. On the other hand, both the two query generator is better
than the cross-encoder when re-ranking top-1000 retrieved passages, it shows
the effectiveness of the query generator in the cross-lingual setting.

Table 12: Performance comparison of different re-rankers on XOR-Retrieve dev set. The best results are in bold. Here, “QG” denotes query generator. Methods | R@2kt | R@5kt
---|---|---
Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg | Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg
Dual-Encoder | 35.3 | 43.1 | 50.3 | 35.7 | 44.6 | 31.2 | 50.0 | 41.5 | 49.5 | 54.9 | 59.2 | 45.2 | 55.1 | 31.2 | 63.4 | 53.4
Re-ranking top-100 retrieved passages
QG w/ Answer | 51.1 | 57.2 | 53.5 | 43.2 | 55.1 | 43.9 | 61.8 | 52.3 | 56.6 | 60.5 | 60.5 | 51.0 | 60.7 | 46.4 | 67.6 | 57.6
QG w/o Answer | 48.9 | 55.6 | 54.1 | 41.9 | 53.3 | 43.5 | 61.8 | 51.3 | 56.6 | 60.9 | 59.9 | 49.8 | 59.3 | 46.8 | 68.1 | 57.3
Cross-Encoder | 49.2 | 53.6 | 57.6 | 41.1 | 54.0 | 41.4 | 63.0 | 51.4 | 55.3 | 59.9 | 62.1 | 49.8 | 60.4 | 47.3 | 66.4 | 57.3
Re-ranking top-1000 retrieved passages
QG w/ Answer | 53.7 | 66.1 | 56.7 | 52.3 | 59.3 | 56.1 | 68.9 | 59.0 | 61.2 | 71.1 | 62.1 | 58.1 | 65.6 | 61.6 | 74.4 | 64.9
QG w/o Answer | 52.4 | 62.8 | 56.1 | 49.0 | 58.2 | 55.7 | 64.3 | 56.9 | 59.9 | 70.7 | 62.1 | 57.3 | 64.9 | 59.9 | 73.5 | 64.0
Cross-Encoder | 50.8 | 58.2 | 55.1 | 45.2 | 59.3 | 50.6 | 65.5 | 55.0 | 61.2 | 67.4 | 63.4 | 53.5 | 66.3 | 56.1 | 73.9 | 63.1

In addition, we show queries generated by the two query generators in Table
11. As we can see, for the query generator that does not take the span answer
as input, the generated queries can be answered by the passage, but they focus
on different segments of the passage and they are not synonymous. On contrary,
for the query generator that takes the span answer as input, generated queries
can be answered by the passage and they are synonymous. It shows that taking
the span answer as input can effectively encourage the generator to generate
synonymous queries.

Table 13: Detailed performance on MKQA test set. “$\ast$” denotes that the results are copied from the Sentri paper. Methods | Da | De | Es | Fr | He | Hu | It | Km | Ms | Nl
---|---|---|---|---|---|---|---|---|---|---
CORA∗ | 44.5 | 44.6 | 45.3 | 44.8 | 27.3 | 39.1 | 44.2 | 22.2 | 44.3 | 47.3
BM25 + MT∗ | 44.1 | 43.3 | 44.9 | 42.5 | 36.9 | 39.3 | 40.1 | 31.3 | 42.5 | 46.5
Sentri∗ | 57.6 | 56.5 | 55.9 | 55.1 | 47.9 | 51.8 | 54.3 | 43.9 | 56.0 | 56.3
w/ Bi-Encoder∗ | 50.0 | 47.8 | 48.7 | 47.4 | 37.7 | 43.4 | 41.8 | 37.8 | 49.5 | 47.3
QuiCK | 58.3 | 56.4 | 55.2 | 55.5 | 44.7 | 52.4 | 52.3 | 42.0 | 56.9 | 57.5
QuiCK w/ LaBSE | 63.3 | 61.8 | 62.2 | 62.4 | 56.1 | 58.9 | 60.6 | 53.0 | 64.2 | 63.0
No | Pl | Pt | Sv | Th | Tr | Vi | Zh-cn | Zh-hk | Zh-tw | Avg
48.3 | 44.8 | 40.8 | 43.6 | 45.0 | 34.8 | 33.9 | 33.5 | 41.5 | 41.0 | 41.1
43.3 | 46.5 | 45.7 | 49.7 | 46.5 | 42.5 | 43.5 | 37.5 | 37.5 | 36.1 | 42.0
56.5 | 55.8 | 54.8 | 56.9 | 55.3 | 53.0 | 54.4 | 50.2 | 50.7 | 49.4 | 53.3
49.1 | 47.0 | 47.7 | 50.0 | 46.5 | 45.6 | 47.3 | 42.6 | 41.5 | 41.0 | 45.3
57.0 | 54.9 | 54.7 | 58.0 | 55.7 | 53.9 | 54.9 | 50.4 | 49.3 | 48.9 | 53.4
62.8 | 62.0 | 61.5 | 63.3 | 60.5 | 60.6 | 61.8 | 57.3 | 56.3 | 56.0 | 60.3
Table 14: Detailed performance for ablation study on XOR-Retrieve dev set.  | R@2kt | R@5kt
---|---|---
Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg | Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg
QuiCK | 52.8 | 70.1 | 60.2 | 54.8 | 62.8 | 57.8 | 70.6 | 61.3 | 63.8 | 78.0 | 65.3 | 63.5 | 69.8 | 67.1 | 74.8 | 68.9
w/o Sampling | 53.1 | 68.4 | 60.2 | 50.2 | 61.1 | 55.7 | 68.1 | 59.5 | 63.4 | 78.0 | 64.6 | 60.6 | 68.1 | 63.7 | 74.4 | 67.5
w/o Alignment | 51.8 | 68.1 | 60.8 | 51.0 | 60.4 | 57.4 | 70.2 | 59.9 | 61.5 | 74.7 | 64.6 | 62.7 | 68.8 | 62.4 | 75.2 | 67.1
w/o Generation | 55.0 | 68.1 | 59.6 | 47.3 | 60.7 | 54.9 | 66.4 | 58.8 | 62.8 | 74.0 | 65.0 | 56.4 | 66.3 | 62.0 | 74.8 | 65.9
w/o All | 35.3 | 43.1 | 50.3 | 35.7 | 44.6 | 31.2 | 50.0 | 41.5 | 49.5 | 54.9 | 59.2 | 45.2 | 55.1 | 31.2 | 63.4 | 53.4
Table 15: Detailed performance of alignment based on different pre-trained languages models.  | R@2kt | R@5kt
---|---|---
Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg | Ar | Bn | Fi | Ja | Ko | Ru | Te | Avg
XLM-R | 52.8 | 70.1 | 60.2 | 54.8 | 62.8 | 57.8 | 70.6 | 61.3 | 63.8 | 78.0 | 65.3 | 63.5 | 69.8 | 67.1 | 74.8 | 68.9
w/o Alignment | 51.8 | 68.1 | 60.8 | 51.0 | 60.4 | 57.4 | 70.2 | 59.9 | 61.5 | 74.7 | 64.6 | 62.7 | 68.8 | 62.4 | 75.2 | 67.1
LaBSE | 67.3 | 78.9 | 65.9 | 59.8 | 66.3 | 63.7 | 80.7 | 68.9 | 72.2 | 83.2 | 69.7 | 68.0 | 70.9 | 71.7 | 84.9 | 74.4
w/o Alignment | 65.7 | 78.3 | 65.0 | 58.9 | 67.0 | 62.9 | 77.3 | 67.9 | 72.5 | 80.9 | 69.7 | 66.8 | 71.9 | 70.5 | 84.5 | 73.8

##  Appendix E Detailed Results

Due to the limited space, we only present average performance for some
experiments in Section 5. Here, we present the detailed performance in all
languages of these experiments. Firstly, we present the detailed performance
of all methods on the MKQA test set in Table 13. Secondly, we present the
detailed performance of ablation results in Table 14. Finally, we present the
detailed performance for evaluating the effect of alignment in Table 15.