capbtabboxtable[][0.25]

# One Question Answering Model for Many Languages with Cross-lingual Dense
Passage Retrieval

Akari Asai†, Xinyan Yu†, Jungo Kasai†, Hannaneh Hajishirzi†‡
†University of Washington, ‡Allen Institute for AI
{akari, xyu530, jkasai<EMAIL_ADDRESS>

###### Abstract

We present Cross-lingual Open-Retrieval Answer Generation (CORA), the first
unified many-to-many question answering (QA) model that can answer questions
across many languages, even for ones without language-specific annotated data
or knowledge sources. We introduce a new dense passage retrieval algorithm
that is trained to retrieve documents across languages for a question.
Combined with a multilingual autoregressive generation model, CORA answers
directly in the target language without any translation or in-language
retrieval modules as used in prior work. We propose an iterative training
method that automatically extends annotated data available only in high-
resource languages to low-resource ones. Our results show that CORA
substantially outperforms the previous state of the art on multilingual open
QA benchmarks across 26 languages, 9 of which are unseen during training. Our
analyses show the significance of cross-lingual retrieval and generation in
many languages, particularly under low-resource settings. Our code and trained
model are publicly available at https://github.com/AkariAsai/CORA.

##  1 Introduction

Multilingual open question answering (QA) is the task of answering a question
from a large collection of multilingual documents. Most recent progress in
open QA is made for English by building a pipeline based on a dense passage
retriever trained on large-scale English QA datasets to find evidence passages
in English (Lee et al., 2019; Karpukhin et al., 2020), followed by a reader
that extracts an answer from retrieved passages. However, extending this
approach to multilingual open QA poses new challenges. Answering multilingual
questions requires retrieving evidence from knowledge sources of other
languages than the original question since many languages have limited
reference documents or the question sometimes inquires about concepts from
other cultures (Asai et al., 2021; Lin et al., 2020). Nonetheless, large-scale
cross-lingual open QA training data whose questions and evidence are in
different languages are not available in many of those languages.

To address these challenges, previous work in multilingual open QA (Ture and
Boschee, 2016; Asai et al., 2021) translates questions into English, applies
an English open QA system to answer in English, and then translates answers
back to the target language. Those pipeline approaches suffer from error
propagation of the machine translation component into the downstream QA,
especially for low-resource languages. Moreover, they are not able to answer
questions whose answers can be found in resources written in languages other
than English or the target languages.

In this paper, we introduce a unified many-to-many QA model that can answer
questions in any target language by retrieving evidence from any language and
generating answers in the target language. Our method (called CORA, Fig. 1)
extends the retrieve-then-generate approach of English open QA (Lewis et al.,
2020; Izacard and Grave, 2021b) with a single cross-lingual retriever and a
generator that do not rely on language-specific retrievers or machine
translation modules. The multilingual retrieval module (mDPR) produces dense
embeddings of a question and all multilingual passages, thereby retrieving
passages across languages. The generation module (mGEN) is trained to output
an answer in the target language conditioned on the retrieved multilingual
passages. To overcome the aforementioned data scarcity issue, we automatically
mine training data using external language links and train mDPR and mGEN
iteratively. In particular, each iteration proceeds over two stages of
updating model parameters with available training data and mining new training
data cross-lingually by Wikipedia language links and predictions made by the
models. This approach does not require any additional human annotations or
machine translation, and can be applied to many new languages with low
resources.

Our experiments show that CORA advances the state of the art on two
multilingual open QA datasets, Xor-TyDi QA (Asai et al., 2021) and MKQA
(Longpre et al., 2020), across 26 typologically diverse languages; CORA
achieves gains of 23.4 and 4.7 F1 points in Xor-TyDi QA and MKQA respectively,
where MKQA data is not used for training. Moreover, CORA achieves F1 scores of
roughly 30 over 8 languages on MKQA that have no training data or even
reference Wikipedia documents, outperforming the state-of-the-art approach by
5.4 F1 points. Our controlled experiments and human analyses illustrate the
impact of many-to-many cross-lingual retrieval in improving multilingual open
QA performance. We further observe that through cross-lingual retrieval, CORA
can find answers to 20% of the multilingual questions that are valid but are
originally annotated as unanswerable by humans due to the lack of evidence in
the English knowledge sources.

Figure 1: Overview of CORA (mDPR and mGEN).

##  2 Method

We define multilingual open QA as the task of answering a question $q^{L}$ in
a target language $L$ given a collection of multilingual reference passages
$\mathbf{C}^{multi}$, where evidence passages can be retrieved from any
language. These passages come from Wikipedia articles that are not necessarily
parallel over languages. We introduce CORA, which runs a retrieve-then-
generate procedure to achieve this goal (Fig. 1). We further introduce a novel
training scheme of iterative training with data mining (§ 2.2).

###  2.1 CORA Inference

CORA directly retrieves evidence passages from any language for questions
asked in any target language, and then generates answers in the target
language conditioned on those passages. More formally, the CORA inference
consists of two steps of (i) retrieving passages $\mathscr{P}^{multi}$ and
(ii) generating an answer $a^{L}$ based on the retrieved passages.
$\mathscr{P}^{multi}$ can be in any language included in $\mathbf{C}^{multi}$.

$\mathscr{P}^{multi}=\text{mDPR}(q^{L},\mathbf{C}^{multi}),\leavevmode\nobreak\
a^{L}=\text{mGEN}(q^{L},\mathscr{P}^{multi}).$

Multilingual Dense Passage Retriever (mDPR). mDPR extends Dense Passage
Retriever (DPR; Karpukhin et al., 2020) to a multilingual setting. mDPR uses
an iterative training approach to fine-tune a pre-trained multilingual
language model (e.g., mBERT; Devlin et al., 2019) to encode passages and
questions separately. Once training is done, the representations for all
passages from $\mathbf{C}^{multi}$ are computed offline and stored locally.
Formally, a passage encoding is obtained as follows:
$\mathbf{e}_{p^{L}}=\operatorname{mBERT}_{p}(p)$, where a passage $p$ is a
fixed-length sequence of tokens from multilingual documents. At inference,
mDPR independently obtains a $d$-dimensional ($d=768$) encoding of the
question $\mathbf{e}_{q^{L}}=\operatorname{mBERT}_{q}(q^{L})$. It retrieves
$k$ passages with the $k$ highest relevance scores to the question, where the
relevance score between a passage $p$ and a question $q^{L}$ is estimated by
the inner product of their encoding vectors,
$\langle\mathbf{e}_{q^{L}},\mathbf{e}_{p}\rangle$.

Multilingual Answer Generator (mGEN).  We use a multilingual sequence-to-
sequence model (e.g., mT5; Xue et al., 2021) to generate answers in the target
language token-by-token given the retrieved multilingual passages
$\mathscr{P}^{multi}$. We choose a generation approach because it can generate
an answer in the target language $L$ from passages across different
languages.111An alternative approach of answer extraction requires translation
for all language pairs (Asai et al., 2021). Moreover, the generator can be
adapted to unseen languages, some of which may have little or no translation
training data. Specifically, the generator outputs the sequence probability
for $a^{L}$ as follows:

$P(a^{L}|q^{L},\mathscr{P}^{multi})=\prod_{i}^{T}p(a^{L}_{i}|a^{L}_{<i},q^{L},\mathscr{P}^{multi}),$
(1)

where $a^{L}_{i}$ denotes the $i$-th token in the answer, and $T$ is the
length of the answer. We append a language tag to the question to indicate the
target language.

###  2.2 CORA Training

Figure 2: Overview of CORA iterative training and data mining.

We introduce an iterative training approach that encourages cross-lingual
retrieval and answer generation conditioned on multilingual passages (sketched
in Fig. 2 and Alg. 1). Each iteration proceeds over two stages: parameter
updates (§ 2.2.1) where mDPR and mGEN are trained on the current training data
and cross-lingual data mining (§ 2.2.2) where training data are automatically
expanded by Wikipedia language links and model predictions.

##### Initial training data.

The initial training data is a combination of multilingual QA datasets: Xor-
TyDi QA and TyDi QA (Clark et al., 2020), and an English open QA dataset
(Natural Questions, Kwiatkowski et al., 2019). Each training instance from
these datasets comprises a question, a positive passage, and an answer. Note
that annotations in the existing QA datasets have critical limitations:
positive passages are taken either from English (Asai et al., 2021) or the
question’s language (Clark et al., 2020). Further, most of the non-English
languages are not covered. Indeed, when we only train mDPR on this initial
set, it often learns to retrieve passages in the same languages or similar
languages with irrelevant context or context without sufficient evidence to
answer.

####  2.2.1 Parameter Updates

mDPR updates (line 3 in Alg. 1). Let $\mathscr{D}=\\{\langle
q_{i}^{L},p^{+}_{i},p^{-}_{i,1},\cdots,p^{-}_{i,n}\rangle\\}_{i=1}^{m}$ be $m$
training instances. Each instance consists of a question $q_{i}^{L}$, a
passage that answers the question (positive passage) $p^{+}_{i}$, and $n$
passages that do not answer the question (negative passages) $p^{-}_{i,j}$.
For each question, we use positive passages for the other questions in the
training batch as negative passages (in-batch negative, Gillick et al., 2019;
Karpukhin et al., 2020). mDPR is updated by minimizing the negative log
likelihood of positive passages:

$\mathscr{L}_{\text{mdpr}}=-\log\frac{\exp(\langle\mathbf{e}_{q_{i}^{L}},\,\mathbf{e}_{p_{i}^{+}}\rangle)}{\exp(\langle\mathbf{e}_{q^{L}_{i}},\,\mathbf{e}_{p_{i}^{+}}\rangle)+\sum_{j=1}^{n}{\exp(\langle\mathbf{e}_{q^{L}_{i}},\,\mathbf{e}_{p^{-}_{i,j}}\rangle)}}.$
(2)

mGEN updates (lines 4-5 in Alg. 1). After updating mDPR, we use mDPR to
retrieve top $k$ passages $\mathscr{P}^{multi}$ for each $q^{L}$. Given these
pairs of the question and the retrieved passages $(q^{L},\mathscr{P}^{multi})$
as input, mGEN is trained to generate answer $a^{L}$ autoregressively (Eq.
(1)) and minimize the cross-entropy loss. To train the model to generate in
languages not covered by the original datasets, we translate $a^{L}$ to other
languages using Wikipedia language links and create new synthetic
answers.222This automatic answer translation is only done after the third
epoch of initial training to prevent the model from overfitting to synthetic
data.  See Appendix § A.2 for more detail.

####  2.2.2 Cross-lingual Data Mining

Data: Input QA pairs: $(q^{L},a^{L})$

1 initialize training data
$\mathbf{B}^{1}=(q^{\mathbf{L}},a^{\mathbf{L}},p_{gold}),\mathbf{L}=\\{{\rm
Eng},L\\}$;

2 while  _$t <T$_ do

$\Theta_{mDPR}^{t}\leftarrow Train(\theta_{mDPR}^{t-1},\mathbf{B}^{t})$/*
Train mDPR */

$\mathscr{P}^{multi}\leftarrow{\rm
mDPR}(q^{\mathbf{L}},\mbox{embedding}(\mathbf{C}^{multi}))$/* Retrieve
passages */

$\theta_{mGEN}^{t}\leftarrow
Train(\theta_{mGEN}^{t-1},(q^{\mathbf{L}},a^{\mathbf{L}},\mathscr{P}^{multi}))$/*
Train mGEN */

For $\mathbf{L}=={\rm Eng}$, $\mathscr{P}^{multi}+={\rm
LangLink}(q^{\mathbf{L}},\mathbf{C}^{multi}))$ /* Mine data using Wikidata */

3 For $p_{i}\in\mathscr{P}^{multi}$: if mGEN$(q^{\mathbf{L}},p_{i})==a^{L}$
then $positives.add(p_{i})$ else  $negatives.add(p_{i})$

$\mathbf{B}^{t+1}$ += $(q^{\mathbf{L}},a^{\mathbf{L}},positives,negatives)$ /*
Add new training data */

4 $t\leftarrow t+1$

5 end while

Algorithm 1 Iterative training that automatically mines training data.

After the parameter updates, we mine new training data using mDPR and
Wikipedia language links and label the new data by mGEN predictions. This step
is skipped in the final iteration.

Mining by trained mDPR and language links (line 4, 6 in Alg. 1). Trained mDPR
can discover positive passages in another language that is not covered by the
initial training data. At each iteration, we use retrieved passages
$\mathscr{P}^{multi}$ for $q^{L}$ (line 4 in Alg. 1) as a source of new
positive and negative passages. This enables expanding data between language
pairs not in the original data.

To cover even more diverse languages, we use language links and find passages
in other languages that potentially include sufficient evidence to answer.
Wikipedia maintains article-level language links that connect articles on the
same entity over languages. We use these links to expand training data from
the English QA dataset of Natural Questions (line 6 in Alg. 1). Denote a
training instance by $(q^{En},a^{En},p_{gold})$. We first translate the
English answer $a^{En}$ to a target language $a^{L}$ using language links. We
use language links again to look up the English Wikipedia article that the
gold passage $p_{gold}$ comes from. We then find articles in non-English
languages in the reference documents $\mathbf{C}^{multi}$ that correspond to
this article. Although the language link-based automatic translation cannot
handle non-entity answers (e.g., short phrases), this helps us to scale to new
languages without additional human annotation or machine translation. We add
all passages from these articles to $\mathscr{P}^{multi}$ as positive passage
candidates, which are then passed to mGEN to evaluate whether each of them
leads to $a^{L}$ or not.

Automatic labeling by mGEN predictions (lines 7-8 in Alg. 1). A passage
$p_{i}$ from $\mathscr{P}^{multi}$ may not always provide sufficient
information to answer the question $q^{L}$ even when it includes the answer
string $a^{L}$. To filter out those spurious passages (Lin et al., 2018; Min
et al., 2019), we take instances generated from the two mining methods
described above, and run mGEN on each passage to predict an answer for the
question. If the answer matches the correct answer $a^{L}$, then the passage
$p_{i}$ is labeled as a positive passage; otherwise we label the input passage
as a negative passage. We assume that when mGEN fails to generate a correct
answer given the passage, the passage may not provide sufficient evidence to
answer; this helps us filter out spurious passages that accidentally contain
an answer string yet do not provide any clue to answer. We add these new
positive and negative passages to the training data, and in the next
iteration, mDPR is trained on this expanded training set (§ 2.2.1).

##  3 Experiments

We evaluate CORA on two multilingual open QA datasets across 28 typologically
diverse languages.333A full list of the language families and script types are
in the appendix. CORA achieves state-of-the-art performance across 26
languages, and greatly outperforms previous approaches that use language-
specific components such as question or answer translation.

###  3.1 Datasets and Knowledge Sources

Multilingual open QA datasets differ in covered languages, annotation schemes,
and target application scenarios. We evaluate F1 and EM scores over the
questions with answer annotations from two datasets, following the common
evaluation practice in open QA (Lee et al., 2019).

Xor-TyDi QA. Xor-TyDi QA (Asai et al., 2021) is a multilingual open QA dataset
consisting of 7 typologically diverse languages, where questions are
originally from TyDi QA (Clark et al., 2020) and posed by information-seeking
native speakers. The answers are annotated by extracting spans from Wikipedia
in the same language as the question (in-language data) or by translating
English spans extracted from English Wikipedia to the target language (cross-
lingual data). Xor-TyDi QA offers both training and evaluation data.

MKQA. MKQA (Longpre et al., 2020) is an evaluation dataset created by
translating 10k Natural Questions (Kwiatkowski et al., 2019) to 25 target
languages. The parallel data enables us to compare the models’ performance
across typologically diverse languages, in contrast to Xor-TyDi QA. MKQA has
evaluation data only; Xor-TyDi QA and MKQA have five languages in common.

Collection of multilingual documents $\mathbf{C}^{multi}$. We use the February
2019 Wikipedia dumps of 13 diverse languages from all Xor-TyDi QA languages
and a subset of MKQA languages.444 Downloaded from
https://archive.org/details/wikimediadownloads?and%5B%5D=year%3A%222019%22. We
choose 13 languages to cover languages with a large number of Wikipedia
articles and a variety of both Latin and non-Latin scripts. We extract plain
text from Wikipedia articles using
wikiextractor,555https://github.com/attardi/wikiextractor and split each
article into 100-token segments as in DPR (Karpukhin et al., 2020). We filter
out disambiguation pages that distinguish pages that share the same article
title666https://en.wikipedia.org/wiki/Category:Disambiguation_pages. as well
as pages with fewer than 20 tokens, resulting in 43.6M passages. See more
details in Appendix § B.2.

Language categories. To better understand the model performance, we categorize
the languages based on their availability during our training. We call the
languages with human annotated gold paragraph and answer data seen languages.
Xor-TyDi QA provides gold passages for 7 languages. For the languages in
$\mathbf{C}^{multi}$ without human-annotated passages, we mine new mDPR
training data by our iterative approach. We call these languages, which are
seen during mDPR training, mDPR-seen. We also synthetically create mGEN
training data as explained in § 2.2.1 by simply replacing answer entities with
the corresponding ones in the target languages. The languages that are unseen
by mDPR but are seen by mGEN mGEN-seen, and all other languages (i.e.,
included neither in mDPR nor mGEN training) unseen languages. 9 of the MKQA
languages are unseen languages.

###  3.2 Baselines and Experimental Setting

We compare CORA with the following strong baselines adopted from Asai et al.
(2021).

Translate-test (MT + DPR). As used in most previous work (e.g., Asai et al.,
2021), this method translates the question to English, extracts an answer in
English using DPR, and then translates the answer back to the target language.
The translation models are obtained from MarianMT (Junczys-Dowmunt et al.,
2018) and trained on the OPUS-MT dataset (Tiedemann, 2012).

Monolingual baseline (BM25). This baseline retrieves passages solely from the
target language and extracts the answer from the retrieved passages. Training
neural network models such as DPR is infeasible with a few thousands of
training examples. Due to the lack of training data in most of the target
languages, we use a BM25-based lexical retriever implementation by Pyserini
(Lin et al., 2021). We then feed the retrieved documents to a multilingual QA
model to extract final answers.

MT+Mono. This baseline combines results from the translate-test method and the
monolingual method to retrieve passages in both English and the target
language. Following Asai et al. (2021), we prioritize predictions from the
monolingual pipeline if they are over a certain threshold tuned on Xor-TyDi QA
development set; otherwise we output predictions from the translate-test
method.777For the languages not supported by Pyserini, we always output
translate-test’s predictions.

Closed-book baseline. This model uses an mT5-base888We did not use larger-
sized variants due to our computational budget. sequence-to-sequence model
that takes a question as input and generates an answer in the target language
without any retrieval at inference time (Roberts et al., 2020). This baseline
assesses the models’ ability to memorize and retrieve knowledge from its
parameters without retrieving reference documents.

CORA details. For all experiments, we use a single retriever (mDPR) and a
single generator (mGEN) that use the same passage embeddings. mDPR uses
multilingual BERT base uncased,999The alternative of XLM-RoBERTa (Conneau et
al., 2020) did not improve our results. and the generator fine-tunes mT5-base.
We found that using other pre-trained language models such as mBART (Liu et
al., 2020) for mGEN or XLM-R (Conneau et al., 2020) for mDPR did not improve
performance and sometimes even hurt performance. We first fine-tune mDPR using
gold passages from Natural Questions, and then further fine-tune it using Xor-
TyDi QA and TyDi QA’s gold passage data. We exclude the training questions in
Natural Questions and TyDi QA that were used to create the MKQA or Xor-TyDi QA
evaluation set. We run two iterations of CORA training (§ 2.2) after the
initial fine-tuning. All hyperparameters are in Appendix § B.5.

##  4 Results and Analysis

###  4.1 Multilingual Open QA Results

#####  Xor-TyDi QA.

Table 1 reports the scores of CORA and the baselines in Xor-TyDi QA. CORA,
which only uses a single retriever and a single generator, outperforms the
baselines and the previous state-of-the-art model on Xor-TyDi QA by a large
margin across all 7 languages. CORA achieves gains of 24.8 macro-averaged F1
points over the previous state-of-the-art method (GMT+GS), which uses external
black-box APIs, and 23.4 points over the concurrent anonymous work (SER).

Models |  Target Language $L_{i}$ F1 | Macro Average
---|---|---
| Ar | Bn | Fi | Ja | Ko | Ru | Te | F1 | EM | BLEU
CORA | 59.8 | 40.4 | 42.2 | 44.5 | 27.1 | 45.9 | 44.7 | 43.5 | 33.5 | 31.1
SER | 32.0 | 23.1 | 23.6 | 14.4 | 13.6 | 11.8 | 22.0 | 20.1 | 13.5 | 20.1
GMT+GS | 31.5 | 19.0 | 18.3 | 8.8 | 20.1 | 19.8 | 13.6 | 18.7 | 12.1 | 16.8
MT+Mono | 25.1 | 12.7 | 20.4 | 12.9 | 10.5 | 15.7 | 0.8 | 14.0 | 10.5 | 11.4
MT+DPR | 7.6 | 5.9 | 16.2 | 9.0 | 5.3 | 5.5 | 0.8 | 7.2 | 3.3 | 6.3
BM25 | 31.1 | 21.9 | 21.4 | 12.4 | 12.1 | 17.7 | – | – | – | –
Closed-book | 14.9 | 10.0 | 11.4 | 22.2 | 9.4 | 18.1 | 10.4 | 13.8 | 9.6 | 7.4
Table 1: Performance on Xor-Full (test data F1 scores and macro-averaged F1,
EM and BLEU scores). “GMT+GS” denotes the previous state-of-the-art model,
which combines Google Custom Search in the target language and Google
Translate + English DPR for cross-lingual retrieval (Asai et al., 2021).
Concurrent to our work, “SER” is a state-of-the-art model, Single Encoder
Retriever, submitted anonymously on July 14 to the Xor-Full leaderboard
(https://nlp.cs.washington.edu/xorqa/). We were not able to find a BM25
implementation that supports Telugu.

MKQA. Tables 2 and 3 report the F1 scores of CORA and the baselines on over
6.7k MKQA questions with short answer annotations101010 Following previous
work in open QA but different from the official script of MKQA (Longpre et
al., 2020), we disregard the questions labeled as “no answer”. As shown in our
human analysis, it is difficult to prove an answer does not exist in the
millions of multilingual documents even if the annotation says so.  under seen
and unseen settings. CORA significantly outperforms the baselines in all
languages by a large margin except for Arabic and English. Note that Longpre
et al. (2020) report results in a simplified setting with gold reference
articles from the original Natural Questions dataset given in advance, and
thus their results are not comparable. CORA yields larger improvements over
the translate-test baseline in the languages that are distant from English and
with limited training data such as Malay (Ms; 27.8 vs. 12.6) and Hebrew (He;
15.8 vs. 8.9). The performance drop of the translate-test model from English
(43.3 F1) to other languages indicates the error propagation from the
translation process. BM25 performs very poorly in some low-resource languages
such as Thai because of the lack of answer content in the target languages’
Wikipedia. MT+Mono underpeforms the MT+DPR baseline in MKQA since it is
challenging to rerank answers from two separate methods with uncaliberated
confidence scores. In contrast, CORA retrieves passages across languages,
achieving around 30 F1 on a majority of the 26 languages.

Setting | – |  Seen (Included in Xor-TyDi QA) | mDPR-seen
---|---|---|---
|  Avg. over all $L$. | En | Ar | Fi | Ja | Ko | Ru | Es | Sv | He | Th
CORA | 21.8 | 40.6 | 12.8 | 26.8 | 19.7 | 12.0 | 19.8 | 32.0 | 30.9 | 15.8 | 8.5
MT+Mono | 14.1 | 19.3 | 6.9 | 17.5 | 9.0 | 7.0 | 10.6 | 21.3 | 20.0 | 8.9 | 8.3
MT+DPR | 17.1 | 43.3 | 16.0 | 21.7 | 9.6 | 5.7 | 17.6 | 28.4 | 19.7 | 8.9 | 6.9
BM25 | – | 19.4 | 5.9 | 9.9 | 9.1 | 6.9 | 8.1 | 14.7 | 10.9 | – | 4.9
Closed | 4.5 | 8.0 | 4.6 | 3.6 | 6.5 | 3.8 | 4.1 | 6.6 | 4.8 | 3.8 | 2.1
Table 2: F1 scores on MKQA seen and mDPR-seen languages.

Setting | mGEN-seen | Unseen
---|---|---
| Da | De | Fr | It | Nl | Pl | Pt | Hu | Vi | Ms | Km | No | Tr | cn | hk | tw
CORA | 30.4 | 30.2 | 30.8 | 29.0 | 32.1 | 25.6 | 28.4 | 18.4 | 20.9 | 27.8 | 5.8 | 29.2 | 22.2 | 5.2 | 6.7 | 5.4
MT+Mono | 19.3 | 21.6 | 21.9 | 20.9 | 21.5 | 24.6 | 19.9 | 16.5 | 15.1 | 12.6 | 1.2 | 17.4 | 16.6 | 4.9 | 3.8 | 5.1
MT+DPR | 26.2 | 25.9 | 21.9 | 25.1 | 28.3 | 24.6 | 24.7 | 15.7 | 15.1 | 12.6 | 1.2 | 18.3 | 18.2 | 3.3 | 3.8 | 3.8
BM25 | 9.5 | 12.5 | – | 13.6 | 12.8 | – | 13.4 | 7.4 | – | – | – | 9.4 | 8.8 | 2.8 | – | 3.3
Closed | 4.7 | 5.6 | 5.8 | 5.3 | 5.5 | 4.0 | 4.4 | 5.5 | 5.9 | 5.3 | 1.9 | 4.1 | 3.8 | 2.6 | 2.3 | 2.4

Table 3: F1 scores on MKQA in mGEN-seen and unseen languages.

“cn”: “Zh-cn” (Chinese, simplified). “hk”: “Zh-hk” (Chinese, Hong Kong).
“tw”:“Zh-tw” (Chinese, traditional).

###  4.2 Analysis

##### Ablations: Impact of CORA components.

Setting |  Xor-TyDi QA | MKQA
---|---|---
| Avg. F1 | Ar | Ja | Te | Avg. F1 | Fi | Ru | Es | Th | Vi
CORA | 31.4 | 42.6 | 33.4 | 26.1 | 22.3 | 25.9 | 20.6 | 33.2 | 6.3 | 22.6
(i) mDPR1 \+ mGEN1 | 27.9 | 36.2 | 29.8 | 21.1 | 17.3 | 23.1 | 13.1 | 28.5 | 5.7 | 18.6
(ii) DPR (trained NQ)+mGEN | 24.3 | 30.7 | 29.2 | 19.0 | 17.9 | 20.1 | 16.9 | 29.4 | 5.5 | 18.2
(iii) CORA, $\mathbf{C}^{multi}$={En} | 19.1 | 20.5 | 23.2 | 11.5 | 20.5 | 24.7 | 15.4 | 28.3 | 8.3 | 21.9
(iv) mDPR+Ext.reader+MT | 11.2 | 11.8 | 10.8 | 5.6 | 12.2 | 16.1 | 10.9 | 25.2 | 1.2 | 12.7
Table 4: Ablation studies on Xor-TyDi QA development set and a subset of MKQA.

We compare CORA with the following four variants to study the impact of
different components. (i) mDPR1 \+ mGEN1 only trains CORA using the initial
labeled, annotated data and measures the impact of the iterative training.
(ii) DPR (trained NQ) + mGEN replaces mDPR with a multilingual BERT-based DPR
trained on English data from Natural Questions (NQ), and encodes all passages
in $\mathbf{C}^{multi}$. This configuration assesses the impact of cross-
lingual training data. (iii) CORA, $\mathbf{C}^{multi}$={En}  only retrieves
from English during inference. This variant evaluates if English reference
documents suffice to answer multilingual questions. (iv) mDPR+Ext.reader+MT
replaces mGEN with an extractive reader model followed by answer translation.
This variant quantifies the effectiveness of using a multilingual generation
model over the approach that combines an extractive reader model with
language-specific translation models. Note that for MKQA experiments, we
sample the same 350 questions ($\sim$5%) from the evaluation set for each
language to reduce the computational cost over varying configurations.

Results in Table 4 show performance drops in all variants. This supports the
following claims: (i) the iterative learning and data mining process is
useful, (ii) mDPR trained with cross-lingual data substantially outperforms
DPR with multilingual BERT, (iii) reference languages other than English are
important in answering multilingual questions, and (iv) a multilingual
generation model substantially boosts the model performance.

##### Retrieval performance and relationship to the final QA performance.

| Setting |  mDPR-Seen | | Unseen
---|---|---|---
| Lang | Es | Fi | Ja | Ru | Th | Pt | Ms | Tr | Zh-Cn | Zh-Hk | Km
| Script | Latn | | Jpan | | Cyrl | | Thai | Latn | | Hant | | Khmr
mDPR | R${}^{\texttt{L}}$@10 | 53.7 | 52.8 | 32.9 | 42.3 | 14.9 | 50.0 | 49.4 | 42.0 | 12.6 | 16.6 | 15.7
R${}^{\texttt{multi}}$@10 | 63.4 | 60.9 | 42.0 | 54.0 | 28.0 | 62.6 | 63.4 | 55.4 | 40.6 | 42.3 | 25.1
DPR(NQ) | R${}^{\texttt{L}}$@10 | 52.3 | 46.0 | 24.6 | 36.0 | 12.6 | 45.7 | 48.8 | 32.0 | 9.1 | 14.0 | 13.4
| R${}^{\texttt{multi}}$@10 | 63.1 | 53.1 | 32.9 | 49.1 | 29.4 | 56.8 | 58.0 | 44.0 | 36.3 | 39.4 | 23.4
Table 5:  Retrieval recall performance on MKQA as the percentage of the
questions where at least one out of the top 10 passages includes an answer
string in the target language (R${}^{\texttt{L}}$@10), or in any language
(R${}^{\texttt{multi}}$@10). The same subset of the MKQA evaluation data are
used as in the ablations.

We evaluate CORA’s retrieval performance on MKQA using two recall metrics that
measure the percentage of questions with at least one passage among the top
$10$ that includes a string in an answer set in the target language
(R${}^{\texttt{L}}$@10) or in the union of answer sets from all languages that
are available in MKQA (R${}^{\texttt{multi}}$@10). MKQA provides answer
translations across 26 languages.

Table 5 reports retrieval results for mDPR and multilingual BERT-based DPR
trained on NQ: DPR (NQ). This is equivalent to (ii) from the ablations. We
observe that mDPR performs well in Indo-European languages with Latin script,
even when the language is unseen. Interestingly, there is a significant
performance gap between R${}^{\texttt{L}}$@10 and R${}^{\texttt{multi}}$@10 in
languages with non-Latin script (e.g., Japanese, Russian, Chinese); this
suggests that our model often uses relevant passages from other languages with
Latin script such as English or Spanish to answer questions in those languages
with non-Latin script. Our mDPR outperforms DPR (NQ) by a large margin in
unseen languages with limited resources, which are consistent with the
findings in Table 3. Nevertheless, we still see low performance on Khmer and
Thai even with the R${}^{\texttt{multi}}$@10 metric. We also observe that
passage and query embeddings for those languages are far from other languages,
which can be further studied in future work. We provide a two-dimensional
visualization of the encoded passage representations in the appendix.

[.98]

Figure 3:  Breakdown of the languages of retrieved reference passages for
sampled MKQA questions (%). The x and y axes indicate target (question) and
retrieval reference languages respectively.

[0.99]  Ja Es retrieval errors 28 48 different lang 18 0 incorrect answer 22
36 annotation error 22 12 underspecified q 10 4 Table 6: Error categories (%)
on 50 errors sampled from Japanese (Ja) and Spanish (Es) data.

##### Breakdown of reference languages.

Fig. 3 breaks down retrieved reference languages for each target language. Our
multilingual retrieval model often retrieves documents from the target
language (if its reference documents are available), English, or its
typologically similar languages. For example, mDPR often retrieves Spanish
passages for Portuguese questions and Japanese passages for Chinese questions;
while they are considered phylogenetically distant, Japanese and Chinese
overlap in script.

To further evaluate this, we conduct a controlled experiment: we remove
Spanish, Swedish and Indonesian document embeddings and evaluate CORA on
related languages: Danish, Portuguese and Malay. We observe performance drops
of 1.0 in Danish, 0.6 in Portuguese, and 3.4 F1 points in Malay. This
illustrates that while CORA allows for retrieval from any language in
principle (many-to-many), cross-lingual retrieval from closer languages with
more language resources is particularly helpful.

##### Error analysis and qualitative examples.

Table 6 analyzes errors from CORA by manually inspecting 50 Japanese and
Spanish wrong predictions from MKQA. We observe six major error categories:
(a) retrieval errors, (b) generating correct answers in a different language
(different lang), (c) incorrect answer generation (incorrect answer), (d)
answer annotation errors (e.g., a correct alias isn’t covered by gold answers,
or Wikipedia information is inconsistent with English.), and (e) ambiguous or
underspecified questions such as “who won X this year” (underspecified q). The
table shows that both in Japanese and Spanish, the retrieval errors are
dominant. In Japanese, CORA often generates correct answers in English, not in
Japanese (different lang).

Fig. 4 shows some qualitative examples. The first example shows an error in
(b): mGEN is generating an answer in Russian, not in French though the answer
itself is correct. This type of error happens especially when retrieved
passages are in languages other than the target and English.

Figure 4: Cross-lingual retrieval and generation examples for three MKQA
questions.

##### Human evaluation on cross-lingual retrieval results.

To observe how cross-lingual retrieval between distant languages is actually
helping, we sample 25 Norwegian questions for which Spanish passages are
included among the top 10 retrieved results. As seen in Fig. 3, CORA retrieves
Spanish (es) passages for 6.8% of the Norwegian (no) questions. A Spanish
speaker judges if the retrieved Spanish passages actually answer the given
Norwegian questions.111111During evaluation, we provide the original English
questions from MKQA.  We found that in 96% of the cases, the retrieved Spanish
passages are relevant in answering the question. One such example is presented
in Fig. 4 (the second example).

##### Human analysis on unanswerable questions.

CORA retrieves passages from a larger multilingual document collection than
the original human annotations. Thus, CORA may further improve the answer
coverage over the original human annotations. MKQA includes questions that are
marked as unanswerable by native English speakers given English knowledge
sources. We sample 400 unanswerable Japanese questions whose top one retrieved
passage is from a non-English Wikipedia article. Among these, 329 unanswerable
questions are underspecified (also discussed in Asai and Choi, 2021). For 17
out of the 71 remaining questions, the answers predicted by CORA are correct.
This finding indicates the significance of cross-lingual retrieval and
potential room for improvement in annotating multilingual open QA datasets.
The third example in Fig. 4 shows one of these cases.

##  5 Related Work and Broader Impacts

English and non-English open QA. Despite the rapid progress in open QA (Chen
et al., 2017; Karpukhin et al., 2020), most prior work has been exclusively on
English (Lewis et al., 2020; Izacard and Grave, 2021b). Several prior attempts
to build multilingual open QA systems often rely on machine translation or
language-specific retrieval models (Ture and Boschee, 2016; Asai et al.,
2021). Lewis et al. (2020) and Guu et al. (2020) introduce a similar retrieve-
then-generate. Izacard and Grave (2021a) introduce an iterative training
framework that uses attention weights from a generator model as a proxy for
text relevance scores. Tran et al. (2020) introduce CRISS, a self-supervised
pre-training approach consisting of a parallel sentence mining module and a
sequence-to-sequence model, which are trained iteratively. Several recent work
such as Xiong et al. (2021) improves DPR by mining and learning with hard
examples. Our work is the first work that introduces a unified multilingual
system for many-to-many open QA, which is a challenging task requiring
massive-scale cross-lingual retrieval and has not been addressed in prior
work. We introduce an iterative training and data mining approach guided by
filtering from an answer generation model to automatically extend annotated
data available only in high-resource languages to low-resource. This approach
contributes to significant performance improvements in languages without
annotated training data.

Many-languages-one models. Several recent work introduces single multilingual
models for many languages using pre-trained multilingual models such as mBERT
or mT5 in many NLP tasks (e.g., entity linking: Botha et al., 2020; De Cao et
al., 2021; semantic role labeling: Mulcaire et al., 2019b; Lyu et al., 2019;
Fei et al., 2020; syntactic parsing: Mulcaire et al., 2019a; Kondratyuk and
Straka, 2019). This work conducts the first large-scale study of a unified
multilingual open QA model across many languages and achieves state-of-the-art
performance in 26 typologically diverse languages.

Synthetic data creation for machine reading comprehension. Alberti et al.
(2019) introduce a method of generating synthetic machine reading
comprehension data by automatically generating questions and filtering them
out by a trained machine reading comprehension model. Several studies augment
multilingual machine reading comprehension training data by generating new
question-answer pairs from randomly sampled non-English Wikipedia paragraphs
(Riabi et al., 2021; Shakeri et al., 2020). This work focuses on multilingual
open QA, which involves not only machine reading comprehension but also cross-
lingual retrieval. A similar augmentation method for machine reading
comprehension can be applied to further improve the answer generation
component in CORA.

Societal impacts. Our code and data are publicly available. CORA can perform
open QA in unseen languages and can benefit society in building QA systems for
low-resource languages, hence enabling research in that direction. Unlike
previous models, CORA removes the necessity of external black-box APIs, and
thus we can examine and address wrong answers due to model errors or
misinformation present on Wikipedia. This would help us mitigate the potential
negative impact from CORA or its subsequent models outputting a wrong answer
when it is used by people who seek information.

##  6 Conclusion

To address the information needs of many non-English speakers, a QA system has
to conduct cross-lingual passage retrieval and answer generation. This work
presents CORA, a unified multilingual many-to-many open QA model that
retrieves multilingual passages in many different languages and generates
answers in target languages. CORA does not require language-specific
translation or retrieval components and can even answer questions in unseen,
new languages. We conduct extensive experiments on two multilingual open QA
datasets across 28 languages, 26 of which CORA advances the state of the art
on, outperforming competitive models by up to 23 F1 points. Our extensive
analysis and manual evaluation reveal that CORA effectively retrieves
semantically relevant passages beyond language boundaries, and can even find
answers to the questions that were previously considered unanswerable due to
lack of sufficient evidence in annotation languages (e.g., English).
Nonetheless, our experimental results show that the retrieval component still
struggles to find relevant passages for queries in some unseen languages. Our
analysis also showed that CORA sometimes fails to generate an answer in the
target language. In future work, we aim to address these issues to further
improve the performance and scale our framework to even more languages.

## Acknowledgement

This research was supported by NSF IIS-2044660, ONR N00014-18-1-2826, gifts
from Google, the Allen Distinguished Investigator Award, the Sloan Fellowship,
and the Nakajima Foundation Fellowship. We thank anonymous reviewers, area
chairs, Eunsol Choi, Sewon Min, David Wadden, and the members of the UW NLP
group for their insightful feedback on this paper, and Gabriel Ilharco for his
help on human analysis.

## References

  * Alberti et al. (2019) Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019\.  Synthetic QA corpora generation with roundtrip consistency.  In _ACL_.
  * Asai and Choi (2021) Akari Asai and Eunsol Choi. 2021.  Challenges in information seeking QA: Unanswerable questions and paragraph retrieval.  In _ACL_.
  * Asai et al. (2021) Akari Asai, Jungo Kasai, Jonathan H. Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021.  XOR QA: Cross-lingual open-retrieval question answering.  In _NAACL_.
  * Botha et al. (2020) Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020.  Entity Linking in 100 Languages.  In _EMNLP_.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017.  Reading Wikipedia to answer open-domain questions.  In _ACL_.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _TACL_.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _ACL_.
  * De Cao et al. (2021) Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2021.  Multilingual autoregressive entity linking.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * Fei et al. (2020) Hao Fei, Meishan Zhang, and Donghong Ji. 2020.  Cross-lingual semantic role labeling with high-quality translated training corpus.  In _ACL_.
  * Gillick et al. (2019) Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019.  Learning dense representations for entity retrieval.  In _CoNLL_.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  In _ICML_.
  * Izacard and Grave (2021a) Gautier Izacard and Edouard Grave. 2021a.  Distilling knowledge from reader to retriever for question answering.  In _ICLR_.
  * Izacard and Grave (2021b) Gautier Izacard and Edouard Grave. 2021b.  Leveraging passage retrieval with generative models for open domain question answering.  In _EACL_.
  * Junczys-Dowmunt et al. (2018) Marcin Junczys-Dowmunt, Roman Grundkiewicz, Tomasz Dwojak, Hieu Hoang, Kenneth Heafield, Tom Neckermann, Frank Seide, Ulrich Germann, Alham Fikri Aji, Nikolay Bogoychev, André F. T. Martins, and Alexandra Birch. 2018.  Marian: Fast neural machine translation in C++.  In _ACL (System Demonstrations)_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _EMNLP_.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR_.
  * Kondratyuk and Straka (2019) Dan Kondratyuk and Milan Straka. 2019.  75 languages, 1 model: Parsing Universal Dependencies universally.  In _EMNLP_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: A benchmark for question answering research.  _TACL_.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _ACL_.
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  In _NeurIPS_.
  * Lin et al. (2021) Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021.  Pyserini: An easy-to-use python toolkit to support replicable ir research with sparse and dense representations.
  * Lin et al. (2020) Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2020.  Pretrained transformers for text ranking: BERT and beyond.
  * Lin et al. (2018) Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018.  Denoising distantly supervised open-domain question answering.  In _ACL_.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _TACL_.
  * Longpre et al. (2020) Shayne Longpre, Yi Lu, and Joachim Daiber. 2020.  MKQA: A linguistically diverse benchmark for multilingual open domain question answering.
  * Lyu et al. (2019) Chunchuan Lyu, Shay B. Cohen, and Ivan Titov. 2019.  Semantic role labeling with iterative structure refinement.  In _EMNLP_.
  * Min et al. (2021) Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, and Tom et al Kwiatkowski. 2021.  NeurIPS 2020 EfficientQA competition: Systems, analyses and lessons learned.  In _PMLR_.
  * Min et al. (2019) Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019.  A discrete hard em approach for weakly supervised question answering.  In _EMNLP_.
  * Mulcaire et al. (2019a) Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019a.  Low-resource parsing with crosslingual contextualized representations.  In _CoNLL_.
  * Mulcaire et al. (2019b) Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. 2019b.  Polyglot contextual representations improve crosslingual transfer.  In _NAACL_.
  * Riabi et al. (2021) Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Staiano. 2021.  Synthetic data augmentation for zero-shot cross-lingual question answering.  In _EMNLP_.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _EMNLP_.
  * Shakeri et al. (2020) Siamak Shakeri, Noah Constant, Mihir Sanjay Kale, and Linting Xue. 2020.  Towards zero-shot multilingual synthetic question and answer generation for cross-lingual reading comprehension.
  * Sugawara et al. (2018) Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018.  What makes reading comprehension questions easier? In _EMNLP_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _LREC_.
  * Tiedemann and Nygaard (2004) Jörg Tiedemann and Lars Nygaard. 2004.  The OPUS corpus - parallel and free.  In _LREC_.
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT — Building open translation services for the World.  In _EAMT_.
  * Tran et al. (2020) Chau Tran, Yuqing Tang, Xian Li, and Jiatao Gu. 2020.  Cross-lingual retrieval for iterative self-supervised training.  In _NeurIPS_.
  * Ture and Boschee (2016) Ferhan Ture and Elizabeth Boschee. 2016.  Learning to translate for multilingual question answering.  In _EMNLP_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _EMNLP (System Demonstrations)_.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate nearest neighbor negative contrastive learning for dense text retrieval.  In _ICLR_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _NAACL_.

## Appendix

##  Appendix A Details of Modeling

###  A.1 Input format

##### Passage representations.

To create a passage representation, the passage title and text are
concatenated ([CLS] title [SEP] passage [SEP]), following common practice
(Karpukhin et al., 2020). We retrieve top 10 passages and use them as input to
mGEN.

##### Generator input.

The input to the generator is a concatenation of $q^{L}$ and
$\mathscr{P}^{multi}$. As described in § 2.1, we append a language tag that
represents $L$ to $q^{L}$. For each passage, we prepend the retrieved ranks
and the original Wikipedia article titles and concatenate them to form a input
paragraph sequence. We differentiate those paragraphs from the question using
special tokens (<P> vs. <Q>). Finally, the concatenated passages are appended
to $q^{L}$ and the language tag. Below is an example input:

<Q>: ロンポールの学部時代の専攻は何？[ja] <P>:<0:ロン・ポール>ロナルド・アーネスト・ポール (英語: Ronald Ernest
“Ron” Paul、1935年8月20日 - ) は、アメリカ合衆国の元政治家。共和党所属でテキサス州選出の元連邦下院議員であった <1: Ron
Paul> Paul went to Gettysburg College, where he was a member of the Lambda Chi
Alpha fraternity. He graduated with a B.S. degree in Biology in 1957.

As in the case of machine translation, we found that the language code does
not need to be specified during inference as our model learns the question
language automatically. Yet, we found that training with language codes is
particularly useful to augment training data for $L_{target}$ without any
question data in $L_{target}$. In particular, given questions from existing
datasets in $L_{source}$ and entities names in $L_{target}$ corresponding to
the original answers in $L_{source}$, our generator learns to generate answers
in $L_{target}$ from the language code even when questions themselves are
written in $L_{source}$. Please see the details of training mGEN with
synthetic data in the next section.

###  A.2 Details of the Data Mining Process

##### Synthetic data for mGEN.

To train mGEN to generate answers in languages that are not covered by
annotated data or our reference sources, we augment English QA data
($q^{En},a^{En})$ from Natural Questions (Kwiatkowski et al., 2019). We first
use an English DPR model to retrieve $\mathscr{P}^{En}$ for each $q^{En}$.
Then, we automatically translate $a^{En}$ to a target language $L$ using
Wikipedia language links. We use Media Wiki
API,121212https://www.wikidata.org/w/api.php. and form new mGEN training data
$(q^{En},a^{L},\mathscr{P}^{En})$. Although the questions and passages are all
written in English, our model knows in which language it should answer from
the language code appended to the question. We limit the target languages for
this augmentation process to Arabic, Finnish, Japanese, Korean, Russian,
Spanish, Swedish, Hebrew, Thai, Danish, French, Italian, Dutch, Polish, and
Portuguese. Interestingly, just adding this language code effectively changes
the outputs as shown in Table 7. Although we could create at most 15 synthetic
data for each$(q^{En},a^{En},\mathscr{P}^{En})$, we sample at most 10
languages from the 15 languages to avoid overfitting. We further subsample 50%
of the synthetically generated questions. Those synthetically generate data is
introduced after training mGEN for 3 epochs to avoid overfitting.

input question | output | gold answers
---|---|---
who is the actor that plays the good doctor [ja] | フレッド・ハイモア | フレディ・ハイモア
who is the actor that plays the good doctor [ko] | 프레디 하이모어 | 프레디 하이모어
who is the actor that plays the good doctor [it] | Freddie Highmore | Freddie Highmore
Table 7: Examples of mGEN outputs with varying language codes.

##  Appendix B Details of Experiments

###  B.1 Details of the knowledge source language selection.

In addition to the English Wikipedia embeddings, we encode all of the passages
from the Wikipedias of all of the ten languages included in Xor-TyDi QA or
TyDi QA. Adding all of the languages available in Wikipedia to our document
collection would significantly increase the index size and slow down
inference. Therefore, we add the languages among the 26 MKQA languages that
satisfy the following criteria: (i) a language is included in Xor-TyDi QA or
TyDi QA, (ii) a language uses non-Latin script and has the largest number of
the Wikipedia articles among the languages in the same language family branch
(e.g., Thai), or (iii) a language uses Latin script and has more than 1.5
million articles as of May
2021.131313https://en.wikipedia.org/wiki/List_of_Wikipedias.

###  B.2 Details of Wikipedia statistics

For our multilingual retriever, we split each article into 100-token chunks
(Karpukhin et al., 2020), while BM25 first splits Wikipedia articles into the
pre-defined paragraph units. We also filter out the short articles with fewer
than $k$ (i.e., $k=20$ in this work) tokens, following common techniques in
open QA (Min et al., 2021) in $\mathbf{C}^{multi}$. As a result, we add more
than 43.6 million articles across the languages. The original passage text
file is 29GB, and the total index size is around 129 GB.

language | The number of articles | The number of passages
---|---|---
English | 6,297,085 | 18,003,200
Arabic | 664,693 | 1,304,828
Finnish | 451,338 | 886,595
Japanese | 1,268,148 | 5,116,905
Korean | 441,316 | 638,864
Russian | 1,522,499 | 4,545,635
Bengali | 64,556 | 179,936
Telugu | 70,356 | 274,230
Indonesian | 452,304 | 820,572
Thai | 129,122 | 520,139
Hebrew | 237,836 | 1,045,255
Swedish | 3,758,071 | 4,525,695
Spanish | 1,453,732 | 5,738,484
Table 8: Statistics of the Wikipedia data.

###  B.3 Licence, ethical considerations and data splits of Xor-TyDi QA and
MKQA

##### Licence.

Both two datasets are under the MIT licence. The dataset can be downloaded
from their official repositories.141414https://github.com/apple/ml-mkqa for
MKQA; https://github.com/AkariAsai/XORQA for Xor-TyDi QA.

##### Potential risk of offensive or personally identifiable information.

MKQA (Longpre et al., 2020) questions are originally from the Natural
Questions data (Kwiatkowski et al., 2019). The questions are anonymized Google
Search queries, and we expect that those questions are not personally
identifiable. The Natural Questions authors conduct several procedures to
filter out noisy questions, and we expect that the questions do not contain
offensive or inappropriate content.

Likewise, Xor-TyDi QA questions are from TyDi QA, where questions are written
by their in-house annotates who have native proficiency in the target
languages. The TyDi QA authors trained those in-house annotators and asked
them to write questions that they are interested in given short prompts. We
expect that the resulting questions are not personally identifiable and have
no risk of offensive information.

##### Data splits.

MKQA does not have any train data, and we use the questions with answer
annotations for evaluation, removing 1,427 unanswerable questions and 1,815
long_answer questions. Consequently, MKQA evaluation data has 6,758 questions
for each target language. Note that the unanswerable and long_answer type
information is provided in the original MKQA dataset, and we do not conduct
any manual data filtering. For the ablation or controlled experiments, we
randomly sample 350 questions from the 6,758 questions with short answers due
to our computational constraints.

We use the train, dev and test data splits from the original Xor-TyDi QA
(full) data.

###  B.4 Language family, branch and script type Information of the languages

Table 9 provides a full list of the 28 languages and their language family,
branch and script type information. The target languages are typologically
diverse; 12 of them use their own script system, which makes answer generation
in those languages harder than in the languages with Latin script.

Ar Bn Da De En name Arabic Bengali Danish German English family Afro-Asiatic
Indo-European Indo-European Indo-European Indo-European branch Semitic Indo-
Iranian Germanic Germanic Germanic script Arab Beng Latn Latn Latn Es Fi Fr He
Hu name Spanish Finnish French Hebrew Hungarian family Indo-European Uralic
Indo-European Afro-Asiatic Uralic branch Italic Finic Italic Semitic Finno-
Ugric script Latn Latn Latn Hebr Latn It Ja Ko Km Ms name Italian Japanese
Korean Khmer Malay family Indo-European Japonic Koreanic Austroasiatic
Austronesian branch Italic Japanese Korean Proto-Mon-Khmer Malayo-Polynesian
script Latn Jpan Hang Khmr Latn Nl No Pl Pt Ru name Dutch Norwegian Polish
Portuguese Russian family Indo-European Indo-European Indo-European Indo-
European Indo-European branch Germanic Germanic Balto-Slavic Italic Balto-
Slavic script Latn Latn Latn Latn Cyrl Sv Te Th Tr Vi name Swedish Telugu Thai
Turkish Vietnamese family Indo-European Dravidian Kra–Dai Altaic Austroasiatic
branch Germanic South-Centra Tai Turkic Vietic script Latn Telu Thai Latn Latn
Zh-cn Zh-hk Zh-tw name Chinese (Simplified) Chinese (Hong Kong) Chinese
(Traditional) family Sino-Tibetan Sino-Tibetan Sino-Tibetan branch Chinese
Chinese Chinese script Hans/Hant Hant Hant

Table 9: List of 28 language we test in this work. The script is based on ISO
15924.

###  B.5 Hyperparameters of CORA

##### mDPR.

We first fine-tune mDPR on the Natural Questions data using the training data
file released by DPR authors.151515https://github.com/facebookresearch/DPR. We
filter out questions that are used to create MKQA evaluation data by comparing
the input questions. We use the same hyperparameters as in the original DPR
(Karpukhin et al., 2020). We then perform fine-tuning on TyDi QA and Xor-TyDi
QA’s gold paragraph data, initializing the checkpoint that achieves the best
performance on the development data. We fine-tune the model for 40 epochs and
use the checkpoint that produces the best retrieval performance on Xor-TyDi
QA’s development data. We use 8 GPUs with 24G RAM, and the total batch size is
128. We empirically found that using the updated query encoder hurt the
retrieval performance on MKQA, while in Xor-TyDi QA we observe a marginal
performance drop. Therefore, at inference, we continue using the query encoder
trained on the initial data, while we use the updated passage encoder to
encode $\mathbf{C}^{multi}$.

##### mGEN.

The full list of the hyperparameters are in Table 10. We first train our mGEN
using the initial data for 15 epochs and use the checkpoint that gives the
highest development score. We use Adam (Kingma and Ba, 2015) for optimization.
We subsequently apply iterative training. During our $t$th iterative step, we
use the best checkpoint so far to label new positive and negative passages,
which will then be used to fine-tune mDPR at the next iteration. After we
finish mGEN training at the $t$-th iteration, we use the best checkpoint for
the final evaluation without performing additional data mining. We use our
internal cluster to run all of the mGEN related training. We use 8 GPUs with
24G RAM, and the total batch size is 32.

##### Inference.

During inference, we first retrieve top 15 passages using mDPR, and then feed
the questions and concatenated passages into the mGEN model, with language
tags. We use the same checkpoints and encoded embeddings for MKQA and Xor-TyDi
QA. There are minor differences in the gold answer format in MKQA and Xor-TyDi
QA due to different annotation methods (e.g., translate English answers by
Wikidata vs. use answers extracted from the target language Wikipedias). One
may fine-tune different models using different subsets of training examples
(e.g., MKQA can benefit from more NQ-based synthetic training data as the
questions are originally from NQ). In this work, we focus on building a
unified QA system across languages and datasets, and thus use the same
checkpoints for all of the experiments.

hyperparameter |
---|---
max source length | 1,000
max target length | 25
batch size (per GPU) | 2
label smoothing | 0.1
dropout | 0.1
warmup steps | 500
learning rate | 3e-5
weight decay | 0.001
adam $epsilon$ | 1e-8
max grad norm | 0.1
gradient accumulation steps | 2
Table 10: Hyperparameters of mGEN.

###  B.6 Details of translate-test baseline

We first translate the MKQA and Xor-TyDi QA questions from various languages
to English, use DPR to retrieve the answers from the knowledge source, use the
reader to extract an answer, and then translate the answer back to its
original language.

##### Details of translation models.

Our translation models (to English and from English) are the pre-trained
MarianMT (Junczys-Dowmunt et al., 2018) style OPUS-MT (Tiedemann and
Thottingal, 2020) models available in Transformers library (Wolf et al., 2020)
that are trained on the OPUS corpus (Tiedemann and Nygaard, 2004). Since there
is no MarianMT pre-trained OPUS-MT model from English to Korean on
Transformers, we use the pre-trained base-sized autoregressive transformers
model provided by the authors of Xor-TyDi
QA.161616https://github.com/jungokasai/XOR_QA_MTPipeline.

Some of the newer OPUS-MT models require a prefix of the target language
before each sentence of the source language (English here) when translating
English answers back to the question’s original language, which is usually the
ISO 639-3 language code. For example, if we want to translate Ron Paul from
English to Arabic, we concatenate the prefix “>>ara<<” and the original
sentence together to specify the target language to be Arabic since the opus-
mt-en-ar 171717https://huggingface.co/Helsinki-NLP/opus-mt-en-ar. model
supports multiple target languages. Then, we feed the concatenated result
“>>ara<< Ron Paul” into the translation model and get the translation.

Such prefixes and the models we use for each language are listed in Table 11.

Ar Da De Es To-English Model opus-mt-ar-en opus-mt-da-en opus-mt-de-en opus-
mt-es-en From-English Prefix >>ara<< N/A N/A N/A From-English Model opus-mt-
en-ar opus-mt-en-da opus-mt-en-de opus-mt-en-es Fi Fr He Hu To-English Model
opus-mt-fi-en opus-mt-fr-en opus-mt-afa-en opus-mt-hu-en From-English Prefix
N/A N/A >>heb<< N/A From-English Model opus-mt-en-fi opus-mt-en-fr opus-mt-en-
afa opus-mt-en-hu It Ja Ko Km To-English Model opus-mt-it-en opus-mt-ja-en
opus-mt-ko-en opus-mt-mul-en From-English Prefix N/A N/A N/A >>khm_Latn<<
From-English Model opus-mt-en-it opus-mt-en-jap N/A opus-mt-en-mul Ms Nl No Pl
To-English Model opus-mt-mul-en opus-mt-nl-en opus-mt-gem-en opus-mt-pl-en
From-English Prefix >>zsm_Latn<< N/A >>nno<< >>pol<< From-English Model opus-
mt-en-mul opus-mt-en-nl opus-mt-en-gem opus-mt-en-sla Pt Sv Th Tr To-English
Model opus-mt-ROMANCE-en opus-mt-sv-en opus-mt-th-en opus-mt-tr-en From-
English Prefix >>pt<< N/A >>tha<< >>tur<< From-English Model opus-mt-en-
ROMANCE opus-mt-en-sv opus-mt-en-mul opus-mt-en-trk Vi Zh-cn Zh-hk Zh-tw To-
English Model opus-mt-vi-en opus-mt-zh-en opus-mt-zh-en opus-mt-zh-en From-
English Prefix >>vie<< >>cmn<< >>yue_Hant<< >>cmn_Hant<< From-English Model
opus-mt-en-vi opus-mt-en-zh opus-mt-en-zh opus-mt-en-zh

Table 11: Translation models and prefixes used for the translate-test
baseline.

##### Details of English DPR model.

For the English DPR model, we use the trained retriever and reader models from
Xor-TyDi QA, which can be downloaded from their official
website.181818https://github.com/AkariAsai/XORQA/tree/main/baselines.

###  B.7 Details of BM25 baseline

We use the February 2019 Wikipedia dumps as the knowledge source and retrieval
corpus for our BM25 baseline. We first use wikiextractor to preprocess the
Wikipedia documents, and then use Pyserini (Lin et al., 2021), which relies on
Apache Lucene
8.0.0191919https://lucene.apache.org/core/8_0_0/index.htmlhttps://lucene.apache.org/core/8_0_0/index.html.
to index the documents and retrieve the BM25 results for Xor-TyDi QA and MKQA
questions. We use 2 paragraphs as one basic unit of retrieval where paragraphs
are separated by ‘\n’. We retrieve the top 10 units that have the highest BM25
score. After we retrieve top units, we concatenate those paragraphs and feed
them into a bert-base-multilingual-uncased extractive QA model that predicts
start and end positions. The final answers are determined as the span with the
highest joint probabilities.

French, Hebrew, Khmer, Malay, Polish, Vietnamese and Chinese (Hong Kong) are
either not supported by Apache Lucene or missing from the Wikipedia dumps, and
therefore are not included in the final results.

###  B.8 Details of MT+Mono baseline

We normalize the predicted probabilities from the the BM25 (monolingual)
baseline so that the score will be between 0 to 1. We output the monolingual
baseline’s answer when the probability is higher than a threshold; otherwise,
we output translated answers from the translate-test baseline. In this work,
we set the threshold to 0.5 given the results on the Xor-TyDi QA development
set.

###  B.9 Details of the Closed-book Baseline

Instead of training a sequence-to-sequence model from scratch using question
and answer only data as in Roberts et al. (2020), we use the same mGEN model
as in CORA, and only at inference time do we skip retrieval. We also tested an
mt5-base based sequence-to-sequence model that is trained to generate answers
from questions only, but this model underperformed the inference-only model on
the Xor-TyDi QA development set.

##  Appendix C Additional Results

###  C.1 Results on Xor-TyDi QA Development Set

Table 12 shows the results on the Xor-TyDi QA development set. We clearly
outperform the previous state-of-the-art model, as well as the competitive
baselines, by a large margin across target languages. The scores on Xor-Full
development set are significantly higher than the Xor-Full test set presented
in Table 1. We have found that the proportions of the questions where answers
can be extracted from the target languages’ Wikipedia are significantly higher
than in Xor-Full test set. Our CORA framework improves the performance on
those “in-language” subsets of Xor-Full and get a large performance jump on
the test set.

Models |  Target Language $L_{i}$ F1 | Macro Average
---|---|---
| Ar | Bn | Fi | Ja | Ko | Ru | Te | F1 | EM | BLEU
CORA | 42.9 | 26.9 | 41.4 | 36.8 | 30.4 | 33.8 | 30.9 | 34.7 | 25.8 | 23.3
GMT+GS | 18.0 | 29.1 | 13.8 | 5.7 | 15.2 | 14.9 | 15.6 | 16.0 | 9.9 | 14.9
MT+ Mono | 15.8 | 9.6 | 20.5 | 12.2 | 11.4 | 16.0 | 0.5 | 17.3 | 7.5 | 10.7
MT+DPR | 7.2 | 4.3 | 17.0 | 7.9 | 7.1 | 13.6 | 0.5 | 8.2 | 3.8 | 6.8
BM25 | 18.4 | 14.9 | 18.8 | 12.7 | 12.1 | 13.5 | – | – | – | –
Closed-book | 14.0 | 8.1 | 11.8 | 19.1 | 9.3 | 10.5 | 7.6 | 11.5 | 8.2 | 4.9
Table 12: Performance on Xor-Full (development data F1 scores and macro-
averaged F1, EM, and BLEU scores). “GMT+GS” denotes the previous state-of-the-
art model, which combines Google Custom Search in the target language and
Google Translate + English DPR for cross-lingual retrieval (Asai et al.,
2021). Pyserini does not support Telugu.

###  C.2 EM Scores on Xor-TyDi QA and MKQA

##### EM scores on Xor-TyDi QA.

The EM scores on the Xor-TyDi QA test data are in Table 13. We significantly
outperform all other baselines and previous state-of-the-art models in all
languages except for Korean. We found that in Korean, our model is often
penalized because outputs are correct yet generated in English, not Korean.
The state-of-the-art model ensures that the answers are in Korean using Google
Translate, which helps the system to get high performance in Korean.

##### EM scores on MKQA.

The EM scores on MKQA test set are shown in Tables 14 and 15. CORA outperforms
the other baselines by large margins in all of the languages except for Arabic
and English. Note that EM scores may underestimate the models’ ability of open
retrieval; generated answers may be correct even if they do not have a
matching sub-span in existing documents, existing Wikidata entries, or human
translated answers (Asai et al., 2021).

Models |  Target Language $L_{i}$ EM
---|---
| Ar | Bn | Fi | Ja | Ko | Ru | Te
CORA | 38.4 | 26.6 | 33.1 | 30.1 | 18.9 | 36.3 | 34.6
SER | 23.0 | 16.1 | 18.5 | 6.3 | 9.3 | 6.9 | 14.4
GMT+GS | 22.1 | 10.9 | 13.3 | 3.0 | 20.1 | 11.4 | 9.1
MT+ Mono | 19.1 | 9.0 | 16.7 | 7.1 | 8.3 | 13.1 | 0.5
MT+DPR | 2.5 | 1.5 | 10.4 | 3.3 | 2.9 | 2.5 | 0.5
BM25 | 23.2 | 14.6 | 17.0 | 5.3 | 9.4 | 13.4 | –
Closed-book | 9.6 | 6.7 | 8.8 | 16.8 | 7.8 | 15.5 | 1.9
Table 13: Performance on Xor-Full (test data EM scores). “GMT+GS” denotes the previous state-of-the-art model, which combines Google Custom Search in the target language and Google Translate + English DPR for cross-lingual retrieval (Asai et al., 2021). Concurrent to our work, “SER” is the current state-of-the-art model, Single Encoder Retriever, submitted anonymously on July 14 to the Xor-Full leaderboard (https://nlp.cs.washington.edu/xorqa/). Pyserini does not support Telugu.  Setting | – |  Included in Xor-TyDi QA | Seen by mDPR and mGEN
---|---|---|---
|  Avg. over all $L$. | En | Ar | Fi | Ja | Ko | Ru | Es | Sv | He | Th
CORA | 17.2 | 31.2 | 7.7 | 21.8 | 7.4 | 7.3 | 13.8 | 25.8 | 26.0 | 10.7 | 4.8
MT+Mono | 9.6 | 32.1 | 3.9 | 12.6 | 3.8 | 4.0 | 6.5 | 15.2 | 14.4 | 4.4 | 5.0
MT+DPR | 10.2 | 32.1 | 8.5 | 15.0 | 3.6 | 2.7 | 8.6 | 18.6 | 4.4 | 2.8 | 4.6
BM25 | – | 12.8 | 3.1 | 6.5 | 2.7 | 3.9 | 4.7 | 9.4 | 6.6 | – | 2.6
Closed | 2.3 | 3.4 | 2.0 | 2.0 | 3.1 | 2.0 | 2.3 | 3.1 | 4.4 | 2.0 | 2.6
Table 14: EM scores on MKQA seen languages over 6.7k questions with short
answer annotations.

Setting Seen in mGEN Unseen Da De Fr It Nl Pl Pt Hu Vi Ms Km No Tr cn hk tw
CORA 25.8 25.4 25.4 24.0 26.5 21.4 23.0 15.4 17.7 23.2 4.8 24.3 18.5 4.1 5.5
4.5 MT+Mono 13.2 15.8 15.4 15.3 15.3 14.6 13.9 11.5 7.4 3.9 0.1 11.6 11.2 3.5
2.8 4.1 MT+DPR 16.9 16.9 15.4 16.1 17.5 14.6 14.6 10.3 7.4 3.9 0.1 13.6 8.7
1.1 2.8 3.6 BM25 5.8 8.7 – 9.4 8.4 – 8.7 4.8 – – – 5.6 5.6 2.5 – 2.8 Closed
2.6 3.2 2.6 2.6 2.8 2.4 2.6 2.3 2.2 2.3 1.4 2.3 1.7 2.1 1.8 1.9

Table 15: EM scores on MKQA in languages unseen by mDPR and not included in
$\mathbf{C}^{multi}$.

“cn”: “zh-cn” (Chinse, simplified). “hk”: “zh-hk” (Chinese, Hong Kong). “tw”:
“zh-tw” (Chinese, traditional).

##  Appendix D Further Analysis

###  D.1 Visualizing the Document Embedding Spaces

We plot two dimensional encoded document representations (PCA) for the
corresponding articles in Fig. 5. The gray dots concentrated in the lower
right part in the first figure represent encoded Thai embeddings. As we can
see from the plot before cross-lingual training, the Thai document embeddings
are far apart from other languages’ embeddings. On the other hand, after
iterative training (Fig. 5(b)), the embeddings from many languages get closer,
though we can still see loose clusters of some languages.

(a) Document embeddings before cross-lingual training.

(b) Document embeddings after iterative training.

Figure 5: Embeddings before and after cross-lingual training (PCA).

###  D.2 Analysis on the Cross-lingual Retrieval

##### Spanish paragraphs retrieved for Norwegian questions.

The Spanish paragraphs retrieved for Norwegian questions are shown in Table
16. As we can see, CORA retrieves the Spanish passages relevant to the given
Norwegian questions and generate correct answers. Although those two languages
both belong to the Indo-European family and use Latin script, their
typological properties (e.g., syntax and vocabulary) differ significantly.

Query |  Paragraph |  Gold Answer
---|---|---
hvem spilte maria magdalena i jesus christ superstar (trans: who played mary magdalene in jesus christ superstar) |  Los actores principales de la película eran Ted Neeley en el papel de Jesús, Carl Anderson en el de Judas e Yvonne Elliman en el papel de María Magdalena. (trans: The main actors in the film were Ted Neeley as Jesus, Carl Anderson as Judas, and Yvonne Elliman as Mary Magdalene. ) |  Yvonne Elliman
hvor mange episoder er det i andre sesong av my hero academia (trans: how many episodes are in season two of my hero academia) |  El estreno fue 7 de abril del 2018. Contando con un total de 25 episodios igual que la segunda. (trans: The premiere was April 7, 2018. With a total of 25 episodes the same as the second.) |  25.0 episodes
Table 16: Cross-lingual retrieval examples between Norweigian questions and
Spanish passages that lead to correct answers.

###  D.3 Analysis on Errors on MKQA Data.

##### Details of the annotation process.

We randomly sample 50 errors from Spanish and Japanese, and we classify those
errors into five categories described in § 4.2. Each sample includes: a
question in the target language, a question in English (the original NQ
question), the top one passage retrieved by mDPR, an answer generated by mGEN,
and gold answers. The error analysis is conducted by bilingual or native
speakers of Spanish or Japanese.

##### Error analysis results on Chinese examples.

We also conduct the same analysis in Chinese to understand the relatively low
performance in the three Chinese languages. Among the three Chinese variants,
we choose simplified Chinese (Zh-cn). The error analysis is conducted by a
native speaker. Table 17 shows the error analysis results in Japanese,
Spanish, and Chinese. Generating answers in different languages is common in
Chinese like Japanese. Retrieval errors account for 70% of the errors in
Chinese, which is significantly higher than the proportions in Japanese or
Spanish. This can be explained by the fact that we do not include Chinese
passages in our $\mathbf{C}^{multi}$, and thus CORA always has to conduct
cross-lingual retrieval to get evidence to answer. Retrieving documents cross-
lingually is more challenging than retrieving documents monolingually, and
future work can improve cross-lingual retrieval particularly between languages
that are distant from each other.

| Japanese | Spanish | Chinese (simplified)
---|---|---|---
retrieval error | 28 | 48 | 70
different lang | 18 | 0 | 10
incorrect answer | 22 | 36 | 4
annotation error | 22 | 12 | 4
underspecified question | 10 | 4 | 12
Table 17: Error categories (%) on 50 errors sampled from Japanese, Spanish and
Chinese (simplified) data. Error questions are sampled from the MKQA
evaluation data.

###  D.4 More Qualitative Examples

##### Examples errors in MKQA Japanese questions.

Table 18 shows examples of (b) a generation language error (different lang),
(c) incorrect answer generation (incorrect answer), and (d) an answer
annotation error (annotation error). The first example shows an error of
different language where generated text is not in the target language. Such
errors are prevalent in Japanese, especially when retrieved passages are
written in languages with Latin script. Transliteration of foreign words into
Japanese is challenging as there are multiple ways to map English words to
Japanese type script (i.e., katakana). Future work can improve those cross-
lingual generations between languages with their own type script and the ones
with Latin script. In the second example, CORA answers the song writers,
instead of answering who sings the song. This shows even state-of-the-art
models still exploit certain (spurious) patterns or lexical overlap to the
question (Sugawara et al., 2018). The final example demonstrates the
annotation difficulty of covering all possible answer aliases for multilingual
open QA. Although the predicted answer is semantically correct, it’s not
covered by the gold answer annotations in MKQA.

We also show questions that are judged as (e) underspecified questions in
Table 19 in this analysis. The first two examples show a question with
temporal dependency and an ambiguous question. In the final example, we found
that the information about the population of Utica, New York is different in
English Wikipedia (60,635) and Japanese Wikipedia (62,235), and CORA generates
an answer based on the information in Japanese.

error type Query Paragraph Prediction [gold Answer] different lang マルコム in the
Middleで父親役は誰でしたか (trans: who played the dad on malcolm in the middle?) The
series is about a boy named Malcolm (Frankie Muniz), the third-born child in a
comically dysfunctional working-class family of four, and later, five boys,
the sons of Lois (Jane Kaczmarek) and Hal (Bryan Cranston). Bryan Cranston
[ブライアン・クランストン (trans:Bryan Cranston)] incorrect answer 「愛はとまらない」を唄っているのは誰ですか
(trans: who sings nothing’s gonna stop us now?)
愛はとまらないはアルバート・ハモンドとダイアン・ウォーレンの共作による楽曲。アメリカ合衆国のロックバンド、 スターシップにより録音された。
(trans:Nothing’s Gonna Stop is a song co-written by Albert Hammond and Diane
Warren. Recorded by the American rock band Starship.) アルバート・ハモンドとダイアン・ウォーレン
(trans: Albert Hammond and Diane Warren) [スターシップ, Starship] annotation error
マクドナルドの最初の店舗はどこ？ (trans: where was the very first mcdonald’s built?)
最初のマクドナルドはアメリカ合衆国・カリフォルニア州サンバーナーディノでマクドナルド兄弟が1940年に始めたものである。 (trans: The first
McDonald’s was started in 1940 by the McDonald’s brothers in San Bernardino,
California, United States.) アメリカ合衆国・カリフォルニア州サンバーナーディノ [アメリカ合衆国, カリフォルニア州,
サンバーナーディノ, アメリカ]

Table 18: Examples of the Japanese error cases. “trans” denotes the English translation. Sub type |  Query |  Prediction [gold Answer]
---|---|---
temporal dependency |  今年のスーパーボウルはどこでありますか (trans: where is the super bowl being played at this year) |  ルイジアナ州ニューオーリンズ [アトランタ]
ambiguous questions |  トワイライトシリーズの本を教えてください (trans: what are the books in the twilight series) |  ステファニー・メイヤー [エクリプス/トワイライト・サーガ, エクリプス, ニュームーン]
inconsistency between Wikipedias |  ニューヨーク州ユーティカの人口はどのくらいですか。 (trans: what is the population of utica new york) |  62,235人 [60635]
Table 19: Examples of questions labeled as underspecified questions in our
error analysis.

##### Unanswerable MKQA questions that CORA could answer.

Table 20 shows unanswerable Japanese MKQA questions for which CORA can
successfully find correct answers from non-English languages’ text. Although
MKQA answers are carefully annotated by crowd workers who extensively search
online knowledge sources in English, around 30% of the questions remain
unanswerable. Among the valid unanswerable questions, we found that in about
20% of the unanswerable questions we can find correct answers by retrieving
evidence passages in another language (e.g., Japanese, Spanish). This
indicates the effectiveness of cross-lingual retrieval to improve answer
coverage.

Query Paragraph Prediction オレンジ・イズ・ニュー・ブラックはいつ放送される？ (trans:when is orange is
the new black on?)
『オレンジ・イズ・ニュー・ブラック』は、2013年7月11日よりネットフリックスで配信開始されているアメリカのテレビドラマ (trans:“Orange
is the New Black” is an American TV drama that has been available on Netflix
since July 11, 2013.) 2013年7月11日 hulkを演じる役者は誰 (trans: who is the actor that
plays the hulk?) Bruce Banner es un personaje interpretado primero por Edward
Norton y actualmente por Mark Ruffalo en la franquicia cinematográfica Marvel
Cinematic Universe (MCU) basado en el personaje de Marvel Comics del mismo
nombre y conocido comúnmente por su alter ego, Hulk (trans: Bruce Banner is a
character played first by Edward Norton and currently by Mark Ruffalo in the
Marvel Cinematic Universe (MCU) film franchise based on the Marvel Comics
character of the same name and commonly known by his alter ego, Hulk.) Mark
Ruffalo 現在の火星の気温は？ (trans: what’s the temperature on mars right now?)
現在の火星の表面での年平均気温は、210K以下であり (trans:The current average annual temperature on
Mars is less than 210K.) 210K以下

Table 20: Examples of unanswerable Japanese MKQA questions where CORA
successfully finds the correct answers. The answers are validated by the
authors of this paper.


# QKD Entity Source Authentication: Defense-in-Depth for Post Quantum
Cryptography

John J. Prisco
CEO, Safe Quantum, Inc.,
Chairman of the QED-C Use Cases Technical Advisory Committee
<EMAIL_ADDRESS>

###### Abstract

Quantum key distribution (QKD) was conceived by Charles Bennett and Gilles
Brassard in December of 1984.[1] In the ensuing 39 years QKD systems have been
deployed around the world to provide secure encryption for terrestrial as well
as satellite communication. In 2016 the National Institute of Standards and
Technology (NIST) began a program to standardize a series of quantum resistant
algorithms to replace our current encryption standards thereby protecting
against future quantum computers breaking public key cryptography. This
program is known as post quantum cryptography or PQC. One of the tenets of
cybersecurity is to use an approach that simultaneously provides multiple
protections known as defense-in-depth. This approach seeks to avoid single
points of failure. The goal of this paper is to examine the suitability of a
hybrid QKD / PQC defense-in-depth strategy. A focus of the paper will be to
examine the sufficiency of initial QKD hardware authentication (entity source
authentication) which is necessary to guard against man-in-the-middle attacks.

##  1 Executive Summary

  * •

Today the security of our information systems is more important than ever for
personal, economic, and national security reasons. At the same time,
attackers, both individual and state sponsored, are continually seeking and
finding vulnerabilities.

  * •

Encryption techniques based on factoring are widely used to protect online
transactions. However, Shor’s algorithm for factoring large numbers can break
this type of encryption once a quantum computer that is sufficiently powerful,
also known as a cryptographically relevant quantum computer (CRQC), is
available.

  * •

One approach to address the threat posed by CRQCs is to implement new
cryptographic standards that are, to the best of our knowledge, not
vulnerable. NIST, in coordination with the National Security Agency (NSA), is
focused on selecting PQC algorithms (which are quantum-resistant mathematical
algorithms) to become the new standard for quantum-resistant encryption.

  * •

This project has been conducted in a public vetting process since 2016 and is
expected to conclude with a new standard within the next year. To select
algorithms with more diversity than the current batch of lattice-based
algorithms, NIST reopened its submission process for new proposals last year
and recently reported that it had received 40 new qualified proposals before
its June 1, 2023 submission deadline.

  * •

Another approach to address the threat of CRQCs is to take advantage of
characteristics of quantum mechanics that enable the detection of the presence
of an eavesdropper and therefore provide assurance of secure exchange of
information. This approach is known as Quantum Key Distribution (QKD).

  * •

QKD is a secure communication method for exchanging encryption keys only known
between shared parties. It uses properties found in quantum physics to
exchange cryptographic keys in such a way that is provable and guarantees
security. QKD enables two parties to produce and share a key that is used to
encrypt and decrypt messages. Specifically, QKD is the method of distributing
the keys between parties.[2] Benefits of QKD include a reduced vulnerability
to increasing computational power and an immediate detection of the presence
of eavesdroppers.

Known limitations of QKD include those described by National Security Agency
researchers who in 2020 published a paper entitled “Quantum Key Distribution
(QKD) and Quantum Cryptography (QC)” [3] in which they describe five reasons
why QKD needs further study and development.

  1. 1.

Quantum key distribution is only a partial solution lacking hardware
authentication.

  2. 2.

Quantum key distribution requires special purpose equipment.

  3. 3.

Quantum key distribution increases infrastructure costs and insider threat
risks.

  4. 4.

Securing and validating quantum key distribution is a significant challenge.

  5. 5.

Quantum key distribution increases the risk of denial-of-service attacks.

In this paper we primarily examine and propose a defensible QKD authentication
solution to address, in part, the concerns raised by NSA. Our ultimate goal is
to add QKD to a post quantum computer defense-in-depth strategy alongside PQC.
Achieving this goal will require time and development of QKD performance.
Currently, there are no QKD resources at NIST/NSA paralleling the PQC effort.
We advocate for them to join in the advancement of QKD and to gain their
endorsement.

We recommend a paper [4] written by Renato Renner and Ramona Wolf in the
Arxiv: entitled “The debate over QKD: A rebuttal to the NSA’s objections”
(arxiv.org). This paper offers thoughts about overcoming NSA’s concerns with
QKD.

##  2 Importance of authentication for QKD security

When implementing secure authentication for QKD systems, sufficient care must
be taken to achieve information-theoretic security for the system. If the
authentication key is not information-theoretically secure, an attacker can
break it and thereby bring all classical and quantum communications under the
attacker’s control with the objective of relaying the information by means of
a man-in-the-middle attack.

In the NSA paper previously sited, there is concern with initial
authentication or “entity source authentication”. For QKD to be secure, the
classical messages used in forming the key must be authenticated. It is not
necessary to authenticate the qubit communication. Quantum mechanics offers
information security assurances that cannot be achieved classically, such as
preserving the confidentiality and privacy of communications when it is
intercepted by sophisticated adversaries with virtually unlimited computing
resources. Due to the “no cloning” theorem of quantum mechanics, it is not
possible to intercept quantum communications without the destruction of the
information carried, leading to noticeable reduction of communication quality
and detection by the communicating parties.

With the technology currently available, each quantum transmitter and receiver
must be specifically tuned to function properly. If an injection attack or
man-in-the-middle attack were to occur in the quantum channel, then the
network operators would immediately become aware because the affected network
links would be out of tolerance upon the change and most likely stop working
all together.

However, there still exists potential attack scenarios where sophisticated
attackers may be able to spoof, tamper or exploit QKD systems to compromise
its communication. In order to guard against potential attack scenarios, it is
essential to securely provide initial authentication or “entity source
authentication” of the QKD hardware. For QKD to be secure, the classical
messages used in forming the keys must be authenticated.

###  2.1 Information theoretical authentication

The QKD protocol that we are examining is BB84 protocol developed by Charles
Bennett and Gilles Brassard in December 1984. The following, in italics, is
from their paper describing the initial entity source authentication which we
rely on in our approach to identify Alice and Bob hardware for a QKD system.

_In quantum public key distribution, the quantum channel is not used directly
to send meaningful messages, but is rather used to transmit a supply of random
bits between two users who share no secret information initially, in such a
way that the users, by subsequent consultation over an ordinary non-quantum
channel subject to passive eavesdropping, can tell with high probability
whether the original quantum transmission has been disturbed in transit, as it
would be by an eavesdropper (it is the quantum channel’s peculiar virtue to
compel eavesdropping to be active). If the transmission has not been
disturbed, they agree to use these shared secret bits in the well-known way as
a one-time pad to conceal the meaning of subsequent meaningful communications,
or for other cryptographic applications (e.g., authentication tags) requiring
shared secret random information. If transmission has been disturbed, they
discard it and try again, deferring any meaningful communications until they
have succeeded in transmitting enough random bits through the quantum channel
to serve as a one-time pad. The need for the public (non-quantum) channel in
this scheme to be immune to active eavesdropping can be relaxed if the Alice
and Bob have agreed beforehand on a small secret key, which they use to create
Wegman-Carter authentication tags for their messages over the public channel.
In more detail the Wegman-Carter multiple message authentication scheme uses a
small random key to produce a message-dependent ’tag’ (rather like a check
sum) for an arbitrary large message, in such a way that an eavesdropper,
ignorant of the key, has only a small probability of being able to generate
any other valid message-tag pairs. The tag thus provides evidence that the
message is legitimate, and was not generated or altered by someone ignorant of
the key. (Key bits are gradually used up in the Wegman-Carter scheme, and
cannot be reused without compromising the system’s provable security; however,
in the present application, these key bits can be replaced by fresh random
bits successfully transmitted through the quantum channel.) The eavesdropper
can still prevent communication by suppressing messages in the public channel,
as of course he can by suppressing or excessively perturbing the photons sent
through the quantum channel. However, in either case, Alice and Bob will
conclude with high probability that their secret communications are being
suppressed, and will not be fooled into thinking their communications are
secure when in fact they’re not._

This is meant to solve the problem of “we have negotiated a key securely over
the quantum channel, but we don’t know who we have negotiated a key with”
because one now knows who specifically the bits came from through the Wegman-
Carter authentication tags described in the above excerpt from the Bennett and
Brassard paper. This solution works best for point-to-point links or simple
QKD networks, with a small number of participants. It offers the best possible
security. However, when the number of potential pairs of users increases, it
can quickly become too complex. This is why another approach can be used.

###  2.2 Authentication with hash-based signatures

The most secure authentication algorithms, which do not rely on the complex
mathematical problems needed by other PQC algorithms are the hash-based
signatures. The security of hash-based signature relies on the existence of
one-way functions alone (and, of course, on correct implementations). Several
candidates have been considered. Stateful signatures are probably the fastest
and smallest, but, since the users must keep track of each signature, in order
to prevent re-use of the same keys, these are plagued with serious
implementation issues. Therefore stateless signatures, such as Sphincs+, have
been proposed. Sphincs+ is one of the signatures algorithms under
standardization from NIST.

###  2.3 Authentication with other PQC signatures

The other signature schemes under consideration rely on the existence of a
one-way function with trapdoors. The immediate question, which may be asked
when relying on PQC signatures for authenticating QKD, is that, if we need PQC
for signatures, why do we need QKD at all? We could indeed use PQC for both
authentication and key exchange. The reason why we believe that PQC for
signature and QKD for key exchange still makes complete sense is a question of
timing. Authentication has to be safe up to the time of the key exchange. On
the other hand, confidentiality has to be safe after the exchange, until the
time of validity of the encrypted information. If we trust an authentication
PQC algorithm to be safe today, we can safely use it. We have to be sure that
a key exchange PQC algorithm stays safe for a potentially much longer period.
This is what is offered by QKD. Moreover, having defense-in-depth with QKD and
PQC eliminates a single point of failure.

After reviewing our approaches with cryptography experts from MITRE, the
University of Chicago and the University of South Florida we settled upon an
authentication approach which uses the PQC standards finalist CRYSTALS
Dilithium. We believe this to be an excellent choice because we are proposing
a hybrid of PQC and QKD to solve the authentication concerns and achieve the
long-term confidentiality offered by QKD.

##  3 Conclusions

No one can afford to rebuff any quantum-safe security solution. Europe, Japan
and China have robust QKD systems and trials in place. Shouldn’t the United
States be as informed as our international friends and foes?

No security expert would ever recommend to use just one system or rely on just
one approach. A defense-in-depth strategy deploying more than one kind of
encryption can provide the layers of protection that will prove resilient.

QKD Entity Source Authenticated is an important part of quantum security. QKD
Entity Source Authentication, as described in the proposed protocol of this
paper, solves a major concern in the overall security of QKD. Currently, there
are no QKD resources at NIST/NSA paralleling the PQC effort. We advocate for
them to join in the advancement of QKD and gain their endorsement.

##  4 Acknowledgments

I would like to thank the following individuals who contributed greatly to
this paper.

Carl Dukatz, Computing Subcommittee Lead of the QED-C Use Cases TAC and
Accenture Managing Director-Global Lead for Quantum Computing at Accenture,
William Trost, Member of the QED-C Communications and Security Subcommittee of
the Use Cases TAC and Lead Member of Technical Staff at AT&T, Chief Security
Office (CSO), Quantum Computing Cybersecurity Initiative, Kirk McGregor,
Member of the QED-C Communications and Security Subcommittee of the Use Cases
TAC and Chief Strategy Officer at Iff Technologies, and affiliate researcher
with the Lung Center in the Department of Internal Medicine, School of
Medicine at the UC Davis Medical Center and with Expolab in the Department of
Computer Science at UC Davis. Elizabeth Wood, Member of the QED-C
Communications and Security Subcommittee of the Use Cases TAC and Professional
in the AT&T Chief Security Office (CSO), Quantum Computing Cybersecurity
Initiative.

Special thanks to Celia Merzbacher, Executive Director of the Quantum Economic
Development Consortium (QED-C) for her helpful review of this paper.

## References

  * [1] Charles H.Bennett and Gilles Brassard.  Quantum cryptography: Public key distribution and coin tossing, 1984.
  * [2] Alexander S.Gillis.  What Is Quantum Key Distribution (QKD)?  https://www.techtarget.com/searchsecurity/definition/quantum-key-distribution-QKD.
  * [3] NSA.  Quantum Key Distribution (QKD) and QuantumCryptography (QC).  https://www.nsa.gov/Cybersecurity/Quantum-Key-Distribution-QKD-and-Quantum-Cryptography-QC/.
  * [4] Renato Renner and Ramona Wolf.  The debate over qkd: A rebuttal to the nsa’s objections, 2023.