# Multifaceted Improvements for Conversational Open-Domain Question Answering

Tingting Liang  Hangzhou Dianzi UniversityHangzhouChina<EMAIL_ADDRESS>,
Yixuan Jiang  Hangzhou Dianzi UniversityHangzhouChina<EMAIL_ADDRESS>,  Congying Xia  University of Illinois at ChicagoChicagoUS<EMAIL_ADDRESS>,
Ziqiang Zhao  Hangzhou Dianzi UniversityHangzhouChina<EMAIL_ADDRESS>,
Yuyu Yin  Hangzhou Dianzi UniversityHangzhouChina<EMAIL_ADDRESS>and
Philip S. Yu  University of Illinois at ChicagoChicagoUS<EMAIL_ADDRESS>
(2018)

###### Abstract.

Open-domain question answering (OpenQA) is an important branch of textual QA
which discovers answers for the given questions based on a large number of
unstructured documents. Effectively mining correct answers from the open-
domain sources still has a fair way to go. Existing OpenQA systems might
suffer from the issues of question complexity and ambiguity, as well as
insufficient background knowledge. Recently, conversational OpenQA is proposed
to address these issues with the abundant contextual information in the
conversation. Promising as it might be, there exist several fundamental
limitations including the inaccurate question understanding, the coarse
ranking for passage selection, and the inconsistent usage of golden passage in
the training and inference phases. To alleviate these limitations, in this
paper, we propose a framework with Multifaceted Improvements for
Conversational open-domain Question Answering (MICQA). Specifically, MICQA has
three significant advantages. First, the proposed KL-divergence based
regularization is able to lead to a better question understanding for
retrieval and answer reading. Second, the added post-ranker module can push
more relevant passages to the top placements and be selected for reader with a
two-aspect constrains. Third, the well designed curriculum learning strategy
effectively narrows the gap between the golden passage settings of training
and inference, and encourages the reader to find true answer without the
golden passage assistance. Extensive experiments conducted on the publicly
available dataset OR-QuAC demonstrate the superiority of MICQA over the state-
of-the-art model in conversational OpenQA task.

Open-Domain, Conversational Question Answering, Curriculum Learning

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: Woodstock ’18: ACM Symposium on Neural
Gaze Detection; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer systems organization
Embedded systems††ccs: Computer systems organization Redundancy††ccs: Computer
systems organization Robotics††ccs: Networks Network reliability

##  1\. Introduction

(a)

(b)

(c)

Figure 1. Illustration of three limitations and the corresponding solutions
(marked with the blue dashed boxes). (a) Inaccurate question understanding:
Feeding rewrite to the encoder might lead to poor performance in question
understanding as the encoder can not learn how to model the concatenated
question. (b) Coarse ranking: The relevant passage may not be picked up even
it is retrieved in the initial ranking list which is roughly obtained by a
frozen passage encoder and learnable question encoder. (c) Inconsistent usage
of golden passage: invariably adding golden passage for training would
increase dependency on it and be adverse to inference where no golden passage
is manually added.

Open-domain question answering (OpenQA) aims to discover answers from an
extremely large text source such as Wikipedia for given questions (Chen et
al., 2017; Yang et al., 2019; Zhu et al., 2021). OpenQA that receives the more
widespread application as its setting is more aligned with real-world QA
process of human beings. Generally, an OpenQA system needs to firstly locate a
small collection of relevant articles and then generate the answer. The widely
adopted architecture of the existing OpenQA system consists of two components,
retriever and reader. The former acts as an information retrieval system to
find relevant passages probably with the correct answer contained (Kratzwald
and Feuerriegel, 2018; Lee et al., 2019; Xiong et al., 2020; Min et al., 2019;
Izacard and Grave, 2021). The latter aims at extracting or generating the
answers from the retrieved passages. Although OpenQA enjoys a sound
development momentum, it is challenged by several issues including question
complexity and ambiguity, as well as insufficient background knowledge.
Conversational OpenQA which executes question answering with open-domain
sources under the conversational setting is able to address the above issues
by providing the context information in the conversations.

Previous work (Qu et al., 2020) firstly defines the task of open-retrieval
conversational question answering (ORConvQA) and proposes an effective end-to-
end system to deal with it. Promising as it might be, several fundamental
limitations still exist: (1) Inaccurate question understanding. The questions
in the dialogues might be hard to understand due to the unspecific pronouns.
To alleviate this issue, ORConvQA uses rewrite questions by replacing pronouns
with real entities to pre-train the retriever. Then, they concatenate the
dialogue history and the question together to jointly train the whole model.
However, the input of the pre-traing and the joint training process is still
often inconsistent which makes the model incapable of really understand those
unclear questions. (2) Coarse ranking. Previous works adopt the retriever-
reader pipeline which consists of a passage retriever and answer reader with a
reranker/selector as regularization (Yang et al., 2019; Karpukhin et al.,
2020; Qu et al., 2020). The reranker/selector here does not influence the
retriever and the reader only takes the top-ranked passages as the input. This
makes the whole pipeline suffer from the coarse ranking, especially for the
situations when golden passages are not retrieved in the top-ranked passages.
(3) Inconsistent usage of golden passage. During training, ORConvQA adds the
golden passage into the training manually when the golden passage is not
retrieved in the top-ranked passages. This is impractical for the testing as
the golden passages are unknown in the inference stage. The inconsistency
usages of golden passage during training and testing might lead to poor
performance when the golden passage cannot be accurately retrieved.

To address the previous issues, we propose a framework which has Multifaceted
Improvements for Conversational open-domain Question Answering (MICQA). MICQA
improves the previous work from multiple aspects including regularizing
retriever pre-training, as well as incorporating post-ranking and curriculum
learning. First, Figure 1(a) shows that, a Kullback_Leibler (KL)-divergence
based regularization is proposed for retriever pre-training, which constrains
the retrieved results outputted by feeding different forms of questions. In
this manner, we can learn a better representation for the concatenated
question by taking its rewrite as supervision and minimizing the information
loss between them. The better question understanding would benefit the
retrieval and question answering performance in different phases. Second, to
further improve the initial retrieval results, we apply a multi-stage pipeline
by adding a post-ranker module as shown in Figure 1(b), which also known as
the post-processing (Zhu et al., 2021) over retrieved passages generated by
the retriever. The post-ranker learning benefits from a two-aspect constrain
posed by a ranking loss with a distance-based contrastive loss. In this way,
the relevant passages would be enforced by post-ranker to appear at the top
positions and selected for answer reading. Third, in the joint training stage,
we design a semi-automatic curriculum learning strategy (the blue dashed boxes
in Figure 1(c)) to reduce the dependency on the golden passages. It encourages
the model to find true answer spans even when the golden passage are not
retrieved as the top-ranked passages. We evaluate MICQA on the publicly
released dataset OR-QuAC (Qu et al., 2020) to demonstrate its effectiveness.

Our main contributions are as follows:

  * •

We propose a KL-divergence based regularization for retriever pre-training,
which greatly strengthens the capability of question understanding.

  * •

We propose a multi-stage pipeline for the conversational OpenQA task by
incorporating a post-ranker module in the conventional retriever-reader
pipline.

  * •

We design a curriculum learning strategy for the joint training of the
retriever, post-ranker, and reader, which effectively closes the gap between
training and inference.

  * •

We conduct extensive experiments on the OR-QuAC dataset to demonstrate the
effectiveness of the proposed MICQA. We further provide in-depth analysis on
model ablation and case study to explore the insights of MICQA.

Figure 2. The overall framework of the MICQA model. The retriever equipped
with one question encoder and one passage encoder searches the top $K$
passages from the large corpus of Wikipedia for the given question reformatted
as $q_{i}^{en}$. The representations of $K$ passages are passed into the post-
ranker to produce a refined ranking list of passages, of which top $T$ are
concatenated with the question reconstructed as $q_{i}^{rd}$ to be the input
for the reader. The reader is used to extract the final answer according to
the post-ranker score and reader score.

##  2\. Preliminary

###  2.1. Retriever-Reader Pipeline

The most typical OpenQA system follows a two-stage pipeline, which has two
components: one retriever and one reader.

####  2.1.1. Retriever

In the retrieval process, the commonly used dual-encoder model (Bromley et
al., 1993) consists of a question encoder and a passage encoder, which encode
the question and passage into low-dimensional vectors, respectively. Many
similarity functions, such as inner product and Euclidean distance, can be
used to measure the relationship between questions and passages. The
evaluation in (Karpukhin et al., 2020) demonstrates that these functions
perform comparably, so the simpler inner product is selected in our work:

(1) $\displaystyle\mathrm{sim}(q,p)=E_{Q}(q)^{\mathrm{T}}E_{P}(p).$

where $q$ and $p$ denote a given question and passage. $E_{Q}$ and $E_{P}$
refer to the question encoder and passage encoder which typically are two pre-
trained model, e.g., BERT(Devlin et al., 2019), ALBERT(Lan et al., 2020). The
retriever score is defined as the similarly of representations of the question
and passage. Given a question $q$, the retriever derives a small subset of
passages with embeddings closest to $q$ from a large corpus.

####  2.1.2. Reader

In reading phase, a conventional BERT-based extractive machine comprehension
model is usually used as the reader. The retrieved passages concatenated with
the corresponding questions are first encoded separately. The reader then
extracts the start and end tokens with the max probabilities among tokens from
all the the top passages. The reader score is defined as the sum of the scores
of start token and end token. The answer score is the sum of retriever score
and reader score. Taking the small collection of passages generated by the
retriever as input, the reader outputs the answer span with the highest answer
score.

###  2.2. Problem Definition

Following the task definition in (Qu et al., 2020), the conversational OpenQA
problem can be formulated as follows: Given the current question $q_{c}$, and
a set of historical question-answer pairs $\\{(q_{i},a_{i})\\}^{c-1}_{i=1}$,
where $q_{i}$ denotes the $i$-th question and $a_{i}$ is the related answer in
a conversation, the task is to identify answer spans for the current question
$q_{c}$ from a large corpus of articles, _e.g.,_ Wikipedia.

Based on the introduced two-stage pipeline, this work applies a multi-stage
pipeline of one pretrained retriever, one post-ranker, and one reader. The
details of the framework and training process of MICQA are introduced in the
next section.

##  3\. Model

The overall framework of our MICQA is shown in Figure 2. MICQA mainly consists
of three components: (1) one regularized retriever for relevant passages
discovery, (2) one contrastive post-ranker for improving passage retrieval
quality, and (3) one reader for detecting answers from a small collection of
ranked passages. These three components work as follows: With the passage
representations offline encoded by the pre-trained passage encoder, the
retriever outputs the top $K$ passages by operating inner product between them
and the embedding of a given question generated by the question encoder. The
post-ranker takes the initially retrieved top $K$ passage embeddings and
question embedding as input, and selects the top $T$ of the newly ranked
passages as output. The outputted passages are respectively concatenated with
the given question, and the concatenations are fed into the reader to find the
final answer. The answer score is decided by the post-ranker score and reader
score, which would be introduced in details in Section 3.3. The retriever is
pre-trained with a KL-divergence based regularization. Then the passage
encoder of it is frozen and the question encoder can be fine-tuned with the
other two components in the next joint training process in an end-to-end
manner. In the following part, we would describe the procedures of pre-
training, joint training, and inference, along with the details of each
component.

###  3.1. KL-Divergence based Regularization for Retriever Pre-training

In order to improve the capability of question understanding, we propose to
exploit both the original questions and question rewrites in retriever pre-
training. Specifically, a regularization mechanism is proposed to force two
distributions of the retrieved results generated by feeding the original
questions and their rewrites into the dual-encoder to be consistent with each
other by minimizing the bidirectional KL divergence between them.

Concretely, given the current question $q_{c}$ and its historical question-
answer pairs $\\{q_{i},a_{i}\\}_{i=1}^{c-1}$, we form the original question by
concatenating the question-answer pairs in a history window of size $w$ with
the current question. Moreover, to mitigate the issue of underspecified and
ambiguous initial questions, the initial question $q_{1}$ is invariably
considered as it makes the constructed original question closer to the
question rewrite. For the question encoder, the input sequence of original
question can be defined as
$q_{c}^{or}=\verb|[CLS]|\;q_{1}\;\verb|[SEP]|\;a_{1}\;\verb|[SEP]|\;q_{c-w}\verb|[SEP]|\;a_{c-w}\;\verb|[SEP]|\cdots\verb|[SEP]|\;q_{c-1}\\\
\verb|[SEP]|\;a_{c-1}\;\verb|[SEP]|\;q_{c}\;\verb|[SEP]|$.

Let
$\mathcal{D}=\\{(q_{i}^{or},q_{i}^{rw},p_{i}^{+},p_{i,1}^{-},...,p_{i,n}^{-})\\}_{i=1}^{m}$
denote the training data that consists of $m$ instances. Each instance
contains two forms of one question (_i.e.,_ original question and question
rewrite), one positive passage, along with $n$ irrelevant passages
$\\{p_{i,j}^{-}\\}_{j=1}^{n}$. These irrelevant passages used for training
contain one hard negative of the given question and $n-1$ in-batch negatives
which are the positive and negative samples of the other questions from the
same mini-batch. As shown in Figure 3, we feed the original question
$q_{i}^{or}$ and question rewrite $q_{i}^{rw}$ to the question encoder
$E_{Q}$, respectively. The derived question embeddings are matched with the
passage embeddings outputted by the passage encoder $E_{P}$ via inner product.
We can obtain two distributions of the retrieved results through softmax
operation, denoted as $P^{or}(p_{i}|q_{i}^{or})$ and
$P^{rw}(p_{i}|q_{i}^{rw})$. In the retrieval pre-training phase, we try to
regularize on the retrieved results by minimizing the bidirectional KL
divergence between the two distributions, which can be formulated as follows:

(2) $\displaystyle\mathcal{L}_{KL}^{i}=$
$\displaystyle\frac{1}{2}(\mathcal{D}_{KL}(P^{or}(p_{i}|q_{i}^{or})||P^{rw}(p_{i}|q_{i}^{rw}))$
$\displaystyle+\mathcal{D}_{KL}(P^{rw}(p_{i}|q_{i}^{rw})||P^{or}(p_{i}|q_{i}^{or}))).$

For the main task of passage retrieval, we apply the widely used negative log
likelilhood of the positive passage given two forms of one question as
objective function:

(3) $\displaystyle\mathcal{L}_{NLL}^{i}=$
$\displaystyle-\frac{1}{2}(\mathcal{L}_{NLL\\_or}^{i}+\mathcal{L}_{NLL\\_rw}^{i})$
$\displaystyle=$
$\displaystyle-\frac{1}{2}(\mathrm{log}P^{or}(p_{i}^{+}|q_{i}^{or})+\mathrm{log}P^{rw}(p_{i}^{+}|q_{i}^{rw})),$

where the probability of retrieving the positive passage can be calculated as:

(4)
$P^{or}(p_{i}^{+}|q_{i}^{or})=\frac{\mathrm{exp}(\mathrm{sim}(q_{i}^{or},p_{i}^{+}))}{\mathrm{exp}(\mathrm{sim}(q_{i}^{or},p_{i}^{+}))+\sum_{j=1}^{n}\mathrm{exp}(\mathrm{sim}(q_{i}^{or},p_{i,j}^{-}))},$
(5)
$P^{rw}(p_{i}^{+}|q_{i}^{rw})=\frac{\mathrm{exp}(\mathrm{sim}(q_{i}^{rw},p_{i}^{+}))}{\mathrm{exp}(\mathrm{sim}(q_{i}^{rw},p_{i}^{+}))+\sum_{j=1}^{n}\mathrm{exp}(\mathrm{sim}(q_{i}^{rw},p_{i,j}^{-}))}.$

The probability of retrieving those negative passages can be obtained in the
same manner.

To pre-train the retriever, the training objective is to minimize the pre-
retrieval loss $\mathcal{L}^{i}_{pre}$ for data ($q_{i}^{or}$, $q_{i}^{rw}$,
$p_{i}^{+}$, $p_{i,1}^{-}$, … , $p_{i,n}^{-})$:

(6) $\mathcal{L}^{i}_{pre}=\mathcal{L}_{NLL}^{i}+\alpha\mathcal{L}_{KL}^{i},$

where $\alpha\in[0,1]$ is a hyperparameter used to control
$\mathcal{L}_{KL}^{i}$.

Figure 3. Retriever pre-training model with KL-divergence based
regularization. Two forms of one question are fed into question encoder while
the related positive and hard negative are fed into passage encoder. The
positive or negative passages of the other questions in the same batch are
used as negatives.

###  3.2. Joint Training with Curriculum Learning

With the passage encoder and question encoder of the retriever pre-trained,
our MICQA jointly trains the question encoder, post-ranker, and reader with a
designed curriculum learning strategy.

####  3.2.1. Retriever Loss

The pre-trained passage encoder is used to embed the open-domain passages
offline and obtain a set of passage representations. Given the current
question $q_{c}$, and its historical question-answer pairs
$\\{q_{i},a_{i}\\}_{i=c-w}^{c-1}$ within a window size $w$, the input for
question encoder is constructed by concatenating the question-answer pairs
with the current question as mentioned in Section 3.1. With the prepared
offline passage representations and outputs of question encoder, each passages
would be assigned with a retrieval score computed by the inner product
operation as shown in Equation 1. We select the top $K$ passages with high
retrieval scores for the post-ranker and reader.

For each question, the reformatted question for the question encoder is
denoted as $q_{i}^{en}$. The question encoder of the retriever is fine-tuned
by optimizing the following retrieval loss:

(7)
$\mathcal{L}^{i}_{retriever}=-\mathrm{log}\frac{\mathrm{exp}(\mathrm{sim}(q_{i}^{en},p_{i}^{+}))}{\sum_{j=1}^{K}\mathrm{exp}(\mathrm{sim}(q_{i}^{en},p_{i,j}))},$

where $p_{i,j}$ denotes the retrieved passages regarding question $q_{i}$.

For a fair comparison, we omit the answers from the concatenated question in
the experiment to keep the same setting with the previous work (Qu et al.,
2020).

####  3.2.2. Post-Ranker with Contrastive Loss

Since only the first $T$ of the top $K$ passages outputted by the retriever
module are fed into the reader for answer extraction, a post-ranker is
proposed to re-rank the $K$ passages to push more relevant passages be ranked
in the top $T$ placement. We build the post-ranker by adding a subsequent
network after the pre-trained passage encoder, which takes the embeddings of
$K$ passages as input. With the subsequent network, post-ranker is able to
learn high-level feature representations for passages. In this work, we use
the linear layer as the subsequent network for simplicity. The output for the
retrieved passage $p_{i,j}$ and the ranking score can be formulated as
follows:

(8) $\mathbf{d}_{i,j}=\mathrm{LinearLayer}(E_{P}(p_{i,j})),$ (9)
$S_{post}(q_{i}^{en},p_{i,j})=\mathbf{d}_{q_{i}}^{\mathrm{T}}\mathbf{d}_{i,j},$

where $\mathbf{d}_{q_{i}}$ is the representation for question $q_{i}$
generated by the question encoder $E_{Q}$. $S_{post}(\cdot)$ refers to the
scores generated by the post-ranker network.

To simultaneously fine-tune the question encoder and post-ranker, we apply the
modified hinge loss combined with a distance-based contrastive loss to pose
constraints for passage reranking from two aspects. The hinge loss function
(Santos et al., 2016) is defined as:

(10) $\mathcal{L}^{i}_{ranker}=\mathrm{max}\\{0,\delta-
S_{post}(q_{i}^{en},p_{i}^{+})+\mathrm{max}_{j}\\{S_{post}(q_{i}^{en},p_{i,j}^{-})\\}\\},$

where $\delta$ is the margin of the hinge loss, $p_{i}^{+}$ and $p_{i,j}^{-}$
denote the positive passage and negative passages retrieved for $q_{i}$. For
contrastive learning, we uses the triplet margin loss (Weinberger et al.,
2006) to measure the distance between positive and negative samples as
follows:

(11)
$\mathcal{L}^{i}_{cl}=\mathrm{max}\\{0,\mu+D(\mathbf{d}_{q_{i}},\mathbf{d}^{+}_{i})-D(\mathbf{d}_{q_{i}},\mathbf{d}_{i,j}^{-})\\},\vspace{-0.05in}$

where $\mu$ is the margin of the triplet margin loss. We apply Euclidean
distance $D$ to compute the distance between the question and passage in the
representation space. The final post-ranker loss is defined as:

(12)
$\mathcal{L}^{i}_{postranker}=\mathcal{L}^{i}_{ranker}+\beta\mathcal{L}^{i}_{cl},\vspace{-0.05in}$

where $\beta$ is hyperparameter that need to be determined.

####  3.2.3. Reader Loss

The neural reader of our MICQA is designed for predicting an correct answer to
the given question. Given the top $T$ retrieved/reranked passages, the reader
intends to detect an answer span from each passage with a span score.
Moreover, a passage selection score is assigned to each passage. The passage
selection is used to select the passages that contains the true answer of the
given question (Lin et al., 2018; Karpukhin et al., 2020), which performs as a
contraint to facilitate the training of reader. The span with the highest
passage selection score and span score is extracted as the final answer.

Our reader applies the widely used pre-trained model, such as BERT (Devlin et
al., 2019), RoBERTa (Liu et al., 2019). Given a question $q_{i}$ and the top
$T$ previously retrieved passages, we form the question input in the same way
as that for retriever described in Section 3.2.1. The only difference is that
the initial question $q_{1}$ is left out as the setting of (Qu et al., 2020).
We denote the reconstructed question as $q_{i}^{rd}$ and concatenate it with a
retrieved passage to form the sequence as the input for the pre-trained model.
Specifically, let $\mathbf{z}_{i,j,t}$ be the outputted token-level
representations of the $t$-th token in the passage $p_{i,j}$ which is
retrieved for question $q_{i}^{rd}$. $\mathbf{z}_{i,j,[\mathrm{cls}]}$ denotes
the sequence-level representation for the input sequence. The scores for the
$t$-th token being the start and end tokens, and a passage being selected are
defined as follows:

(13)
$S_{s}(q_{i}^{rd},p_{i,j},[t])=\mathbf{z}_{i,j,t}^{\mathrm{T}}\mathbf{w}_{start},$
(14)
$S_{e}(q_{i}^{rd},p_{i,j},[t])=\mathbf{z}_{i,j,t}^{\mathrm{T}}\mathbf{w}_{end},$
(15)
$S_{select}(q_{i}^{rd},p_{i,j})=\mathbf{z}_{i,j,[\mathrm{cls}]}^{\mathrm{T}}\mathbf{w}_{select},$

where $\mathbf{w}_{start}$, $\mathbf{w}_{end}$, and $\mathbf{w}_{select}$ are
trainable vectors.

The loss function for the start token prediction is defined as:

(16)
$\mathcal{L}^{i}_{start}=-\mathrm{log}\frac{\mathrm{exp}(S_{s}(q_{i}^{rd},p_{i}^{+},[st_{i}^{+}]))}{\sum_{j=1}^{M}\sum_{[t]\in
p_{i,j}}\mathrm{exp}(S_{s}(q_{i}^{rd},p_{i,j},[t]))},$

where $[st_{i}^{+}]$ denotes the start token of the true answer in the golden
passage $p_{i}^{+}$. The objective function for the end token prediction
denoted as $\mathcal{L}^{i}_{end}$ is defined in the same manner. In addition,
the passage selecting loss is computed as follows:

(17)
$\mathcal{L}^{i}_{select}=-\mathrm{log}\frac{\mathrm{exp}(S_{select}(q_{i}^{rd},p_{i}^{+}))}{\sum_{j=1}^{M}\mathrm{exp}(S_{select}(q_{i}^{rd},p_{i,j}))}.$

Finally, the loss function of the reader is defined as follows:

(18)
$\mathcal{L}_{reader}^{i}=\frac{1}{2}(\mathcal{L}_{start}^{i}+\mathcal{L}_{end}^{i})+\mathcal{L}_{select}^{i}.$

####  3.2.4. Joint Training with Curriculum Learning Strategy

We propose a semi-automatic curriculum learning (CL) strategy consisting of
two core components: an automatic difficulty measurer and a discrete training
scheduler, to improve the joint training of retriever, post-ranker, and
reader. The CL strategy aims at reducing the chance of adding golden passage
during the joint training process to help the settings of training and
inference be more consistent. It would effectively push the retriever to find
the passage with the true answer contained by itself and encourage the reader
to discover the correct answer without the assistance of golden passage. To
the best of our knowledge, this is the first work that designs CL strategy for
conversational OpenQA joint learning.

Specifically, our semi-automatic CL strategy automates the difficulty measurer
by taking the question-wise training loss as criteria. A higher retriever loss
is recognized a harder mode for the retriever to discover the positive
passages. For the hard mode, the training scheduler of our CL strategy
introduces the golden passage in the retrieval results to make the learning
easier. Our training scheduler is like a Baby Step scheduler (Bengio et al.,
2009; Spitkovsky et al., 2010) with a finer granularity as shown in Algorithm
1.

Formally, for each iteration $l$, let
$\mathcal{P}^{(l)}_{K}=\\{\mathcal{P}^{(l)}_{i}|q_{i}\in Q^{(l)}\\}$ be the
set of all the collections of top $K$ passages retrieved by the retriever to
questions of the batch in the iteration. For the hard mode, the training
scheduler brings the golden passage of each question to
$\mathcal{P}^{(l)}_{K}$ to form the set of retrieved passages with the
positive one, denoted as
$\mathcal{P}^{(l)}_{KG}=\\{\mathcal{P}^{(l)}_{i}\cup\\{p^{+}_{i}\\}|q_{i}\in
Q^{(l)}\\}$. With these retrieved passages, the final loss to be optimized is
defined as:

(19)
$\mathcal{L}^{(l)}_{final}=v^{(l)}\mathcal{L}^{(l)}(\mathcal{P}_{KG}^{(l)})+(1-v^{(l)})\mathcal{L}^{(l)}(\mathcal{P}^{(l)}_{K}),$

where $v^{(l)}$ is determined by the difficulty measurer with the retriever
loss of the previous iteration:

(20)
$v^{(l)}=\left\\{\begin{array}[]{ccl}1,&&{\mathcal{L}^{(l-1)}_{retriever}>\lambda_{upper}},\\\
0,&&{\mathcal{L}^{(l-1)}_{retriever}<\lambda_{lower}},\\\
I(p_{b}),&&{\text{otherwise}},\end{array}\right.$

where $\lambda_{upper}$ and $\lambda_{lower}$ are the pre-defined upper and
lower thresholds. $I(p_{b})$ denotes an indicator function whose output is
sampled from a Bernoulli distribution:

(21) $I(p_{b})=\left\\{\begin{array}[]{ccl}1,&&{\text{with probability}\
p_{b}},\\\ 0,&&{\text{with probability}\ 1-p_{b}},\end{array}\right.$

where the probability is evaluated by the min-max normalization denoted as
$p_{b}=\frac{\mathcal{L}^{(l-1)}_{retriever}-\lambda_{lower}}{\lambda_{upper}-\lambda_{lower}}$,
which measures the degree of approximating upper threshold for the retriever
loss. It can be intuitively explained that if the retrieval loss of the
previous iteration is close to an upper threshold $\lambda_{upper}$, the
training status is regarded in the hard mode. And the value of $v^{(l)}$ is
assigned with 1 in a higher probability to select the set of retrieved
passages with golden passage for training in the current iteration.

$\mathcal{L}^{(l)}(\cdot)$ in Equation 19 denotes the objective function
integrated by all the three modules with the retrieved passages in the $l$-th
iteration as follows:

(22)
$\mathcal{L}^{(l)}=\mathcal{L}^{(l)}_{retriever}+\mathcal{L}^{(l)}_{postranker}+\mathcal{L}^{(l)}_{reader}.$

The loss of each module is averaged over the questions in the $l$-th
iteration.

Input: Training data
$\mathcal{D}_{train}=\\{(q_{i},a_{i},p_{i}^{+})\\}_{i=1}^{m}$,

Iteration number $L$.

Output: A set of optimal model parameters.

1

2for  _$l=1,\cdots,L$_ do

3 Sample a batch of questions $Q^{(l)}$

4 for  _$q_{i}\in Q^{(l)}$_ do

5
$\mathcal{P}_{i}^{(l)}\leftarrow\mathrm{arg\,max}_{p_{i,j}}(\mathrm{sim}(q_{i}^{en},p_{i,j}),K)$

6 $\mathcal{P}_{Gi}^{(l)}\leftarrow\mathcal{P}_{i}^{(l)}\cup\\{p^{+}_{i}\\}$

7 Compute $\mathcal{L}^{i}_{retriever}$, $\mathcal{L}^{i}_{postranker}$,
$\mathcal{L}^{i}_{reader}$

8 according to Eq.7, Eq.10, Eq.18

9

10 end for

11
$\mathcal{L}^{(l)}\leftarrow\frac{1}{|Q^{(l)}|}\sum_{i}(\mathcal{L}^{i}_{retriever}+\mathcal{L}^{i}_{postranker}+\mathcal{L}^{i}_{reader})$

12 $\mathcal{P}^{(l)}_{K}\leftarrow\\{\mathcal{P}^{(l)}_{i}|q_{i}\in
Q^{(l)}\\}$,
$\mathcal{P}^{(l)}_{KG}\leftarrow\\{\mathcal{P}^{(l)}_{Gi}|q_{i}\in
Q^{(l)}\\}$

13 Compute the coefficient $v^{(l)}$ according to Eq. 20

14 if  _$v^{(l)}=1$_ then

15
$\mathcal{L}^{(l)}_{final}\leftarrow\mathcal{L}^{(l)}(\mathcal{P}_{KG}^{(l)})$

16

17 else

18
$\mathcal{L}^{(l)}_{final}\leftarrow\mathcal{L}^{(l)}(\mathcal{P}^{(l)}_{K}),$

19

20 end if

21 Optimize $\mathcal{L}^{(l)}_{final}$

22 end for

Algorithm 1 Training Scheduler

###  3.3. Inference

In the inference stage, for a given question $q_{c}$ and its historical
question-answer pairs $\\{q_{i},a_{i}\\}_{i=c-w}^{c-1}$ within a window size
$w$, we first obtain a collection of the top $T$ relevant passages through the
two consecutive modules, namely retriever and post-ranker. For each passage
$p_{c,j}$ in the collection, we get the post-ranker score
$S_{post}(q_{c}^{en},p_{c,j})$ according to Equation 9. The reader score
includes two parts, one is the select score $S_{select}(q_{c}^{rd},p_{c,j})$
calculated by Equation 15, the other one is the span score which consists of
the start score and end score as shown in Equations 13 and 14. The span score
can be obtained by:

(23) $S_{span}(q_{c}^{rd},p_{c,j},sp)=\max\limits_{sp\in
p_{c,j}}\\{S_{s}(q_{c}^{rd},p_{c,j},[t_{s}])+S_{e}(q_{c}^{rd},p_{c,j},[t_{e}])\\},$

where $sp=([t_{s}],[t_{e}])$ is the answer span starting with token $[t_{s}]$
and ending with token $[t_{e}]$. Following the previous work (Kenton and
Toutanova, 2019; Qu et al., 2020), we pick out the top 20 spans and discard
the invalid predictions including the cases where the start token comes after
the end token, or the predicted span overlaps with the question part of the
input sequence. The final prediction score is defined as follows:

(24)
$S(q_{c}^{en},q_{c}^{rd},p_{c,j},sp)=S_{post}(q_{c}^{en},p_{c,j})+S_{reader}(q_{c}^{rd},p_{c,j},sp),$

where
$S_{reader}(q_{c}^{rd},p_{c,j},sp)=S_{select}(q_{c}^{rd},p_{c,j})+S_{span}(q_{c}^{rd},p_{c,j},sp)$.
For the given question $q_{c}$ in a conversation, the answer span $sp$ in the
retrieved passage $p_{c,j}$ that has the largest score is the predicted
answer.

##  4\. Experiments

###  4.1. Dataset

To evaluate the effectiveness of the proposed MICQA, we conduct comprehensive
experiments on the public available dataset OR-QuAC (Qu et al., 2020), which
integrates three datasets including the QuAC (Choi et al., 2018), CANARD
(Elgohary et al., 2019), and the Wikipedia corpus. OR-QuAC consists of totally
5,644 conversations containing 40,527 questions and answers with the rewrites
of questions obtained from CANARD. The question rewrites support the KL
divergence-based regularization for pretraining a better passage retriever.
OR-QuAC also provides a collection of more than 11 million passages obtained
from the English Wikipedia dump from
10/20/2019111ttps://dumps.wikimedia.org/enwiki/20191020/ for open-retrieval.
Table 1 summarizes the statistics of the aggregated OR-QuAC dataset.

Table 1. Data statistics of the OR-QuAC dataset. | Items | Train | Dev | Test
---|---|---|---|---
Coversations | # Dialogs | 4,383 | 490 | 771
# Questions / Rewrites | 31,526 | 3430 | 5571
# Avg. Questions / Dialog | 7.2 | 7 | 7.2
# Avg. Tokens / Question | 6.7 | 6.6 | 6.7
# Avg. Tokens / Rewrite | 10 | 10 | 9.8
Wikipedia | # Passages | 11 million
Table 2. Performance comparison of MICQA and baseline models. The number in the parentheses is the batch size during the retriever pre-training. Methods | Development | Test
---|---|---
F1 | HEQ-Q | HEQ-D | Rt MRR | Rt Recall | F1 | HEQ-Q | HEQ-D | Rt MRR | Rt Recall
DrQA (Chen et al., 2017) | 4.5 | 0.0 | 0.0 | 0.1151 | 0.2000 | 6.3 | 0.1 | 0.0 | 0.1574 | 0.2253
BERTserini (Yang et al., 2019) | 19.3 | 14.1 | 0.2 | 0.1767 | 0.2656 | 26.0 | 20.4 | 0.1 | 0.1784 | 0.2507
DPR (16) (Karpukhin et al., 2020) | 25.9 | 16.4 | 0.2 | 0.3993 | 0.5440 | 26.4 | 21.3 | 0.5 | 0.1739 | 0.2447
ORConvQA-bert (64) (Qu et al., 2020) | 26.9 | 17.5 | 0.2 | 0.4286 | 0.5714 | 29.4 | 24.1 | 0.6 | 0.2246 | 0.3141
ORConvQA-roberta (64) | 26.5 | 17.8 | 0.2 | 0.4284 | 0.5624 | 28.7 | 24.2 | 0.8 | 0.2330 | 0.3226
ours-bert (16) | 28.0 | 19.4 | 0.2 | 0.4639 | 0.6157 | 31.7 | 27.8 | 1.2 | 0.2763 | 0.3668
ours-roberta (16) | 28.1 | 19.5 | 0.4 | 0.4639 | 0.6169 | 33.4 | 29.4 | 1.7 | 0.2887 | 0.3819
ours-bert (32) | 27.6 | 19.6 | 0.0 | 0.4675 | 0.6236 | 32.6 | 29.1 | 0.8 | 0.3013 | 0.4130
ours-roberta (32) | 29.4 | 20.2 | 0.4 | 0.4656 | 0.6248 | 35.0 | 30.8 | 1.8 | 0.3073 | 0.4202

###  4.2. Experimental Settings

####  4.2.1. Evaluation Metrics

Following the evaluation protocols used in (Qu et al., 2020), we apply the
word-level F1 and the human equivalence score (HEQ) that are provided by the
QuAC challenge (Choi et al., 2018) to evaluate our MICQA. As the core
evaluation metric for the overall performance of answer retrieval, F1 is
computed by considering the portion of words in the prediction and groud truth
that overlap after removing stopwords. HEQ is used to judge whether a system’s
output is as good as that of an average human. It measures the percentage of
examples for which system F1 exceeds or matches human F1. Here two variants
are considered: the percentage of questions for which this is true (HEQ-Q),
and the percentage of dialogs for which this is true for every questions in
the dialog (HEQ-D). Furthermore, we use another two metrics, Mean Reciprocal
Rank (MRR) and Recall for the evaluation of the retrieval performance. MRR
reflects the abilities of post-ranker to return the passages containing true
answers in a high place. Recall is indicative of post-ranker’s capability of
providing relevant passages for the next modules. For the sake of fairness, we
follow (Qu et al., 2020) and calculate the two metrics for the top $T$
passages that are retrieved for the reader.

####  4.2.2. Implementation Settings

  1. (1)

Retriever Pretraining. As question and passage encoders, we employ two ALBERT
Base models. The maximum sequence length for the question encoder is 128,
while the maximum length for the passage encoder is 384. The models are
trained on NVIDIA TITAN X GPU. The training batch size is set to 16, the
number of training epochs is set to 12, the learning rate is set to 5e-5, the
window size $w$ is set to 6 and the coefficient of KL divergence $\alpha$ is
set to 0.2. Every 5,000 steps, we save checkpoints and assess on the
development set provided by (Qu et al., 2020). Then we select several models
that performed well on the development set and apply them on test questions
and wiki passages. Finally, we select the best model for future training.

  2. (2)

Post-ranker and Reader. For the post-ranker, we utilize a linear layer with a
size of 128 for simplicity. The number $K$ of passage embeddings it takes as
input is set to 100, and the number $T$ of its outputted passages for reader
is set to 5. For the reader, we apply the BERT and RoBERTa model. The max
sequence length is assigned with 512. The sequence is concatenated by a
question and a passage. The maximum passage length is set to 384, with the
remainder reserved for the question and other tokens such as `[CLS]` and
`[SEP]`.

  3. (3)

Joint Training. We use the pre-trained passage encoder to compute an embedding
for each passage, and build a single MIPS index using FAISS(Johnson et al.,
2017) for fast retrieval. Models are jointly trained on NVIDIA TITAN X GPU.
The training batch is set to 2, the number of epochs is 3, the learning rate
is 5e-5, and the optimizer is Adam. The hyper-parameters including the margin
of hinge loss $\delta$, triplet margin loss $\mu$, coefficient of contrastive
loss $\beta$, and pre-defined threshold $\lambda_{upper}$ and
$\lambda_{lower}$ are tuned based on the development set to select the optimal
model for inference. We save checkpoints and evaluate on the development set
every 5,000 steps, and then select the best model for the test set.

####  4.2.3. Baselines

To demonstrate the effectiveness of our proposed MICQA, we compare it with the
following state-of-the-art question answering models, including three
representative OpenQA models (DrQA, BERTserini, and DPR) and one
conversational OpenQA model (ORConvQA):

  * •

DrQA (Chen et al., 2017) is composed of a document retriever which uses bigram
hashing and TF-IDF matching to return the relevant passages for a given
question, and a multi-layer RNN based document reader for answer spans
detection in those retrieved passages.

  * •

BERTserini (Yang et al., 2019) uses a BM25 retriever from
Anserini222http://anserini.io/ and a BERT reader to tackle end-to-end question
answering. The retriever directly identifies segments of open-domain texts and
pass them to the reader. Compared to DPR, ORConvQA and our MICQA, it has no
selection loss in the reader and benefits less from the joint learning.

  * •

DPR (Karpukhin et al., 2020) increases retrieval by learning dense
representations instead of using typical IR methods. It innovatively proposes
to introduce hard negatives in the training process of the retriever. For
OpenQA, DPR consists of a dual encoder as a retriever and BERT as a reader.

  * •

ORConvQA (Qu et al., 2020) is first proposed to solve the conversational open-
domain QA problem with the retriever, rerank, and reader pipeline. This is the
key work to compare for our MICQA.

The results of all the baselines except DPR come from the previous work (Qu et
al., 2020). The implementation setting of DPR is the same with ORConvQA,
including the encoder network, window size, learning rate, and so on.

###  4.3. Overall Results

The overall experimental results are reported in Table 2. The results of the
baseline models are public in (Qu et al., 2020) except for DPR. The retrieval
metrics denoted by “Rt MRR”, and “Rt Recall” are used to evaluate the
retrieval results of retriever for baselines while evaluating that of post-
ranker for MICQA. Generally, our MICQA outperforms all the baseline models. In
detail, several observations can be achieved:

  1. (1)

Both DrQA and BERTserini achieve poor performance. The primary reason is they
use the sparse retriever which can not be fine-tuned in the downstream reader
training to discover relevant passages for answer extraction. DrQA performs
rather badly in answer reading as it uses RNN-based reader which does not have
the strong ability of representation learning as those pre-trained language
model. The reader of BERTserini is similar to the other compared methods
except DrQA. But it benefits less from the multi-task training process as
there is no select component (reranker) in reader. Compared with DrQA and
BERTserini, DPR improves the retriever with the dense representation learning.
The performance of DPR is limited by the batch size of pre-training.

  2. (2)

As the first system designed for the task of conversational OpenQA, ORConvQA
provides the best performance among the baselines. ORConvQA is similar with
DPR, where the difference is that it does not use hard negatives for retriever
pre-training while DPR uses. The main reason ORConvQA performs better is the
batch size of retriever pre-training is assigned with 64 as it uses 4 GPUs and
set batch size to 16 per GPU. Actually, with the same batch size, the
retriever of DPR is stronger, detailed results of which can be seen in the
further analysis of Section 4.5.1.

  3. (3)

Our MICQA gives the significantly better performance than ORConvQA, even
though our retriever is pre-trained with batch size of 16. This indicates that
our MICQA can perform better with the lower memory cost. When the batch size
is increased to 32, the performance can be further improved. The results
convincingly demonstrate the effectiveness of the multifaceted improvements of
KL-divergence regularization based pre-training, the added post-ranker, and
the semi-automatic curriculum learning strategy.

  4. (4)

To investigate the influence of the pre-trained language model of reader, we
evaluate our MICQA and ORConvQA based on two language models, namely BERT and
RoBERTa. For ORConvQA, the results obtained by using BERT and RoBERTa are
comparable. For our MICQA, applying RoBERTa for reader effectively improves
the performance compared to using BERT. Overall, MICQA achieves the best
performance whether with BERT or RoBERTa.

###  4.4. Ablation Studies

Table 3. Performance of ablation on different components. MICQA refers to the full system. Settings | MICQA |  w/o KL retriever |  w/o post-ranker |  w/o curriculum
---|---|---|---|---
Dev | F1 | 28.1 | 27.7 | 27.8 | 27.8
HEQ-Q | 19.5 | 19.4 | 19.4 | 18.3
HEQ-D | 0.4 | 0.0 | 0.0 | 0.0
Rt MRR | 0.4639 | 0.4033 | 0.4604 | 0.4560
Rt Recall | 0.6169 | 0.5324 | 0.6157 | 0.6140
Test | F1 | 33.4 | 31.5 | 32.2 | 31.7
HEQ-Q | 29.4 | 29.0 | 28.2 | 27.1
HEQ-D | 1.7 | 1.4 | 1.6 | 0.8
Rt MRR | 0.2887 | 0.2110 | 0.2810 | 0.2812
Rt Recall | 0.3819 | 0.2968 | 0.3830 | 0.3764

To investigate the effectiveness of three improved parts in our MICQA, we
evaluate several variants of our system. As shown in Table 3, once we remove
one of the three components, both the retrieval and QA performance generally
decrease. The detailed observations are summarized as follows:

  1. (1)

When we remove the KL-based regularized pre-trained retriever and use the
retriever of ORConvQA as replacement, the performance drops significantly,
especially the retrieval performance. It shows the importance of the KL-based
regularization in our pre-trained retriever.

  2. (2)

Removing the post-ranker also brings a degradation in the overall performance.
The influence is slightly smaller than that of KL-based regularization, which
is probably caused by the simple linear layer used for post-ranker. This is
our limitation and we plan to explore the more flexible and effective neural
network for post-ranker in future work. From another perspective, some
improvements can be achieved just by adding a linear layer for further passage
representation learning. It is notable that the retrieval recall is higher
than the full system. It is mainly because each question in the test set has
more than one golden passages. Retrieving more golden passages does not
necessarily lead to the better answer span which has the higher coverage of
the ground truth answer.

  3. (3)

The variant without curriculum learning in the joint training performs worse
than the full MICQA. By comparison, the QA performance decreases more
noticeably. The reason behind is that the curriculum learning strategy
encourages the reader to find correct answer with no golden passage assistance
at the joint training time. It makes the reader more suitable for answer
extraction in the inference phase.

Table 4. Results of retriever pre-training. B is batch size, Q is the form of question used in training, HD is the number of hard negatives. | B | Q | HD | Recall@20 | Recall@100
---|---|---|---|---|---
BM25 | / | / | / | 0.3711 | 0.5100
ORConvQA | 16 | $q^{rw}$ | 0 | 0.1672 | 0.2916
16$\times$4 | $q^{rw}$ | 0 | 0.3561 | 0.5034
DPR | 16 | $q^{rw}$ | 1 | 0.2395 | 0.3759
ours | 16 |  $q^{rw}$/$q^{rw}$ | 1 | 0.3214 | 0.4689
16 |  $q^{rw}$/$q^{or}$ | 1 | 0.3690 | 0.5070
| 32 |  $q^{rw}$/$q^{or}$ | 1 | 0.4675 | 0.5882

###  4.5. Further Analysis

####  4.5.1.  Retriever Performance

We evaluate some existing pre-trained retriever with our KL-divergence
regularized retriever on the full collection of Wikipedia passages. The
evaluation results are reported in Table 4. As the typical sparse retriever,
BM25 achieves a good retrieval performance while its QA performance is limited
as it can not be fine-tuned in subsequent joint training. Both ORConvQA and
DPR retrievers take the question rewrite as input following the setting in (Qu
et al., 2020). The only difference between them is that DPR uses a TF-IDF hard
negative provided by the dataset in addition to the in-batch negatives. DPR
outperforms ORConvQA when the batch size is the same, which indicates that the
hard negative play a key role during training. Considering the batch size of
ORConvQA in (Qu et al., 2020) is 64, we also show the retrieval performance,
which is improved dramatically as the number of in-batch negatives increases.
Our KL-divergence based regularized retriever uses two question forms, rewrite
and original question, as input, and offers the best performance with the same
batch size of 16 among these dense retrieval model. When the batch size is
raised to 32, our retriever achieves a better performance and exceeds BM25.
Moreover, to explore the advantage of the original question which is the
concatenation of the historical and current questions, we evaluate our
retriever by feeding two of the same rewrite. The results shows that employing
two different forms of one question leads to a better performance. It is
probably because the input of two same rewrites works just as a regularization
while that of original question and rewrite is able to add a supervision for
question understanding learning.

####  4.5.2. Case Study for Post-Ranker

To explore the effect of post-ranker, we count the number of golden passages
before and after post-ranker in the inference phase. Concretely, the number of
golden passages in the top $T$ ($T=5$) increases from 2,228 to 2,269 while the
golden passages ranked between $T+1$ and $K$ ($K=100$) decreases from 1,292 to
1,251, which verifies post-ranker’ ability in enforcing the relevant passages
to appear in the higher place. Figure 4 shows an example of the passage
retrieval results before and after post-ranker for a question. Before applying
post-ranker, it can be observed that golden passage is not contained in the
top 5 passages but ranked in the 14 place of the ranking list generated by the
retriever. Taking the passage representations in the ranking list as input,
post-ranker provides a new passage ranking where the golden passage is
reranked in the top 5. Moreover, the top 5 passages of the newly generated
ranking list contains more relevant content with the question and its
historical questions, showing the advantage of the proposed post-ranker module
in our MICQA.

Figure 4. Case study of the passage retrieval results before and after post-
ranker for an example question.

####  4.5.3. Impact of Curriculum Learning Strategy

To investigate the impact of curriculum learning, we save checkpoints and
evaluate on the test set every 500 steps, the results of all the checkpoints
are shown in Figure 5. It can be observed that the QA metric F1 with
curriculum learning increases more steadily and reached better final
performance faster. For retrieval metric, the improvement achieved by
curriculum learning is the higher final recall value, which is not quite
noticeable. It indicates that the curriculum learning strategy makes more
pronounced contribution for the answer reading other than the passage
retrieval, which in reasonable as the inconsistent problem addressed by
curriculum learning mainly reflects in the reading part.

##  5\. Related Work

This section briefly summarizes some existing works on open domain question
answering and conversational question answering, which are most relevant to
this work.

###  5.1. Open Domain Question Answering

Open-domain question answering (OpenQA) (Voorhees et al., 1999) is a task that
uses a huge library of documents to answer factual queries. The two-stage
design, which included a passage retriever to choose a subset of passages and
a machine reader to exact answers, became popular after DrQA (Chen et al.,
2017).

The passage retriever is an important component of OpenQA system since it
searches relevant paragraphs for the next stage. Traditional sparse retrieval
models, such as TF-IDF or BM25 (Robertson and Zaragoza, 2009), have been
widely adopted as retriever in OpenQA systems (Chen et al., 2017; Yang et al.,
2019; Lin et al., 2018). While sparse retrieval cannot handle the case of high
semantic correlation with little lexical overlap and it is untrainable, dense
passage retrievers have lately gained popularity (Lee et al., 2019; Guu et
al., 2020; Karpukhin et al., 2020; Qu et al., 2021). In general, the dense
retrieval model is a dual-encoder architecture that encodes both the question
and the passage individually. Both encoders are trained during the retriever
pre-training process. When training with the reader for the QA task, only the
question encoder is normally fine-tuned. In order to increase the retrieval
impact of dense retrievers, some studies incorporate hard negatives. BM25 top
passages which do not contain answers are utilized as hard negatives
(Karpukhin et al., 2020; Gao et al., 2020). And dynamic hard negatives are
employed in (Xiong et al., 2021; Zhan et al., 2021; Guu et al., 2020), which
are the top-ranked irrelevant documents given by dense retriever during
training.

A contemporary OpenQA system also includes a reader as a key component. Its
goal is to deduce the answer to a query from a collection of documents.
Existing Readers may be divided into two types: (1) Extractive readers (Chen
et al., 2017; Yang et al., 2019; Karpukhin et al., 2020), which anticipate an
answer span from the retrieved texts, (2) Generative readers (Lewis et al.,
2020; Izacard and Grave, 2021), which produce natural language replies using
sequence-to-sequence (Seq2Seq) models.

(a)

(b)

Figure 5. Comparison of performance improvement with / without curriculum
learning.

###  5.2. Conversational Question Answering

Conversational Question Answering (CQA) is required to understand the given
context and history dialogue to answer the question. As a main type of CQA,
Conversational Machine Reading Comprehension (CMRC) (Qu et al., 2019a; Qu et
al., 2019b; Qiu et al., 2021) does the QA task with text-based corpora. For
CMRC, the number of conversational history turns is critical, as context
utterances that are relevant to the inquiry are valuable, while irrelevant
ones may introduce additional noise. For example, (Qu et al., 2019a, 2020)
makes use of conversation history by including $K$ rounds of history turns.
(Qu et al., 2019b) weights previous conversation rounds based on their
contribution to the answer to the current question.

The approaches outlined above rely extensively either on the provided material
or a given paragraph to extract or generate answers. However, this is
impractical in real world since golden passage is not always available. Open
retrieval methods which try to obtain evidence from a big collection, have
lately been popular in the CMRC. ORConvQA (Qu et al., 2020) is the first work
proposing three primary modules for open-retrieval CQA: (1) a passage
retriever, (2) a passage reranker, and (3) a passage reader. Given a query and
its conversational history, the passage retriever retrieves the top $K$
relevant texts from a large-scale corpus. The passage reranker and reader then
rerank and read the top texts to discover the correct answer. The research for
the conversational OpenQA needs to be further explored. This work tries to
improve ORConvQA from multiple aspects including regularizing retriever pre-
training, incorporating post-ranking, and curriculum learning.

##  6\. Conclusion

This paper proposes multifaceted improvements for the conversational OpenQA
task. Concretely, a KL-divergence based regularization is proposed in pre-
training for a better question understanding. A post-ranker module is added to
realize the joint training of question and passage representations to generate
a better passage ranking. A semi-automatic curriculum learning strategy is
designed to encourage the reader to find the answer without manually adding
golden passages. The experimental evaluation demonstrates the effectiveness of
our MICQA.

## References

  * (1)
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009\.  Curriculum learning. In _Proceedings of the 26th Annual International Conference on Machine Learning_. 41–48.
  * Bromley et al. (1993) Jane Bromley, James W Bentz, Léon Bottou, Isabelle Guyon, Yann LeCun, Cliff Moore, Eduard Säckinger, and Roopak Shah. 1993\.  Signature verification using a “siamese” time delay neural network.  _International Journal of Pattern Recognition and Artificial Intelligence_ 7, 04 (1993), 669–688.
  * Chen et al. (2017) Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017\.  Reading Wikipedia to Answer Open-Domain Questions. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_. 1870–1879.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018\.  QuAC: Question Answering in Context. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 2174–2184.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  (2019), 4171–4186.
  * Elgohary et al. (2019) Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019\.  Can You Unpack That? Learning to Rewrite Questions-in-Context. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 5918–5924.
  * Gao et al. (2020) Luyu Gao, Zhuyun Dai, Tongfei Chen, Zhen Fan, Benjamin Van Durme, and Jamie Callan. 2020\.  Complementing lexical retrieval with semantic residual embedding.  _arXiv preprint arXiv:2004.13969_ (2020).
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_ (2020).
  * Izacard and Grave (2021) Gautier Izacard and Édouard Grave. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. 874–880.
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017\.  Billion-scale similarity search with GPUs.  _arXiv preprint arXiv:1702.08734_ (2017).
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 6769–6781.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Kratzwald and Feuerriegel (2018) Bernhard Kratzwald and Stefan Feuerriegel. 2018.  Adaptive Document Retrieval for Deep Question Answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 576–581.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.  arXiv:1909.11942 [cs.CL]
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019\.  Latent retrieval for weakly supervised open domain question answering.  _arXiv preprint arXiv:1906.00300_ (2019).
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _arXiv preprint arXiv:2005.11401_ (2020).
  * Lin et al. (2018) Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018\.  Denoising distantly supervised open-domain question answering. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1736–1745.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Min et al. (2019) Sewon Min, Danqi Chen, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019.  Knowledge guided text retrieval and reading for open domain question answering.  _arXiv preprint arXiv:1911.03868_ (2019).
  * Qiu et al. (2021) Minghui Qiu, Xinjing Huang, Cen Chen, Feng Ji, Chen Qu, Wei Wei, Jun Huang, and Yin Zhang. 2021.  Reinforced history backtracking for conversational question answering. In _Proceedings of the 35th Conference on Artificial Intelligence, AAAI_.
  * Qu et al. (2020) Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W Bruce Croft, and Mohit Iyyer. 2020\.  Open-retrieval conversational question answering. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 539–548.
  * Qu et al. (2019a) Chen Qu, Liu Yang, Minghui Qiu, W Bruce Croft, Yongfeng Zhang, and Mohit Iyyer. 2019a.  BERT with history answer embedding for conversational question answering. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 1133–1136.
  * Qu et al. (2019b) Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W Bruce Croft, and Mohit Iyyer. 2019b.  Attentive history selection for conversational question answering. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1391–1400.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 5835–5847.
  * Robertson and Zaragoza (2009) Stephen Robertson and Hugo Zaragoza. 2009.  _The probabilistic relevance framework: BM25 and beyond_.  Now Publishers Inc.
  * Santos et al. (2016) Cicero dos Santos, Ming Tan, Bing Xiang, and Bowen Zhou. 2016\.  Attentive pooling networks.  _arXiv preprint arXiv:1602.03609_ (2016).
  * Spitkovsky et al. (2010) Valentin I Spitkovsky, Hiyan Alshawi, and Dan Jurafsky. 2010\.  From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_. 751–759.
  * Voorhees et al. (1999) Ellen M Voorhees et al. 1999\.  The TREC-8 question answering track report. In _Trec_ , Vol. 99. Citeseer, 77–82.
  * Weinberger et al. (2006) Kilian Q Weinberger, John Blitzer, and Lawrence K Saul. 2006\.  Distance metric learning for large margin nearest neighbor classification. In _Advances in neural information processing systems_. 1473–1480.
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021\.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In _International Conference on Learning Representations (ICLR)_.  https://www.microsoft.com/en-us/research/publication/approximate-nearest-neighbor-negative-contrastive-learning-for-dense-text-retrieval/
  * Xiong et al. (2020) Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, et al. 2020\.  Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval. In _International Conference on Learning Representations_.
  * Yang et al. (2019) Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019\.  End-to-End Open-Domain Question Answering with BERTserini. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics_. 72–77.
  * Zhan et al. (2021) Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma. 2021\.  Optimizing Dense Retrieval Model Training with Hard Negatives.  _arXiv preprint arXiv:2104.08051_ (2021).
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021\.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_ (2021).