# Can Question Rewriting Help Conversational Question Answering?

Etsuko Ishii, Yan Xu∗, Samuel Cahyawijaya∗, Bryan Wilie
The Hong Kong University of Science and Technology
{eishii, yxucb, scahyawijaya<EMAIL_ADDRESS>
Equal Contribution

###### Abstract

Question rewriting (QR) is a subtask of conversational question answering
(CQA) aiming to ease the challenges of understanding dependencies among
dialogue history by reformulating questions in a self-contained form. Despite
seeming plausible, little evidence is available to justify QR as a mitigation
method for CQA. To verify the effectiveness of QR in CQA, we investigate a
reinforcement learning approach that integrates QR and CQA tasks and does not
require corresponding QR datasets for targeted CQA. We find, however, that the
RL method is on par with the end-to-end baseline. We provide an analysis of
the failure and describe the difficulty of exploiting QR for CQA.

##  1 Introduction

The question rewriting (QR) task has been introduced as a mitigation method
for conversational question answering (CQA). CQA asks a machine to answer a
question based on the provided passage and a multi-turn dialogue (Reddy et
al., 2019; Choi et al., 2018), which poses an additional challenge to
comprehend the dialogue history. To ease the challenge, QR aims to teach a
model to paraphrase a question into a self-contained format using its dialogue
history (Elgohary et al., 2019a; Anantha et al., 2021a). Except for Kim et al.
(2021), however, no one has provided evidence that QR is effective for CQA in
practice. Existing works on QR often (i) depend on the existence of a QR
dataset for every target CQA dataset, and (ii) focus more on generating high-
quality rewrites than improving CQA performance, making them unsatisfactory
for the justification of QR.

To verify the effectiveness of QR, we explore a reinforcement learning (RL)
approach that integrates QR and CQA tasks without corresponding labeled QR
datasets. In the RL framework, a QR model plays the role of “the agent” that
receives rewards from a QA model that acts as “the environment.” During
training, the QR model aims to maximize the performance on the CQA task by
generating better rewrites of the questions.

Figure 1: Overview of the RL approach. The current question $Q_{t}$ and its
dialogue history are reformulated into a self-contained question
$\acute{Q}_{t}$ by the QR model. Then, $\acute{Q}_{t}$ is passed to the QA
model to extract an answer span $\tilde{A}_{t}$ from the evidence document. We
train the QR model by maximizing the reward obtained by comparing the
predicted answer span $\tilde{A}_{t}$ with the gold span $A_{t}$.

Despite the potential and plausibility of the RL approach, our experimental
results suggest an upper bound of the performance, and it is on par with the
baselines without QR. In this paper, we provide analysis to (i) understand the
reason for the failure of the RL approach and (ii) reveal that QR cannot
improve CQA performance even with the non-RL approaches. The code is available
at https://github.com/HLTCHKUST/cqr4cqa.

Models | CoQA | QuAC
---|---|---
Overall F1 | Child. | Liter. | M&H | News | Wiki. | F1 | HEQ-Q | HEQ-D
| end-to-end | 84.5 | 84.4 | 82.4 | 82.9 | 86.0 | 86.9 | 67.8 | 63.5 | 7.9
QReCC | pipeline | 82.9 | 82.9 | 80.9 | 81.5 | 84.4 | 84.8 | 66.3 | 62.0 | 6.6
ours | 84.7 | 84.3 | 83.1 | 82.7 | 86.3 | 86.8 | 67.6 | 63.2 | 7.8
CANARD | pipeline | 82.8 | 83.4 | 80.1 | 80.8 | 84.4 | 85.6 | 66.5 | 62.5 | 7.4
EXCORD† | 83.4 (+0.6) | 84.4 (1.9) | 81.2 (+1.0) | 79.8 (-0.3) | 84.6 (+0.3) | 87 (0.0) | 67.7 (+1.2) | 64.0 (+1.6) | 9.3 (+2.1)
ours | 84.4 | 84.1 | 82.7 | 82.6 | 86.0 | 86.7 | 67.4 | 62.7 | 8.1

Table 1: Evaluation results of our approach and baselines on the test set.
EXCORD† follows the results reported by Kim et al. (2021) and ($\pm$x.x)
indicate the improvement compared to their original baseline. Bold are the
best results amongst all. Underlined represents the best score on each
combination of the CQA and QR datasets.

##  2 Related Work

The CQA task aims to assist users in seeking information (Reddy et al., 2019;
Choi et al., 2018; Campos et al., 2020). The key challenge is to resolve the
conversation history and understand a highly-contextualized question. Most
prior works focus on model structures (Zhu et al., 2018; Yeh and Chen, 2019;
Zhang et al., 2021b; Zhao et al., 2021) or training techniques Ju et al.
(2019); Xu et al. (2021) to improve the performance. QR tasks have been
proposed to further improve CQA systems by paraphrase a question into a self-
contained styles (Elgohary et al., 2019a; Petrén Bach Hansen and Søgaard,
2020; Anantha et al., 2021a). While many of the existing works on QR put more
effort toward generating high-quality rewrites (Lin et al., 2020; Vakulenko et
al., 2021), Kim et al. (2021) introduced a framework to leverage QR to
finetune CQA models with a consistency-based regularization. QR has also been
studied in single-turn QA and other information-seeking tasks (Nogueira and
Cho, 2017; Buck et al., 2018).

##  3 Methodology

We denote a CQA dataset as $\\{\mathcal{D}^{n}\\}_{n=1}^{N}$ and the dialogue
history at turn $t$ as $\mathcal{D}_{t}=\\{(Q_{i},A_{i})\\}_{i=1}^{t}$, where
$Q_{t}$ is the question and $A_{t}$ is the answer. Along with the QA pairs,
the corresponding evidence documents $Y_{t}$ are also given.

As depicted in Figure 1, our proposed RL framework involves a QA model as an
environment and a QR model as an agent. Let
$\acute{Q}_{t}=\\{\acute{q}_{l}\\}_{l=1}^{L}$ denote a generated rewritten
question sequence of $Q_{t}$. The objective of the QR model is to rewrite the
question $Q_{t}$ at turn $t$ into a self-contained version, based on the
current question and the dialogue history $\mathcal{D}_{t-1}$. The agent takes
an input state $X_{t}=(\mathcal{D}_{t-1},Q_{t})$ and generates a paraphrase
$\acute{Q}_{t}$. Then, $\acute{X}_{t}=(\mathcal{D}_{t-1},\acute{Q}_{t})$ and
an evidence document $Y_{t}$ are provided to an environment, namely, the QA
model $f_{\phi}$, which extracts an answer span
$\tilde{A}_{t}=f_{\phi}(\acute{X_{t}},Y_{t})$. We aim for the agent, a QR
model $\pi_{\theta}$, to learn to generate a high-quality paraphrase of the
given question based on the reward received from the environment.

The policy, in our case the QR model, assigns probability

$\displaystyle\pi_{\theta}(\acute{Q}_{t}|X_{t})=\prod_{l=1}^{L}p(\acute{q}_{l}|\acute{q}_{1},\ldots,\acute{q}_{l-1},X_{t}).$
(1)

Our goal is to maximize the expected reward of the answer returned under the
policy, namely,

$\displaystyle\mathrm{E}_{\acute{q}_{t}\sim\pi_{\theta}(\cdot|q_{t})}[r(f_{\phi}(\acute{X_{t}}))],$
(2)

where $r$ is a reward function. We apply the token-level F1-score between the
predicted answer span $\tilde{A}_{t}$ and the gold span $A_{t}$ as the reward
$r$. We can directly optimize the expected reward in Eq. 2 using RL
algorithms.

Prior to the training process, the QA model $f_{\phi}$ is fine-tuned on
$\\{\mathcal{D}^{n}\\}$ and the QR model is initialized with
$\pi_{\theta}=\pi_{\theta_{0}}$, where $\pi_{\theta_{0}}$ is a pretrained
language model. We apply Proximal Policy Optimization (PPO) (Schulman et al.,
2017; Ziegler et al., 2019) to train $\pi_{\theta}$. PPO is a policy gradient
method which alternates between sampling data through interaction with the
environment and optimizing a surrogate objective function via stochastic
gradient ascent. Following Ziegler et al. (2019), we apply a KL-penalty to the
reward $r$ so as to prevent the policy $\pi_{\theta}$ from drifting too far
away from $\pi_{\theta_{0}}$:

$\displaystyle
R_{t}=R(\acute{X}_{t})=r(f_{\phi}(\acute{X}_{t}))-\beta\mathrm{KL}(\pi_{\theta},\pi_{\theta_{0}}),$

where $\beta$ represents a weight factor and $R_{t}$ is the modified reward of
$r$.

Question | F1 Score | Question | F1 Score
---|---|---|---
$Q_{t}$ | What is the Vat the  library of? | 1.0 | $Q_{t}$ | Where  did the band The Smashing Pumpkins put on display? | 1.0
$\acute{Q}_{t}$ | What is the Vat the  Library of? | 0.22 | $\acute{Q}_{t}$ | Where  was the band The Smashing Pumpkins put on display? | 0.0
$Q_{t}$ | What was  everybody doing? | 0.91 | $Q_{t}$ | Which company produced the movie  Island of Misfit Toys? | 1.0
$\acute{Q}_{t}$ | What was  everyone doing? | 0.0 | $\acute{Q}_{t}$ | Which company produced the movie, The Island of Misfit Toys? | 0.0

Table 2: Minor modification of questions may cause a drastic change in CQA
performance.

##  4 Experiments

###  4.1 Setup

We use a pretrained RoBERTa Liu et al. (2019) model as the initial QA model
and adapt it to the CQA tasks. For the QR models, we leverage pretrained GPT-2
Radford et al. (2019) and first finetune them with QR datasets for better
initialization. We attempt three settings: (a) directly fine-tune the QA model
on the CQA datasets (end-to-end), (b) fine-tune the QA model with questions
rewritten by the QR model (pipeline), and (c) train the QR model based on the
reward obtained from the QA model. More details of the experiments can be
found in Appendix A.

#### Datasets

We conduct our experiments on two crowd-sourced CQA datasets, CoQA (Reddy et
al., 2019) and QuAC (Choi et al., 2018). Since the test set is not publicly
available for both CoQA and QuAC, following Kim et al. (2021), we randomly
sample 5% of dialogues in the training set and adopt them as our validation
set and report the test results on the original development set for the CoQA
experiments. We apply the same split as Kim et al. (2021) for the QuAC
experiments.

For the QR model pre-training, we use two QR datasets: QReCC Anantha et al.
(2021b) and CANARD Elgohary et al. (2019b). CANARD is generated by rewriting a
subset of the original questions in the QuAC datasets, and contains 40K
questions in total. QReCC is built upon three publicly available datasets:
QuAC, TREC Conversational Assistant Track (CAsT) Dalton et al. (2020) and
Natural Questions (NQ) Kwiatkowski et al. (2019). QReCC contains 14K dialogues
with 80K questions, and 9.3K dialogues are from QuAC.

#### Evaluation Metrics

Following the leaderboards, we utilize the unigram F1 score to evaluate the QA
performance. In CoQA evaluation, the QA models are also evaluated with the
domain-wise F1 score. In QuAC evaluation, we incorporate the human equivalence
score HEQ-Q and HEQ-D as well. HEQ-Q indicates the percentage of questions on
which the model outperforms human beings and HEQ-D represents the percentage
of dialogues on which the model outperforms human beings for all questions in
the dialogue.

###  4.2 Results

We report our experimental results in Table 1. We see that our RL approach
yields 0.9–1.6 F1 improvement over the pipeline setting regardless of the
dataset combinations and performs almost as well as the end-to-end setting.
This partially supports our expectation that RL lifts the CQA performance.
However, we find it almost impossible to bring significant improvement over
the end-to-end baseline despite our extensive trials. One reason why we cannot
provide as much improvement as reported in Kim et al. (2021) would be related
to the inputs of the QA model. Their EXCORD feeds the original questions
together with the rewritten questions, whereas we only use the rewritten
questions. It is also noteworthy that their results are consistently lower
than ours, even lower than our end-to-end settings.

Our inspection of the questions generated by the QR models reveals that the
models learn to copy the original questions by PPO training, and this is the
direct reason that our method cannot outperform the end-to-end baselines.
Indeed, on average, 89.6% of the questions are the same as the original
questions after PPO training, although this value is 34.5% in the pipeline
settings. We also discover a significant correlation between the performance
and how much the QR models copy the original question (the correlation
coefficient is 0.984 for CoQA and 0.967 for QuAC) and the edit distance from
the original question (the correlation coefficient is -0.996 for CoQA and
-0.989 for QuAC).

Perturb | Sentiment Analysis | CQA
---|---|---
Amazon | Yelp | CoQA | QuAC
Original | 95.8 | 98.2 | 84.5 | 67.8
UPC | 95.8 (-) | 96.7 (-1.5) | 74.8 (-9.8) | 57.4 (-10.5)
SLW | 91.9 (-3.9) | 97.0 (-1.1) | 83.0 (-1.6) | 66.7 (-1.1)
WIF | 94.3 (-1.5) | 97.7 (-0.5) | 82.6 (-2.0) | 65.6 (-2.2)
SPP | 94.8 (-1.0) | 97.7 (-0.5) | 78.3 (-6.2) | 65.5 (-2.4)

Table 3: Robustness test on Sentiment Analysis and CQA tasks. We apply four
perturbations: UPC (upper casing), SLW (slang word), WIF (word inflection),
and SPP (sentence paraphrasing).

##  5 Discussion

Datasets | QuAC Model | CANARD Model
---|---|---
F1 | EM | F1 | EM
QuAC | 67.7 | 51.5 | 62.9 | 46.8
CANARD | 65.1 | 49.9 | 63.3 | 46.9

Table 4: Results of the supervised learning approach. “XX Model” denotes the
QA model trained on XX, and EM the percentage of the predictions the same as
the gold.

In this section, we provide an analysis to (i) raise a sensitivity problem of
the QA model to explain the failure of RL and (ii) disclose that there is no
justification for QR, even in the non-RL approaches.

###  5.1 Sensitivity of the QA model

It appears that the QA models are more sensitive to trivial changes than the
reward models in other successful language generation tasks, and this could
account for our lower performance on CQA. As can be seen from the examples in
Table 2, a subtle alteration such as uppercasing or replacement with synonyms
can significantly change F1 scores.

To quantify the sensitivity of the reward models, we compare model robustness
between our QA models and sentiment analysis models that have been reported in
Ziegler et al. (2019) to be effective for stylistic language generation. We
adopt publicly available models that are fine-tuned sentiment analysis
datasets: BERT-based trained on Amazon polarity McAuley and Leskovec
(2013)111https://huggingface.co/fabriceyhc/bert-base-uncased-amazon_polarity
and RoBERTa-base trained on Yelp polarity Zhang et al.
(2015)222https://huggingface.co/VictorSanh/roberta-base-finetuned-yelp-
polarity. To test the robustness of the models, we introduce small
perturbations to the samples in the test set using the NL-Augmenter toolkit
Dhole et al. (2021), and compare F1 scores on each task (experimental details
in Appendix B).

Based on the robustness test given in Table 3, the QA models are shown to be
significantly less robust against most perturbations compared to the sentiment
analysis models. It is conceivable that this sensitivity of the QA model leads
to a sparse reward problem for the agent, which causes instability for the
model learning the optimal policy. An important direction for future studies
is to ease the sparse reward problem by, for example, enhancing the robustness
of the QA models.

###  5.2 Can QR Help in Non-RL Approaches?

First, we evaluate with a simple supervised learning approach using rewrites
provided by CANARD. Extracting the QuAC samples that have a CANARD annotation,
we (i) evaluate the CANARD annotations with the QA model trained on QuAC (the
model used in the main experiments) and (ii) train another QA model with the
CANARD annotations. Training is under the same conditions of the QA model
initialization as in the main experiments. As the results in Table 4 show, we
can hardly observe the effectiveness of the CANARD annotations. This supports
the claim in Buck et al. (2018) that better rewrites in the human eye are not
necessarily better for machines and implies the difficulty of exploiting QR
for CQA.

Datasets | CoQA | QuAC
---|---|---
F1 | EM | F1 | EM
end-to-end | 84.5 | 76.4 | 67.83 | 51.47
QReCC | 84.1 | 76.0 | 67.83 | 51.48
CANARD | 83.7 | 75.8 | 67.81 | 51.50

Table 5: Results of the data augmentation approach. EM denotes the percentage
of the predictions the same as the gold.

Moreover, we explore a data-augmentation approach to integrate QR and CQA.
First, we generate ten possible rewrites using top-$k$ sampling Zhang et al.
(2021a) for all the questions of the CQA datasets. To guarantee the quality of
the rewrites, we select the best F1 scoring ones from every ten candidates and
use them to teach another QR model how to reformulate questions (experimental
details in Appendix C). As the results in Table 5 show, we consistently get
worse scores compared to the end-to-end settings in CoQA, and almost the same
scores for QuAC, not finding justification to apply QR in the manner of the
data augmentation approach.

##  6 Conclusion

In this paper, we explore the RL approach to verify the effectiveness of QR in
CQA, and report that the RL approach is on par with simple end-to-end
baselines. We find the sensitivity of the QA models would disadvantage the RL
training. Future work is needed to verify that QR is a promising mitigation
method for CQA since even the non-RL approaches perform unsatisfactorily.

## References

  * Anantha et al. (2021a) Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021a.  Open-domain question answering goes conversational via question rewriting.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 520–534, Online. Association for Computational Linguistics.
  * Anantha et al. (2021b) Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021b.  Open-domain question answering goes conversational via question rewriting.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 520–534.
  * Buck et al. (2018) Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Gesmundo, Neil Houlsby, and Wei Wang. 2018.  Ask the right questions: Active question reformulation with reinforcement learning.  In _International Conference on Learning Representations_.
  * Campos et al. (2020) Jon Ander Campos, Arantxa Otegi, Aitor Soroa, Jan Deriu, Mark Cieliebak, and Eneko Agirre. 2020.  DoQA - accessing domain-specific FAQs via conversational QA.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7302–7314, Online. Association for Computational Linguistics.
  * Choi et al. (2018) Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018.  QuAC: Question answering in context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics.
  * Dalton et al. (2020) Jeffrey Dalton, Chenyan Xiong, and Jamie Callan. 2020.  Trec cast 2019: The conversational assistance track overview.  _arXiv preprint arXiv:2003.13624_.
  * Dhole et al. (2021) Kaustubh D. Dhole, Varun Gangal, Sebastian Gehrmann, Aadesh Gupta, Zhenhao Li, Saad Mahamood, Abinaya Mahendiran, Simon Mille, Ashish Srivastava, Samson Tan, Tongshuang Wu, Jascha Sohl-Dickstein, Jinho D. Choi, Eduard Hovy, Ondrej Dusek, Sebastian Ruder, Sajant Anand, Nagender Aneja, Rabin Banjade, Lisa Barthe, Hanna Behnke, Ian Berlot-Attwell, Connor Boyle, Caroline Brun, Marco Antonio Sobrevilla Cabezudo, Samuel Cahyawijaya, Emile Chapuis, Wanxiang Che, Mukund Choudhary, Christian Clauss, Pierre Colombo, Filip Cornell, Gautier Dagan, Mayukh Das, Tanay Dixit, Thomas Dopierre, Paul-Alexis Dray, Suchitra Dubey, Tatiana Ekeinhor, Marco Di Giovanni, Rishabh Gupta, Rishabh Gupta, Louanes Hamla, Sang Han, Fabrice Harel-Canada, Antoine Honore, Ishan Jindal, Przemyslaw K. Joniak, Denis Kleyko, Venelin Kovatchev, Kalpesh Krishna, Ashutosh Kumar, Stefan Langer, Seungjae Ryan Lee, Corey James Levinson, Hualou Liang, Kaizhao Liang, Zhexiong Liu, Andrey Lukyanenko, Vukosi Marivate, Gerard de Melo, Simon Meoni, Maxime Meyer, Afnan Mir, Nafise Sadat Moosavi, Niklas Muennighoff, Timothy Sum Hon Mun, Kenton Murray, Marcin Namysl, Maria Obedkova, Priti Oli, Nivranshu Pasricha, Jan Pfister, Richard Plant, Vinay Prabhu, Vasile Pais, Libo Qin, Shahab Raji, Pawan Kumar Rajpoot, Vikas Raunak, Roy Rinberg, Nicolas Roberts, Juan Diego Rodriguez, Claude Roux, Vasconcellos P. H. S., Ananya B. Sai, Robin M. Schmidt, Thomas Scialom, Tshephisho Sefara, Saqib N. Shamsi, Xudong Shen, Haoyue Shi, Yiwen Shi, Anna Shvets, Nick Siegel, Damien Sileo, Jamie Simon, Chandan Singh, Roman Sitelew, Priyank Soni, Taylor Sorensen, William Soto, Aman Srivastava, KV Aditya Srivatsa, Tony Sun, Mukund Varma T, A Tabassum, Fiona Anting Tan, Ryan Teehan, Mo Tiwari, Marie Tolkiehn, Athena Wang, Zijian Wang, Gloria Wang, Zijie J. Wang, Fuxuan Wei, Bryan Wilie, Genta Indra Winata, Xinyi Wu, Witold Wydmański, Tianbao Xie, Usama Yaseen, M. Yee, Jing Zhang, and Yue Zhang. 2021\.  Nl-augmenter: A framework for task-sensitive natural language augmentation.
  * Elgohary et al. (2019a) Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019a.  Can you unpack that? learning to rewrite questions-in-context.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5918–5924, Hong Kong, China. Association for Computational Linguistics.
  * Elgohary et al. (2019b) Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019b.  Can you unpack that? learning to rewrite questions-in-context.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5918–5924.
  * Ju et al. (2019) Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. 2019\.  Technical report on conversational question answering.  _arXiv preprint arXiv:1909.10772_.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019.  CTRL: A conditional transformer language model for controllable generation.  _CoRR_ , abs/1909.05858.
  * Kim et al. (2021) Gangwoo Kim, Hyunjae Kim, Jungsoo Park, and Jaewoo Kang. 2021.  Learn to resolve conversational dependency: A consistency training framework for conversational question answering.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6130–6141, Online. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kokhlikyan et al. (2020) Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-Richardson. 2020.  Captum: A unified and generic model interpretability library for pytorch.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2020) Sheng-Chieh Lin, Jheng-Hong Yang, Rodrigo Nogueira, Ming-Feng Tsai, Chuan-Ju Wang, and Jimmy Lin. 2020.  Conversational question reformulation via sequence-to-sequence architectures and pretrained language models.  _arXiv preprint arXiv:2004.01909_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * McAuley and Leskovec (2013) Julian McAuley and Jure Leskovec. 2013.  Hidden factors and hidden topics: Understanding rating dimensions with review text.  In _Proceedings of the 7th ACM Conference on Recommender Systems_ , RecSys ’13, page 165–172, New York, NY, USA. Association for Computing Machinery.
  * Miller et al. (2009) Frederic P. Miller, Agnes F. Vandome, and John McBrewster. 2009.  _Levenshtein Distance: Information Theory, Computer Science, String (Computer Science), String Metric, Damerau?Levenshtein Distance, Spell Checker, Hamming Distance_.  Alpha Press.
  * Mudrakarta et al. (2018) Pramod Kaushik Mudrakarta, Ankur Taly, Mukund Sundararajan, and Kedar Dhamdhere. 2018.  Did the model understand the question? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1896–1906, Melbourne, Australia. Association for Computational Linguistics.
  * Nogueira and Cho (2017) Rodrigo Nogueira and Kyunghyun Cho. 2017.  Task-oriented query reformulation with reinforcement learning.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 574–583, Copenhagen, Denmark. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  * Petrén Bach Hansen and Søgaard (2020) Victor Petrén Bach Hansen and Anders Søgaard. 2020.  What do you mean ‘why?’: Resolving sluices in conversations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7887–7894.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019.  CoQA: A conversational question answering challenge.  _Transactions of the Association for Computational Linguistics_ , 7:249–266.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms. _CoRR_ , abs/1707.06347.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning - Volume 70_ , ICML’17, page 3319–3328. JMLR.org.
  * Tan et al. (2020) Samson Tan, Shafiq Joty, Min-Yen Kan, and Richard Socher. 2020.  It’s morphin’ time! Combating linguistic discrimination with inflectional perturbations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2920–2935, Online. Association for Computational Linguistics.
  * Vakulenko et al. (2021) Svitlana Vakulenko, Shayne Longpre, Zhucheng Tu, and Raviteja Anantha. 2021.  Question rewriting for conversational question answering.  In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ , pages 355–363.
  * Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019.  EDA: easy data augmentation techniques for boosting performance on text classification tasks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 6381–6387. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xu et al. (2021) Yan Xu, Etsuko Ishii, Genta Indra Winata, Zhaojiang Lin, Andrea Madotto, Zihan Liu, Peng Xu, and Pascale Fung. 2021.  Caire in dialdoc21: Data augmentation for information seeking dialogue system.  In _Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021)_ , pages 46–51.
  * Yeh and Chen (2019) Yi-Ting Yeh and Yun-Nung Chen. 2019.  Flowdelta: Modeling flow information gain in reasoning for conversational machine comprehension.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 86–90.
  * Zhang et al. (2021a) Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2021a.  Trading off diversity and quality in natural language generation.  In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_ , pages 25–33, Online. Association for Computational Linguistics.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level Convolutional Networks for Text Classification .  _arXiv:1509.01626 [cs]_.
  * Zhang et al. (2021b) Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2021b.  Retrospective reader for machine reading comprehension.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(16):14506–14514.
  * Zhao et al. (2021) Jing Zhao, Junwei Bao, Yifan Wang, Yongwei Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2021.  RoR: Read-over-read for long document machine reading comprehension.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1862–1872, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhu et al. (2018) Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2018.  Sdnet: Contextualized attention-based deep network for conversational question answering.  _arXiv preprint arXiv:1812.03593_.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_.

##  Appendix A Additional Details of Experiments

Our implementation is based on Wolf et al. (2020), and we plan to release our
codes as well as trained models. Before applying our reinforcement learning
training, the QA and the QR models are initialized with the best QA and QR
models. The QA models are trained on CoQA and QuAC datasets, and model
selection is based on their F1 score. The QR models are trained on QReCC and
CANARD, and model selection is based on the BLEU Papineni et al.
(2002)A.1A.1A.1https://github.com/moses-
smt/mosesdecoder/blob/master/scripts/generic/multi-bleu-detok.perl score for
CANARD and ROUGE-1R Lin (2004) for QReCC, respectively. We report the other
hyperparameters in Table A.1 and Table A.2. We use Adam optimizer Kingma and
Ba (2015) for all the training.

The hyperparameters used for the PPO training are reported in Table A.3. For
rewrites generation with the QR model, we use beam search with beam width of
5, preventing generation repetition Keskar et al. (2019) with using repetition
penalty of 1.1, and set the maximum input sequence length to 512. We then run
the PPO with value function coefficient of 1.0, while ensuring the sequence
length of question rewriting model input to be 150 tokens maximum and the
generations length to be 50 tokens at maximum. To ensure that the learned
policy does not deviate too much, we apply an additional reward signal,
adaptive KL factor $\beta$ according to the magnitude of the KL-penalty with a
KL-coefficient $K_{\beta}=0.1$ following Ziegler et al. (2019).

Hyperparameter settings | CoQA | QuAC
---|---|---
Model architecture | RoBERTa-base
Model size | 125M parameters
Optimizer | Adam
Learning rate | $3\mathrm{e}{-5}$
Warmup steps | 1000 | 0
Weight decay | 0.01 | 0.01
Gradient accumulation steps | 10 | 20
Early stopping patience | 3 | 4
Batch size | 6
Maximum epoch | 10
Document stride | 128
Maximum sequence length | 512
Maximum answer length | 50

Table A.1: Hyperparameters for initialization of QA models.

Hyperparameter settings | Value
---|---
Model architecture | GPT-2 base
Model size | 117M parameters
Optimizer | Adam
Learning rate | $3\mathrm{e}{-5}$
Warmup steps | 500
Batch size | 8
Gradient accumulation steps | 8
Early stopping patience | 3
Maximum epoch | 20
History length (utterances) | 3
Maximum sequence length | 256

Table A.2: Hyperparameters for initialization of QR models both in QRECC and
CANARD dataset training.

Hyperparameter settings | Value
---|---
Training settings |
Optimizer | Adam
Learning rate | 1E-5, 1E-6, (1E-7), 1E-8, 1E-9
Batch size | 8 (CoQA), 16 (QuAC)
Early stopping patience | 3
Max epoch | 6
QA history length | 3
Max query rewrite length | 100
QR max sequence length | 150
Max sequence length | 512
Max question length | 128
Pad to maximum length | TRUE
Document stride | 128
N best answers to generate | 20
Max answer length | 5
PPO settings |
Maximum PPO epoch | 4
KL coefficient init | 0.2
Target | 6
Horizon | 10000
gamma | 1
lambda | 0.95
cliprange | 0.2
vf_coef | 0.5
ce_coef | 1

Table A.3: Hyperparameters for PPO training and inference (Best parameter is
in bracket).

##  Appendix B Details of Robustness Experiment

We perform robustness testing by introducing a minor perturbation which
minimally change the semantic of the sentence. We utilize NL-Augmenter toolkit
Dhole et al. (2021) B.1B.1B.1https://github.com/GEM-benchmark/NL-Augmenter to
generate 5 different type of perturbations, i.e., random upper casing Wei and
Zou (2019) B.2B.2B.2https://github.com/GEM-benchmark/NL-
Augmenter/tree/main/transformations/random_upper_transformation, contraction
expansion Ribeiro et al. (2020) B.3B.3B.3https://github.com/GEM-benchmark/NL-
Augmenter/tree/main/transformations/contraction_expansions, word inflection
variation Tan et al. (2020) B.4B.4B.4https://github.com/GEM-benchmark/NL-
Augmenter/tree/main/transformations/english_inflectional_variation, slang word
B.5B.5B.5https://github.com/GEM-benchmark/NL-
Augmenter/tree/main/transformations/slangificator, and sentence paraphrasing
using yoda transformation B.6B.6B.6https://github.com/GEM-benchmark/NL-
Augmenter/tree/main/transformations/yoda_transform perturbations. For random
upper casing, we apply a probability of 10% to randomly upper case letters.
For contraction expansion, inflection variation, and slang word variation
perturbations, we apply them without randomness by replacing all occurrences
of words that fulfill the corresponding rules of each perturbation so as to
increase the difference to the original sentence. For sentence paraphrasing,
we apply a rule-based yoda paraphrasing which reconstruct the sentence into
its “XSV” syntax format, where the ‘S’ stands for subject, the ‘V’ stands for
verb, and the ‘X’ being a stand-in for whatever chunk of the sentence goes
after the verb from the original “SVX” syntax format. Table B.1 shows the BLEU
score Papineni et al. (2002) and the Levensthein distance Miller et al. (2009)
from each perturbation compared to the original sentence in each dataset. We
do not report contraction expansion perturbation due to the minute differences
with the original sentence.

Perturb | Amazon | Yelp | CoQA | QuAC
---|---|---|---|---
LD | BLEU | LD | BLEU | LD | BLEU | LD | BLEU
UPC | 1.0% | 94.0 | 0.2% | 98.1 | 1.2% | 93.6 | 0.8% | 95.6
WCE | 7.3% | 43.1 | 7.4% | 34.4 | 6.8% | 36.7 | 7.1% | 33.8
SLW | 5.0% | 89.9 | 5.1% | 86.8 | 3.5% | 91.3 | 1.5% | 95.7
WIF | 8.8% | 52.7 | 7.9% | 54.6 | 10.4% | 44.0 | 9.1% | 47.8
SPP | 31.1% | 57.3 | 32.7% | 68.5 | 53.4% | 34.6 | 46.4% | 39.9

Table B.1: Perturbation statistics on Amazon, Yelp, CoQA, and QuAC tasks. LD
denotes the Levensthein distance divided by the number of characters in the
original dataset. BLEU denotes BLEU score. We apply five perturbations: UPC
(upper casing), WCE (word contraction expansion), SLW (slang word), WIF (word
inflection), and SPP (sentence paraphrasing).

##  Appendix C Details of Data Augmentation Approach

Settings | CoQA | QuAC
---|---|---
Overall F1 | Child. | Liter. | M&H | News | Wiki. | F1 | HEQ-Q | HEQ-D
end-to-end | 84.5 | 84.4 | 82.4 | 82.9 | 86.0 | 86.9 | 67.8 | 63.5 | 7.9
QReCC-augment | 84.1 | 84.1 | 81.9 | 82.4 | 85.4 | 86.7 | 67.8 | 63.4 | 8.1
CANARD-augment | 83.7 | 83.8 | 81.6 | 81.9 | 85.1 | 86.4 | 67.8 | 63.5 | 7.8

Table C.1: Evaluation results of the data augmentation methods. Our data
augmentation approach scores constantly worse than the end-to-end setting for
CoQA, and almost the same for QuAC.

We investigate a data augmentation approach. While we adopt the question
rewrites obtained by the QR model to fine-tune the QA model regardless of
their quality, we filter rewrites based on their F1 scores. We first generate
10 possible rewrites using $k=20$ and $p=0.95$ with top-$k$ sampling Zhang et
al. (2021a) for all the questions in the training and validation set of CoQA
and QuAC with the QR model pretrained either on QReCC or CANARD. To guarantee
the quality of the rewrites, we select the best scoring one out of 10
candidates on the corresponding CQA or use the original question if the
original question gets a better F1 score. Then, we finetune another QR model
using the newly annotated question rewrites pairs $\\{X_{t},\acute{Q}_{t}\\}$
so that the model learns how to reformulate questions. Here, following our
main experiments, we use pretrained GPT-2 as initialization for finetuning. We
use Adam optimizer Kingma and Ba (2015) with the learning rate of
$3\mathrm{e}{-5}$, and the other hyperparameters follow Table A.2.

We report the evaluation results on CoQA and QuAC in Table C.1. We cannot find
any justification to apply the data augmentation approach from our
experiments. We constantly get worse scores compared to the end-to-end
settings in CoQA, and almost the same for QuAC. We could not observe any
meaningful improvement, except that it performs better than the pipeline
settings (see Table 1 for comparison).

##  Appendix D Additional Analysis: Gradient Saliency Analysis

Inspired by Mudrakarta et al. (2018), we construct a saliency map for each
sample in CoQA to observe which part of the inputs counts for the predictions
using Integrated Gradient analysis (Kokhlikyan et al., 2020; Sundararajan et
al., 2017). Based on our QA model implementation, we construct a saliency map
for the predictions on a start index, end index, and on a label to determine
whether it is a yes/no/unknown question.

Examples of the saliency map are depicted in Figure D.3, D.3, and D.3
respectively. For the sake of better interpretability, we compute the
distribution of tokens that affects the QA model predictions either positively
or negatively. We first count each tokens’ positive ($>0$) and negative ($<0$)
attributions, and then normalized with each input lengths. The normalized
value then taken as a distribution over questions, dialogue histories, and
contexts. Results are reported in Table D.1 \- D.4. Note that the total per
column is not always 100% due to rounding. From the comparison between Table
D.1 and Table D.3 or between Table D.2 and Table D.4, we only observe marginal
differences between the distributions. It is linked to the issue we report in
the main paper that the QR models learned to copy the original questions.

From the comparison between Table D.1 and Table D.2 or between Table D.3 and
Table D.4, we can see that the dialogue histories have more effects on the
model predictions than questions, however, the questions give more significant
contributions to them. Throughout the gradient saliency analysis, we cannot
find any justification to apply QR for CQA, since our results could be
interpreted as the QA model can leverage the dialogue history without QR.

Input | Distribution of Contributing Tokens
---|---
start index | end index | yes/ no/ unknown
>0 | <0 | >0 | <0 | >0 | <0
$\acute{Q}_{t}$ | 32.7% | 25.9% | 29.9% | 30.0% | 3.5% | 0.8%
$\mathcal{D}_{t-1}$ | 33.2% | 37.5% | 33.5% | 37.5% | 28.6% | 14.0%
$Y_{t}$ | 33.9% | 36.5% | 36.5% | 32.4% | 67.7% | 85.1%

Table D.1: Results of Integrated Gradient Analysis on the CoQA inputs using
the rewritten questions.

Input | Distribution of Contributing Tokens
---|---
start index | end index | yes/ no/ unknown
>0.5 | <= -0.5 | >0.5 | <= -0.5 | >0.5 | <= -0.5
$\acute{Q}_{t}$ | 3.9% | 0.9% | 8.4% | 1.0% | 0.0% | 0.0%
$\mathcal{D}_{t}$ | 1.4% | 1.7% | 1.9% | 1.0% | 100.0% | 100.0%
$Y_{t}$ | 94.6% | 97.2% | 89.5% | 97.9% | 0.0% | 0.0%

Table D.2: Results of Integrated Gradient Analysis on the CoQA inputs using
the rewritten questions for significantly contributing tokens.

Input | Distribution of Significantly Contributing Tokens
---|---
start index | end index | yes/ no/ unknown
>0 | <0 | >0 | <0 | >0 | <0
$Q_{t}$ | 32.8% | 25.6% | 29.8% | 29.8% | 3.5% | 0.8%
$\mathcal{D}_{t}$ | 33.2% | 37.7% | 33.5% | 37.6% | 28.7% | 14.0%
$Y_{t}$ | 33.9% | 36.6% | 36.5% | 32.5% | 67.6% | 85.1%

Table D.3: Results of Integrated Gradient Analysis on the CoQA inputs using
the original questions.

Input | Distribution of Significantly Contributing Tokens
---|---
start index | end index | yes/ no/ unknown
>0.5 | <= -0.5 | >0.5 | <= -0.5 | >0.5 | <= -0.5
$Q_{t}$ | 3.9% | 1.0% | 8.2% | 1.0% | 0.0% | 0.0%
$\mathcal{D}_{t}$ | 1.6% | 2.0% | 2.0% | 1.0% | 100.0% | 100.0%
$Y_{t}$ | 94.4% | 96.9% | 89.6% | 97.9% | 0.0% | 0.0%

Table D.4: Results of Integrated Gradient Analysis on the CoQA inputs using
the original questions for significantly contributing tokens.

Figure D.1: Visualization of Integrated Gradient saliency analysis on our RL
approach that predicts the start index correctly. Round brackets indicate the
QA model predictions of the start and end index, and square brackets the gold
start and end index.

Figure D.2: Visualization of Integrated Gradient saliency analysis on our RL
approach that predicts the end index correctly. Round brackets indicate the QA
model predictions of the start and end index, and square brackets the gold
start and end index.

Figure D.3: Visualization of Integrated Gradient saliency analysis on our RL
approach that correctly the answers yes/no/unknown question.