* parametric or nonparametric

* context-dependent or context-independent

* composite measures or holistic measures

* human judgment-dependent or human

* transfer learning-based or non-transfer learning-based

It is important to note that the above properties lead to a certain self-similarity in the taxonomy of error scores. For instance, while all S-UAES and A-UAES do not involve human judgement, some scores in the category MTES do, e.g., ADEM or RUSE.

In the following, we discuss briefly the main drawback of particular properties of error scores.

§.§  Drawback of word-overlap scores

A drawback of word-overlap scores (e.g., BLEU, ROUGE) is their inability to capture the semantic similarity between the answer and reference responses when there are few or even no common words. In [Lowe et al., 2017] it was argued that this problem is less severe for machine translation than dialog systems because for the latter the number of appropriate responses given in a context is much larger implying a response diversity. Indeed, almost all word-overlap scores were proposed for machine translation; and then adapted for other NLG tasks. However, in [Callison-Burch et al., 2006] it was shown that even in machine translation word-overlap scores still have multiple weaknesses and cannot reflect the quality of a model fully.

Another drawback of word-overlap scores is not considering the context for the evaluation. Instead, they use only the response and reference. Importantly, in [Lin, 2004] it was shown that neither of the word-overlap-based scores has any correlation to human judgment [Deriu et al., 2021]. For this reason, [Galley et al., 2015] proposed to include human judgment into the BLEU score, which they called deltaBLEU. For deltaBLEU human judges rate the reference responses of the test set according to the relevance of the context. However, creating a high-quality reference dataset for this is a costly and arduous task. Furthermore, due to the fact that one question or query can have multiple correct answers, more than one reference needs to be created for each query.

§.§  Drawback of learning-based scores

Despite the fact that learning-based scores constantly outperform word-overlap scores in evaluating semantic similarities,
there are some drawbacks.
Some of the learning-based scores such as RUSE and ADEM are trained on data with human annotations, which is considered expensive as human annotation is time-consuming and costly. Due to this limitation, the training data is limited, hence they often suffer from significant performance loss when the test data is out-of-distribution.

Even though some later proposed metrics, e.g., BLEURT and BERTscore can utilize transfer learning to accommodate training data that cover more domains, the problem of domain shift still remains when the training data do not cover the entire scope of test data. Meanwhile, such metrics can only be used for evaluating a language that is trained for evaluation metrics, however, the evaluation quality across different languages can vary.

Another problem is that learning-based scores generally lack interpretability. This is related to the fact that human annotators usually assign several scores, e.g., according to different criteria (fluency, coherence, relevance, etc.). Furthermore, word-overlap scores are considered to capture only the syntactic similarity of a given context. Hence, people fail to explain what aspects the resulting score reflects. Instead, a general statement, for instance, "capturing semantic similarity" is provided by most authors to summarize the meaning of learning-based scores.

§.§  Drawback of scores based on a pre-trained model

Usually, the training of a language model from scratch requires large amounts of data, which is not available in all situations. Hence, many scores are based on pre-trained models. However, the quality of the scores depends on the quality of the pre-trained models. Although the pre-trained models generally perform well, there is no guarantee that the performance is consistent in different situations, domains, and languages.

§.§ Drawback of scores based on human judgement

In general, HCES based on human judgement is expensive, time-consuming, tedious, and requires domain expertise, which makes it cumbersome and impractical when dealing with a large task that may involve some dynamic behavior. Also, human judgments may differ from person-to-person, which reduces their reliability and reproducibility among different groups of experts [Gatt and Krahmer, 2018, Reiter and Belz, 2009, Doddington, 2002].

§.§ Major problem

We hypothesize that the major problem with current evaluations of QA systems can be formulated as follows:

The best way of evaluating QA systems is by human judgement. However, the problem is that such an evaluation is done qualitatively based on personal sentiments but the quantification of such a human judgement is still largely undefined.

Hence, the lack of a clear definition for the quantification of human judgement is the major obstacle to defining an MTCE and providing an automatic evaluation.

As a first step toward such a formulaization one could say the quantification of human judgement is based on a high-dimensional feature vector $v \in \mathbb{R}^d$ where $d\in \mathbb{N}$ is the number of features. As a second step, one could learn the representation of the vector space of $v$ via a deep learning neural network. This establishes the evaluation as a second model, while the first model is the QA system itself. In step three, we hypothezise that the evaluation model is at least as complex as the QA system. The reason for this is that it requires a complete comprehension of language semantics.

§ DISCUSSION

In general, comparing two or more texts with each other is a non-trivial task. This is even further complicated by the fact that one can define dedicated sub-tasks thereof. Specifically, the problem of comparing two texts occurs in (I) machine translation, (II) text summarization, (III) question answering, (IV) dialogue systems (also called conversational agents), and (V) visual question answering. While each of these tasks produces a response in the form of text, the resulting forms of the output are quite different from each other. Put briefly, machine translation converts text automatically from one language into another language,  text summarization compresses a longer text into a shorter text without removing the semantic meaning of the longer text, question answering produces a response for a given question, and a dialogue system leads to an ongoing conversation and visual question answering describes information visually presented in form of an image or video. Due to the fact that each of the above sub-tasks is highly complex and an ongoing research field, we focus in this survey on question answering.

Starting from a discussion of the question answering framework (in Section <ref>) and its four main components, (i) QA algorithms, (ii), knowledge source, (iii) question types, and (vi) answer types, we introduced a formal definition of a question answering system (Section <ref>). This definition allows a summarization of practical elements of general QA systems by emphasizing their functional mappings. Hence, this abstract perspective enables an efficient focus on integral aspects of question answering.

For the practical constructing of QA systems, we discussed the three main paradigms: (1) Information Retrieval-Based Question Answering (IRQA), (2) Knowledge Base Question Answering (KBQA), and (3) Generative Question Answering (GQA). Due to the fact that IRQA systems consist of two different components (retriever and reader), each with different functionalities, it is reasonable to use different scores to evaluate each component separately. The retriever component uses to retrieve the most candidate answers according to the top-k parameter that is chosen to determine the number of candidates to be returned. Evaluation scores such as Recall and MRR are then used to evaluate the retriever component and to test if the document that contains the right answer is among the retrieved candidates. Then the IRQA system directly ranks the answers retrieved from the knowledge source by considering information in the question itself. From the topic entity, the system may use different methods to extract all the entities and relationships related to a question, e.g., nodes and edges, and then arrange them, respectively. Next, the system encodes and analyzes the semantics of the input question and represents it as vectors. The semantic matching is then taken via vector-based measures to spread and aggregate the information along with the contiguous entities in the graph or within a given context. The answers are then arranged, and the higher-ranked entities are utilized to find the predicted answer for a given question. The reader component, in contrast, may use a simple score such as Exact Match or F1-score to evaluate the extracted (predicted) answer and find out to which extent it is close or matched with the gold answer. Such scores may capture the lexical overlap, but they cannot capture the similarity between two correct answers that differ in their semantic content. Some examples for Exact Match and F1-score are given in Section <ref>. However, more complex error scores like BERTScore or BLEURT are needed in order to capture semantic similarity.

Predicting an answer in a KBQA system depends on the ability of reading-comprehension that is responsible for fully absorbing and understanding the question [Yue et al., 2017, Kumar et al., 2016]. This is achieved by conducting a semantic and grammatical analysis that is required for obtaining the encoded question, and then representing it logically in a justified manner according to grammatical rules defined by the system. The logical form can be validated by a semantic alignment for the structured KB or by simultaneously performing a logical analysis with knowledge base grounding. The logical models are then validated in the KB during partial parsing. Thus, the logic model of a KB is analyzed and the predicted answers are extracted.

It is interesting to note that some IRQA systems have also been designed to perform complex reasoning on the question structure and flexible semantic matching that may fit easily into the nature of end-to-end training. However, the interphases of this system may be less interpretable. In general, KBQA can provide good insights and explanations by developing a logical form, but it still relies heavily on the logical form design and the parsing algorithm, which has been proven to be a bottleneck in improving performance. Due to the complexity of knowledge representation, mapping user questions to logical or semantic queries, and providing an answer, is not an easy task for KBQA, especially for complex questions.

In contrast, a GQA system can be considered an open-ended text-generation problem that may not include all possible references related to a particular question. This implies that for a semantic evaluation one would not only need human annotated references but many alternative forms thereof to capture the many variations which can be generated by a GQA system itself. This is an unsolved problem because of the associated costs and complexity. Therefore, GQA systems are frequently not evaluated semantically but lexically, e.g., using n-gram overlap, for UAES (such as Precision, Recall, F1, BLEU, ROUGE, METEOR) [Chen et al., 2019]. However, it should be clear that computing the n-gram matches between the generated question and reference does usually not allow to distinguish between  salient or important from non-salient or non-important types of n-grams. Hence, such scores are not able to weigh different n-grams based on their importance or based on the actual content, instead, they weigh all  n-grams equally, and this indifference can affect the performance negatively.

In general, drawing a conclusion about the quality of a QA system requires an informed assessment using annotated datasets together with evaluation scores that allow the quantification of the performance. For this reason, many datasets with different structures (e.g., document-based or knowledge-based) have been prepared for different question types (e.g., standalone, sequential, multiple-choice) along with different answer types (i.e., simple, abstractive, agnostic) for benchmarking QA systems (see Section <ref>). The different benchmark data address the different requirements and variations of  QA tasks. It is interesting to note that since the beginnings question-answering datasets have evolved dramatically becoming more and more complex. This acknowledges also the requirements of GQA systems due to their flexibility to generate text in an open-ended form (see discussion above).

The heterogeneous tasks of QA systems demand not only problem-specific benchmark data but also heterogeneous evaluation scores. This explains the large number of different evaluation scores that have been proposed over the years. In Section <ref>, we introduced a hierarchical taxonomy showing that such error scores can be broadly classified into human-centric evaluation scores (HCES) and automatic evaluation scores (AES). In general, HCES is considered the best evaluation that gives the most trustworthy score, however, due to drawbacks mentioned in Section <ref>, AES are more practical. ASE measures are further sub-categorized in Untrained Automatic Evaluation Scores (UAES) and Machine-trained Evaluation Scores (MTES) to address the requirements of the agnostic or specific tasks of a QA system.

The UAES measures do not require any pre-training, because such measures do not depend on parameters. In contrast, MTES contain adjustable components, i.e., parameters, which need to be estimated for a given task via a learnable model. The MTES measures can be further sub-categorized in composite and holistic measures. The composite measures are also called feature-based measures, and holistic measures are called end-to-end measures. The difference is that a holistic measure cannot be separated into individual error scores while this is the case for feature-based measures where the features correspond to such individual error scores.

Finally, we would like to highlight that the availability of many evaluation scores posses the problem of selecting an appropriate one [Emmert-Streib et al., 2019]. This is another non-trivial task because the interpretation and selection of the presented evaluation scores is challenging and currently there is no universal score that would be best in all situations. For instance, a score such as F1 may show adequate results on some existing spanning-based datasets, but it may be inadequate for other QA tasks. Hence, for each situation a comparative analysis is needed, similar to model selection [Ding et al., 2018, Emmert-Streib and Dehmer, 2019], for identifying the most favorable evaluation score.

§ CONCLUSION

The question-answering task is among the oldest challenges in artificial intelligence and it is still one of the most important tasks in natural language processing to this day as it enables humans to interact with a machine in a natural way. For dealing with domain-specific information and different formats of data which could be either structured or unstructured data there are three main paradigms for constructing QA systems: (1) Information Retrieval-Based Question Answering, (2) Knowledge Base Question Answering, and (3) Generative Question Answering.

In this paper, a comprehensive survey of question-answering systems was presented based on the general architecture of the question-answering framework. Despite the heterogeneity of QA tasks and design principles, we identified commonalities and introduced a general definition of a question-answering system allowing us to summarize its formal structure abstractly. In order to understand key assumptions and design principles behind QA systems, this paper described also task-specific benchmark datasets and evaluation scores. For the latter, taxonomy has been introduced where its main branches are separating human-centric evaluation scores (HCES)from automatic evaluation scores (AES) and untrained automatic evaluation scores (UAES) from machine-trained evaluation scores (MTES). Overall, we have seen that not only the construction of a QA system is complex but also its evaluation and an open problem is the quantitative formalization of the human judgment underlying HCES which could be translated into MTES.

[Bates, 1995]
Madeleine Bates.
Models of natural language understanding.
Proceedings of the National Academy of Sciences, 920
(22):0 9977–9982, 1995.

[Liu et al., 2021]
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser.
Benchmarking natural language understanding services for building
conversational agents.
In Increasing Naturalness and Flexibility in Spoken Dialogue
Interaction, pages 165–183. Springer, 2021.

[Radford et al., 2018]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
Improving language understanding by generative pre-training.

[Reiter and Dale, 1997]
Ehud Reiter and Robert Dale.
Building applied natural language generation systems.
Natural Language Engineering, 30 (1):0 57–87,

[Gatt and Krahmer, 2018]
Albert Gatt and Emiel Krahmer.
Survey of the state of the art in natural language generation: Core
tasks, applications and evaluation.
Journal of Artificial Intelligence Research, 61:0
65–170, 2018.

[Cho et al., 2014]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
Learning phrase representations using rnn encoder-decoder for
statistical machine translation.
arXiv preprint arXiv:1406.1078, 2014.

[Sutskever et al., 2014]
Ilya Sutskever, Oriol Vinyals, and Quoc V Le.
Sequence to sequence learning with neural networks.
Advances in neural information processing systems, 27, 2014.

[Bahdanau et al., 2014]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align and
arXiv preprint arXiv:1409.0473, 2014.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Peñas et al., 2012]
Anselmo Peñas, Bernardo Magnini, Pamela Forner, Richard Sutcliffe, Alvaro
Rodrigo, and Danilo Giampiccolo.
Question answering at the cross-language evaluation forum 2003–2010.
Language resources and evaluation, 460 (2):0
177–217, 2012.

[Androutsopoulos et al., 1995]
Ion Androutsopoulos, Graeme D Ritchie, and Peter Thanisch.
Natural language interfaces to databases–an introduction.
Natural language engineering, 10 (1):0 29–81,

[Kolomiyets and Moens, 2011]
Oleksandr Kolomiyets and Marie-Francine Moens.
A survey on question answering technology from an information
retrieval perspective.
Inform Sciences, 1810 (24):0 5412–5434, 2011.

[Green Jr et al., 1961]
Bert F Green Jr, Alice K Wolf, Carol Chomsky, and Kenneth Laughery.
Baseball: an automatic question-answerer.
In Papers presented at the May 9-11, 1961, western joint
IRE-AIEE-ACM computer conference, pages 219–224, 1961.

[Woods, 1973]
William A Woods.
Progress in natural language understanding: an application to lunar
In Proceedings of the June 4-8, 1973, national computer
conference and exposition, pages 441–450, 1973.

[Voorhees et al., 1999]
Ellen M Voorhees et al.
The trec-8 question answering track report.
In Trec, volume 99, pages 77–82, 1999.

[Yih et al., 2015]
Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao.
Semantic parsing via staged query graph generation: Question
answering with knowledge base.
In Proceedings of the Joint Conference of the 53rd Annual
Meeting of the ACL and the 7th International Joint Conference on Natural
Language Processing of the AFNLP, 2015.

[Indurthi et al., 2017]
Sathish Reddy Indurthi, Dinesh Raghu, Mitesh M Khapra, and Sachindra Joshi.
Generating natural language question-answer pairs from a knowledge
graph using a rnn based question generation model.
In Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 1, Long Papers,
pages 376–385, 2017.

[Pan et al., 2019]
Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan.
Recent advances in neural question generation.
arXiv preprint arXiv:1905.08949, 2019.

[Zhang et al., 2021]
Ruqing Zhang, Jiafeng Guo, Lu Chen, Yixing Fan, and Xueqi Cheng.
A review on question generation from natural language text.
ACM Transactions on Information Systems (TOIS), 400
(1):0 1–43, 2021.

[Sukhbaatar et al., 2015]
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.
End-to-end memory networks.
Advances in neural information processing systems, 28, 2015.

[Yue et al., 2017]
Chunyi Yue, Hanqiang Cao, Kun Xiong, Anqi Cui, Haocheng Qin, and Ming Li.
Enhanced question understanding with dynamic memory networks for
textual question answering.
Expert Systems with Applications, 80:0 39–45, 2017.

[Kumar et al., 2016]
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan
Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher.
Ask me anything: Dynamic memory networks for natural language
In International conference on machine learning, pages
1378–1387. PMLR, 2016.

[Ojokoh and Adebisi, 2018]
Bolanle Ojokoh and Emmanuel Adebisi.
A review of question answering systems.
J Web Eng, 170 (8):0 717–758, 2018.

[Luo et al., 2018]
Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu.
Knowledge base question answering via encoding of complex query
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 2185–2194, 2018.

[Zhu et al., 2020]
Shuguang Zhu, Xiang Cheng, and Sen Su.
Knowledge-based question answering by tree-to-sequence learning.
Neurocomputing, 372:0 64–72, 2020.

[Qu et al., 2018]
Yingqi Qu, Jie Liu, Liangyi Kang, Qinfeng Shi, and Dan Ye.
Question answering over freebase via attentive rnn with similarity
matrix based cnn.
arXiv preprint arXiv:1804.03317, 38, 2018.

[Lan and Jiang, 2020]
Yunshi Lan and Jing Jiang.
Query graph generation for answering multi-hop complex questions from
knowledge bases.
Association for Computational Linguistics, 2020.

[Dimitrakis et al., 2020]
Eleftherios Dimitrakis, Konstantinos Sgontzos, and Yannis Tzitzikas.
A survey on question answering systems over linked data and
Journal of intelligent information systems, 550
(2):0 233–259, 2020.

[Cortes et al., 2020]
Eduardo Cortes, Vinicius Woloszyn, Arne Binder, Tilo Himmelsbach, Dante Barone,
and Sebastian Möller.
An empirical comparison of question classification methods for
question answering systems.
In Proceedings of the 12th Language Resources and Evaluation
Conference, pages 5408–5416, 2020.

[Zhang et al., 2019]
Siheng Zhang, Wensheng Zhang, and Jinghao Niu.
Improving short-text representation in convolutional networks by
dependency parsing.
Knowledge and Information Systems, 610 (1):0
463–484, 2019a.

[Ben Abacha and Demner-Fushman, 2019]
Asma Ben Abacha and Dina Demner-Fushman.
A question-entailment approach to question answering.
BMC bioinformatics, 200 (1):0 1–23, 2019.

[Hu et al., 2017]
Sen Hu, Lei Zou, Jeffrey Xu Yu, Haixun Wang, and Dongyan Zhao.
Answering natural language questions by subgraph matching over
knowledge graphs.
IEEE Trans. Knowl. Data Eng., 300 (5):0
824–837, 2017.

[Lan et al., 2019]
Yunshi Lan, Shuohang Wang, and Jing Jiang.
Knowledge base question answering with a matching-aggregation model
and question-specific contextual relations.
IEEE/ACM Trans. Audio, Speech, Language Process., 270
(10):0 1629–1638, 2019a.

[He et al., 2019]
Junqing He, Mingming Fu, and Manshu Tu.
Applying deep matching networks to chinese medical question
answering: a study and a dataset.
BMC medical informatics and decision making, 190
(2):0 91–100, 2019.

[Zhang et al., 2019]
Xu Zhang, Wenpeng Lu, Fangfang Li, Xueping Peng, and Ruoyu Zhang.
Deep feature fusion model for sentence semantic matching.
Computers, Materials and Continua, 2019b.

[Abbasiantaeb and Momtazi, 2021]
Zahra Abbasiantaeb and Saeedeh Momtazi.
Text-based question answering from information retrieval and deep
neural network perspectives: A survey.
Wiley Interdisciplinary Reviews: Data Mining and Knowledge
Discovery, 110 (6):0 e1412, 2021.

[Lan et al., 2021]
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong
A survey on complex knowledge base question answering: Methods,
challenges and solutions.
arXiv preprint arXiv:2105.11644, 2021.

[Hao et al., 2022]
Tianyong Hao, Xinxin Li, Yulan He, Fu Lee Wang, and Yingying Qu.
Recent progress in leveraging deep learning methods for question
Neural Computing and Applications, pages 1–19, 2022.

[Lee et al., 2019]
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.
Latent retrieval for weakly supervised open domain question
arXiv preprint arXiv:1906.00300, 2019a.

[Yang et al., 2017]
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W Cohen.
Semi-supervised qa with generative domain-adaptive nets.
arXiv preprint arXiv:1702.02206, 2017.

[Fabbri et al., 2020]
Alexander R Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang.
Template-based question generation from retrieved sentences for
improved unsupervised question answering.
arXiv preprint arXiv:2004.11892, 2020.

[Bashath et al., 2022]
Samar Bashath, Nadeesha Perera, Shailesh Tripathi, Kalifa Manjang, Matthias
Dehmer, and Frank Emmert Streib.
A data-centric review of deep transfer learning with applications to
text data.
Information Sciences, 585:0 498–528, 2022.

[Alwaneen et al., 2021]
Tahani H Alwaneen, Aqil M Azmi, Hatim A Aboalsamh, Erik Cambria, and Amir
Arabic question answering system: a survey.
Artif Intell Rev, pages 1–47, 2021.

[Mishra and Jain, 2016]
Amit Mishra and Sanjay Kumar Jain.
A survey on question answering systems with classification.
Journal of King Saud University-Computer and Information
Sciences, 280 (3):0 345–361, 2016.

[Diefenbach et al., 2018]
Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret.
Core techniques of question answering systems over knowledge bases: a
Knowledge and Information systems, 550 (3):0
529–569, 2018.

[Ni et al., 2021]
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga, and Erik Cambria.
Recent advances in deep learning based dialogue systems: A systematic
arXiv preprint arXiv:2105.04387, 2021.

[Wu et al., 2017]
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van
Den Hengel.
Visual question answering: A survey of methods and datasets.
Computer Vision and Image Understanding, 163:0 21–40,

[Srivastava et al., 2020]
Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee.
Visual question answering using deep learning: A survey and
performance analysis.
In International Conference on Computer Vision and Image
Processing, pages 75–86. Springer, 2020.

[Deriu et al., 2021]
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset,
Eneko Agirre, and Mark Cieliebak.
Survey on evaluation methods for dialogue systems.
Artificial Intelligence Review, 540 (1):0
755–810, 2021.

[Emmert-Streib et al., 2019]
Frank Emmert-Streib, Salisou Moutari, and Matthias Dehmer.
A comprehensive survey of error measures for evaluating binary
decision making in data science.
Wiley Interdisciplinary Reviews: Data Mining and Knowledge
Discovery, 90 (5):0 e1303, 2019.

[Sai et al., 2022]
Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra.
A survey of evaluation metrics used for nlg systems.
ACM Computing Surveys (CSUR), 550 (2):0 1–39,

[Celikyilmaz et al., 2020]
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao.
Evaluation of text generation: A survey.
arXiv preprint arXiv:2006.14799, 2020.

[Wang et al., 2007]
Mengqiu Wang, Noah A Smith, and Teruko Mitamura.
What is the jeopardy model? a quasi-synchronous grammar for qa.
In Proceedings of the 2007 joint conference on empirical
methods in natural language processing and computational natural language
learning (EMNLP-CoNLL), pages 22–32, 2007.

[Sammut and Banerji, 1986]
Claude Sammut and Ranan B Banerji.
Learning concepts by asking questions.
Machine learning: An artificial intelligence approach,
2:0 167–192, 1986.

[Liu et al., 2017]
Tianyu Liu, Bingzhen Wei, Baobao Chang, and Zhifang Sui.
Large-scale simple question generation by template-based seq2seq
In National CCF Conference on Natural Language Processing and
Chinese Computing, pages 75–87. Springer, 2017.

[Straach and Truemper, 1999]
Janell Straach and Klaus Truemper.
Learning to ask relevant questions.
Artificial Intelligence, 1110 (1-2):0
301–327, 1999.

[Harabagiu et al., 2005]
Sanda Harabagiu, Andrew Hickl, John Lehmann, and Dan Moldovan.
Experiments with interactive question-answering.
In Proceedings of the 43rd annual meeting of the association
for computational linguistics (ACL’05), pages 205–214, 2005.

[Heilman and Smith, 2010]
Michael Heilman and Noah A Smith.
Good question! statistical ranking for question generation.
In Human Language Technologies: The 2010 Annual Conference of
the North American Chapter of the Association for Computational Linguistics,
pages 609–617, 2010.

[Levy and Andrew, 2006]
Roger Levy and Galen Andrew.
Tregex and tsurgeon: Tools for querying and manipulating tree data
In LREC, pages 2231–2234. Citeseer, 2006.

[Dhole and Manning, 2020]
Kaustubh D Dhole and Christopher D Manning.
Syn-qg: Syntactic and shallow semantic rules for question generation.
arXiv preprint arXiv:2004.08694, 2020.

[Yao and Zhang, 2010]
Xuchen Yao and Yi Zhang.
Question generation with minimal recursion semantics.
In Proceedings of QG2010: The Third Workshop on Question
Generation, pages 68–75. Citeseer, 2010.

[Ko et al., 2007]
Jeongwoo Ko, Luo Si, and Eric Nyberg.
A probabilistic framework for answer selection in question answering.
In Human Language Technologies 2007: The Conference of the
North American Chapter of the Association for Computational Linguistics;
Proceedings of the Main Conference, pages 524–531, 2007.

[Zhou et al., 2016]
Xiangyang Zhou, Daxiang Dong, Hua Wu, Shiqi Zhao, Dianhai Yu, Hao Tian, Xuan
Liu, and Rui Yan.
Multi-view response selection for human-computer conversation.
In Proceedings of the 2016 conference on empirical methods in
natural language processing, pages 372–381, 2016.

[Wu et al., 2016]
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li.
Sequential matching network: A new architecture for multi-turn
response selection in retrieval-based chatbots.
arXiv preprint arXiv:1612.01627, 2016.

[Jing et al., 2019]
Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin
Soljacic, and Yoshua Bengio.
Gated orthogonal recurrent units: On learning to forget.
Neural computation, 310 (4):0 765–783, 2019.

[Cho et al., 2019]
Woon Sang Cho, Yizhe Zhang, Sudha Rao, Chris Brockett, and Sungjin Lee.
Generating a common question from multiple documents using
multi-source encoder-decoder models.
arXiv preprint arXiv:1910.11483, 2019a.

[Huang et al., 2018]
Heyan Huang, Xiaochi Wei, Liqiang Nie, Xianling Mao, and Xin-Shun Xu.
From question to text: Question-oriented feature attention for answer
ACM Transactions on Information Systems (TOIS), 370
(1):0 1–33, 2018.

[Kim et al., 2018]
Seongsoon Kim, Donghyeon Park, Yonghwa Choi, Kyubum Lee, Byounggun Kim, Minji
Jeon, Jihye Kim, Aik Choon Tan, Jaewoo Kang, et al.
A pilot study of biomedical text comprehension using an
attention-based deep neural reader: Design and experimental analysis.
JMIR medical informatics, 60 (1):0 e8751,

[Zhang et al., 2016]
Liwen Zhang, John Winn, and Ryota Tomioka.
Gaussian attention model and its application to knowledge base
embedding and question answering.
arXiv preprint arXiv:1611.02266, 2016.

[Chen et al., 2017]
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.
Reading wikipedia to answer open-domain questions.
arXiv preprint arXiv:1704.00051, 2017.

[Seo et al., 2016]
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi.
Bidirectional attention flow for machine comprehension.
arXiv preprint arXiv:1611.01603, 2016.

[Devlin et al., 2018]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
arXiv preprint arXiv:1810.04805, 2018.

[Dong et al., 2019]
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao,
Ming Zhou, and Hsiao-Wuen Hon.
Unified language model pre-training for natural language
understanding and generation.
Adv Neur In, 32, 2019.

[Lopez et al., 2021]
Luis Enrico Lopez, Diane Kathryn Cruz, Jan Christian Blaise Cruz, and Charibeth
Simplifying paragraph-level question generation via transformer
language models.
In Pacific Rim International Conference on Artificial
Intelligence, pages 323–334. Springer, 2021.

[Lin et al., 2017]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
In Proceedings of the IEEE international conference on computer
vision, pages 2980–2988, 2017.

[Liu et al., 2019]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.

[Lan et al., 2019]
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and
Radu Soricut.
Albert: A lite bert for self-supervised learning of language
arXiv preprint arXiv:1909.11942, 2019b.

[Sanh et al., 2019]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
Distilbert, a distilled version of bert: smaller, faster, cheaper and
arXiv preprint arXiv:1910.01108, 2019.

[Roberts et al., 2020]
Adam Roberts, Colin Raffel, and Noam Shazeer.
How much knowledge can you pack into the parameters of a language
arXiv preprint arXiv:2002.08910, 2020.

[Joshi et al., 2017]
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.
Triviaqa: A large scale distantly supervised challenge dataset for
reading comprehension.
arXiv preprint arXiv:1705.03551, 2017.

[Chuang et al., 2019]
Yung-Sung Chuang, Chi-Liang Liu, Hung-Yi Lee, and Lin-shan Lee.
Speechbert: An audio-and-text jointly learned language model for
end-to-end spoken question answering.
arXiv preprint arXiv:1910.11559, 2019.

[Fayek and Johnson, 2020]
Haytham M Fayek and Justin Johnson.
Temporal reasoning via audio question answering.
IEEE/ACM Transactions on Audio, Speech, and Language
Processing, 28:0 2283–2294, 2020.

[Ji and Grishman, 2011]
Heng Ji and Ralph Grishman.
Knowledge base population: Successful approaches and challenges.
In Proceedings of the 49th annual meeting of the association
for computational linguistics: Human language technologies, pages
1148–1158, 2011.

[Berant and Liang, 2014]
Jonathan Berant and Percy Liang.
Semantic parsing via paraphrasing.
In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425,

[Berant et al., 2013]
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.
Semantic parsing on freebase from question-answer pairs.
In Proceedings of the 2013 conference on empirical methods in
natural language processing, pages 1533–1544, 2013.

[Yih et al., 2016]
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina
The value of semantic parse labeling for knowledge base question
In Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Papers), pages 201–206,

[Dubey et al., 2019]
Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann.
Lc-quad 2.0: A large dataset for complex question answering over
wikidata and dbpedia.
In International semantic web conference, pages 69–78.
Springer, 2019.

[Zhang et al., 2018]
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song.
Variational reasoning for question answering with knowledge graph.
In Thirty-second AAAI conference on artificial intelligence,

[Bisk et al., 2016]
Yonatan Bisk, Siva Reddy, John Blitzer, Julia Hockenmaier, and Mark Steedman.
Evaluating induced ccg parsers on grounded semantic parsing.
arXiv preprint arXiv:1609.09405, 2016.

[Talmor and Berant, 2018]
Alon Talmor and Jonathan Berant.
The web as a knowledge-base for answering complex questions.
arXiv preprint arXiv:1803.06643, 2018.

[Zhou et al., 2018]
Mantong Zhou, Minlie Huang, and Xiaoyan Zhu.
An interpretable reasoning network for multi-relation question
arXiv preprint arXiv:1801.04726, 2018.

[Bollacker et al., 2008]
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.
Freebase: a collaboratively created graph database for structuring
human knowledge.
In Proceedings of the 2008 ACM SIGMOD international conference
on Management of data, pages 1247–1250, 2008.

[Lehmann et al., 2015]
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas,
Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef,
Sören Auer, et al.
Dbpedia–a large-scale, multilingual knowledge base extracted from
Semantic web, 60 (2):0 167–195, 2015.

[Pellissier Tanon et al., 2016]
Thomas Pellissier Tanon, Denny Vrandečić, Sebastian Schaffert, Thomas
Steiner, and Lydia Pintscher.
From freebase to wikidata: The great migration.
In Proceedings of the 25th international conference on world
wide web, pages 1419–1428, 2016.

[Wilcke et al., 2017]
Xander Wilcke, Peter Bloem, and Victor De Boer.
The knowledge graph as the default data model for learning on
heterogeneous knowledge.
Data Science, 10 (1-2):0 39–57, 2017.

[Vakulenko et al., 2019]
Svitlana Vakulenko, Javier David Fernandez Garcia, Axel Polleres, Maarten
de Rijke, and Michael Cochez.
Message passing for complex question answering over knowledge graphs.
In Proceedings of the 28th acm international conference on
information and knowledge management, pages 1431–1440, 2019.

[Wang et al., 2019]
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua.
Kgat: Knowledge graph attention network for recommendation.
In Proceedings of the 25th ACM SIGKDD international conference
on knowledge discovery & data mining, pages 950–958, 2019.

[Bordes et al., 2015]
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston.
Large-scale simple question answering with memory networks.
arXiv preprint arXiv:1506.02075, 2015.

[Mihalcea and Csomai, 2007]
Rada Mihalcea and Andras Csomai.
Wikify! linking documents to encyclopedic knowledge.
In Proceedings of the sixteenth ACM conference on Conference on
information and knowledge management, pages 233–242, 2007.

[Radford et al., 2019]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya
Sutskever, et al.
Language models are unsupervised multitask learners.
OpenAI blog, 10 (8):0 9, 2019.

[Lukovnikov et al., 2019]
Denis Lukovnikov, Asja Fischer, and Jens Lehmann.
Pretrained transformers for simple question answering over knowledge
In International Semantic Web Conference, pages 470–486.
Springer, 2019.

[Zelle and Mooney, 1996]
John M Zelle and Raymond J Mooney.
Learning to parse database queries using inductive logic programming.
In Proceedings of the national conference on artificial
intelligence, pages 1050–1055, 1996.

[Iyer et al., 2017]
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke
Learning a neural semantic parser from user feedback.
arXiv preprint arXiv:1704.08760, 2017.

[Rajpurkar et al., 2016]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
Squad: 100,000+ questions for machine comprehension of text.
arXiv preprint arXiv:1606.05250, 2016.

[Zhang et al., 2018]
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu.
Multi-scale attentive interaction networks for chinese medical
question answer selection.
IEEE Access, 6:0 74061–74071, 2018b.

[Lee et al., 2019]
Chia-Hsuan Lee, Hung-yi Lee, Szu-Lin Wu, Chi-Liang Liu, Wei Fang, Juei-Yang
Hsu, and Bo-Hsiang Tseng.
Machine comprehension of spoken content: Toefl listening test and
spoken squad.
IEEE/ACM Trans. Audio, Speech, Language Process., 270
(9):0 1469–1480, 2019b.

[Nguyen et al., 2016]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan
Majumder, and Li Deng.
Ms marco: A human generated machine reading comprehension dataset.
In CoCo@ NIPS, 2016.

[Yang et al., 2015]
Yi Yang, Wen-tau Yih, and Christopher Meek.
Wikiqa: A challenge dataset for open-domain question answering.
In Proceedings of the 2015 conference on empirical methods in
natural language processing, pages 2013–2018, 2015.

[Rajpurkar et al., 2018]
Pranav Rajpurkar, Robin Jia, and Percy Liang.
Know what you don't know: Unanswerable questions for squad.
arXiv preprint arXiv:1806.03822, 2018.

[Yang et al., 2018]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan
Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question
arXiv preprint arXiv:1809.09600, 2018.

[Kwiatkowski et al., 2019]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, et al.
Natural questions: a benchmark for question answering research.
Transactions of the Association for Computational Linguistics,
7:0 453–466, 2019.

[Clark et al., 2020]
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski,
Vitaly Nikolaev, and Jennimaria Palomaki.
Tydi qa: A benchmark for information-seeking question answering in
typologically diverse languages.
Transactions of the Association for Computational Linguistics,
8:0 454–470, 2020.

[Moldovan et al., 1999]
Dan I Moldovan, Sanda M Harabagiu, Marius Pasca, Rada Mihalcea, Richard
Goodrum, Roxana Girju, and Vasile Rus.
Lasso: A tool for surfing the answer net.
In TREC, volume 8, pages 65–73, 1999.

[Voorhees et al., 2001]
Ellen M Voorhees et al.
Overview of the trec 2001 question answering track.
In Trec, pages 42–51, 2001.

[Tan et al., 2017]
Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Bowen Du, Weifeng Lv, and Ming
Context-aware answer sentence selection with hierarchical gated
recurrent neural networks.
IEEE/ACM Transactions on Audio, Speech, and Language
Processing, 260 (3):0 540–549, 2017.

[Dai et al., 2016]
Zihang Dai, Lei Li, and Wei Xu.
Cfo: Conditional focused neural question answering with large-scale
knowledge bases.
arXiv preprint arXiv:1606.01994, 2016.

[Trivedi et al., 2017]
Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann.
Lc-quad: A corpus for complex question answering over knowledge
In International Semantic Web Conference, pages 210–218.
Springer, 2017.

[Ferrucci, 2012]
David A Ferrucci.
Introduction to “this is watson”.
IBM Journal of Research and Development, 560
(3.4):0 1–1, 2012.

[Luhn, 1957]
Hans Peter Luhn.
A statistical approach to mechanized encoding and searching of
literary information.
IBM Journal of research and development, 10
(4):0 309–317, 1957.

[Jones, 1972]
Karen Sparck Jones.
A statistical interpretation of term specificity and its application
in retrieval.
Journal of documentation, 1972.

[Kamphuis et al., 2020]
Chris Kamphuis, Arjen P de Vries, Leonid Boytsov, and Jimmy Lin.
Which bm25 do you mean? a large-scale reproducibility study of
scoring variants.
In European Conference on Information Retrieval, pages 28–34.
Springer, 2020.

[Robertson et al., 1995]
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu,
Mike Gatford, et al.
Okapi at trec-3.
Nist Special Publication Sp, 109:0 109, 1995.

[Manning et al., 2010]
Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze.
Introduction to information retrieval.
Nat Lang Eng, 160 (1):0 100–103, 2010.

[Alberti et al., 2019]
Chris Alberti, Kenton Lee, and Michael Collins.
A bert baseline for the natural questions.
arXiv preprint arXiv:1901.08634, 2019.

[Furnas et al., 1987]
George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais.
The vocabulary problem in human-system communication.
Commun. ACM, 300 (11):0 964–971, 1987.

[Deerwester et al., 1990]
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and
Richard Harshman.
Indexing by latent semantic analysis.
Journal of the American society for information science,
410 (6):0 391–407, 1990.

[Mahmoud and Zrigui, 2019]
Adnen Mahmoud and Mounir Zrigui.
Sentence embedding and convolutional neural network for semantic
textual similarity detection in arabic language.
Arabian Journal for Science and Engineering, 440
(11):0 9263–9274, 2019.

[Karpukhin et al., 2020]
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu,
Sergey Edunov, Danqi Chen, and Wen-tau Yih.
Dense passage retrieval for open-domain question answering.
arXiv preprint arXiv:2004.04906, 2020.

[Liu et al., 2016]
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin,
and Joelle Pineau.
How not to evaluate your dialogue system: An empirical study of
unsupervised evaluation metrics for dialogue response generation.
arXiv preprint arXiv:1603.08023, 2016.

[Xu et al., 2015]
Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao.
Question answering via phrasal semantic parsing.
In International conference of the cross-language evaluation
forum for european languages, pages 414–426. Springer, 2015a.

[Sun et al., 2022]
Yawei Sun, Pengwei Li, Gong Cheng, and Yuzhong Qu.
Skeleton parsing for complex question answering over knowledge bases.
J Web Semant, 72:0 100698, 2022.

[Zhang et al., 2022]
Jinhao Zhang, Lizong Zhang, Bei Hui, and Ling Tian.
Improving complex knowledge base question answering via structural
information learning.
Knowledge-Based Systems, page 108252, 2022.

[Dong et al., 2015]
Li Dong, Furu Wei, Ming Zhou, and Ke Xu.
Question answering over freebase with multi-column convolutional
neural networks.
In Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 260–269, 2015.

[Hu et al., 2018]
Sen Hu, Lei Zou, and Xinbo Zhang.
A state-transition framework to answer complex questions over
knowledge base.
In Proceedings of the 2018 conference on empirical methods in
natural language processing, pages 2098–2108, 2018.

[Duan et al., 2017]
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou.
Question generation for question answering.
In Proceedings of the 2017 conference on empirical methods in
natural language processing, pages 866–874, 2017.

[Reddy et al., 2019]
Siva Reddy, Danqi Chen, and Christopher D Manning.
Coqa: A conversational question answering challenge.
Transactions of the Association for Computational Linguistics,
7:0 249–266, 2019.

[Zhou et al., 2017]
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou.
Neural question generation from text: A preliminary study.
In National CCF Conference on Natural Language Processing and
Chinese Computing, pages 662–671. Springer, 2017.

[Pan et al., 2020]
Youcheng Pan, Baotian Hu, Qingcai Chen, Yang Xiang, and Xiaolong Wang.
Learning to generate diverse questions from keywords.
In ICASSP 2020-2020 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 8224–8228. IEEE,

[Gao et al., 2018]
Yifan Gao, Lidong Bing, Wang Chen, Michael R Lyu, and Irwin King.
Difficulty controllable generation of reading comprehension
arXiv preprint arXiv:1807.03586, 2018.

[Ali et al., 2010]
Husam Ali, Yllias Chali, and Sadid A Hasan.
Automatic question generation from sentences.
In Actes de la 17e conférence sur le Traitement Automatique
des Langues Naturelles. Articles courts, pages 213–218, 2010.

[Du et al., 2017]
Xinya Du, Junru Shao, and Claire Cardie.
Learning to ask: Neural question generation for reading
arXiv preprint arXiv:1705.00106, 2017.

[Zhao et al., 2018]
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke.
Paragraph-level neural question generation with maxout pointer and
gated self-attention networks.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 3901–3910, 2018.

[Pan et al., 2020]
Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan.
Semantic graphs for generating deep questions.
arXiv preprint arXiv:2004.12704, 2020b.

[Cho et al., 2019]
Woon Sang Cho, Yizhe Zhang, Sudha Rao, Asli Celikyilmaz, Chenyan Xiong,
Jianfeng Gao, Mengdi Wang, and Bill Dolan.
Contrastive multi-document question generation.
arXiv preprint arXiv:1911.03047, 2019b.

[Song et al., 2018]
Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea.
Leveraging context information for natural question generation.
In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 2 (Short Papers), pages 569–574, 2018.

[Hochreiter, 1998]
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets
and problem solutions.
International Journal of Uncertainty, Fuzziness and
Knowledge-Based Systems, 60 (02):0 107–116, 1998.

[Gers et al., 2000]
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins.
Learning to forget: Continual prediction with lstm.
Neural computation, 120 (10):0 2451–2471,

[Yuan et al., 2017]
Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman,
Sandeep Subramanian, Saizheng Zhang, and Adam Trischler.
Machine comprehension by text-to-text neural question generation.
arXiv preprint arXiv:1705.02012, 2017.

[Zhang and Bansal, 2019]
Shiyue Zhang and Mohit Bansal.
Addressing semantic drift in question generation for semi-supervised
question answering.
arXiv preprint arXiv:1909.06356, 2019.

[Qi et al., 2020]
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
Zhang, and Ming Zhou.
Prophetnet: Predicting future n-gram for sequence-to-sequence
arXiv preprint arXiv:2001.04063, 2020.

[Lewis et al., 2020]
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel.
Question and answer test-train overlap in open-domain question
answering datasets.
arXiv preprint arXiv:2008.02637, 2020.

[Trischler et al., 2016]
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni,
Philip Bachman, and Kaheer Suleman.
Newsqa: A machine comprehension dataset.
arXiv preprint arXiv:1611.09830, 2016.

[Dunn et al., 2017]
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and
Kyunghyun Cho.
Searchqa: A new q&a dataset augmented with context from a search
arXiv preprint arXiv:1704.05179, 2017.

[Bajaj et al., 2016]
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu,
Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al.
Ms marco: A human generated machine reading comprehension dataset.
arXiv preprint arXiv:1611.09268, 2016.

[Kočiskỳ et al., 2018]
Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer,
Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette.
The narrativeqa reading comprehension challenge.
Transactions of the Association for Computational Linguistics,
6:0 317–328, 2018.

[He et al., 2017]
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu,
Yizhong Wang, Hua Wu, Qiaoqiao She, et al.
Dureader: a chinese machine reading comprehension dataset from
real-world applications.
arXiv preprint arXiv:1711.05073, 2017.

[Chen et al., 2018]
Guanliang Chen, Jie Yang, Claudia Hauff, and Geert-Jan Houben.
Learningq: a large-scale dataset for educational question generation.
In Twelfth International AAAI Conference on Web and Social
Media, 2018.

[Choi et al., 2018]
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy
Liang, and Luke Zettlemoyer.
Quac: Question answering in context.
arXiv preprint arXiv:1808.07036, 2018.

[Hill et al., 2015]
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston.
The goldilocks principle: Reading children's books with explicit
memory representations.
arXiv preprint arXiv:1511.02301, 2015.

[Lai et al., 2017]
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.
Race: Large-scale reading comprehension dataset from examinations.
arXiv preprint arXiv:1704.04683, 2017.

[Richardson et al., 2013]
Matthew Richardson, Christopher JC Burges, and Erin Renshaw.
Mctest: A challenge dataset for the open-domain machine comprehension
of text.
In Proceedings of the 2013 conference on empirical methods in
natural language processing, pages 193–203, 2013.

[Van Der Lee et al., 2019]
Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel
Best practices for the human evaluation of automatically generated
In Proceedings of the 12th International Conference on Natural
Language Generation, pages 355–368, 2019.

[Lin and Hovy, 2003]
Chin-Yew Lin and Eduard Hovy.
Automatic evaluation of summaries using n-gram co-occurrence
In Proceedings of the 2003 human language technology conference
of the North American chapter of the association for computational
linguistics, pages 150–157, 2003.

[Likert, 1932]
Rensis Likert.
A technique for the measurement of attitudes.
Archives of psychology, 1932.

[Amidei, 2021]
Jacopo Amidei.
Evaluating the Evaluators: Subjective Bias and Consistency in
Human Evaluation of Natural Language Generation.
Open University (United Kingdom), 2021.

[Belz and Kow, 2010]
Anja Belz and Eric Kow.
Comparing rating scales and preference judgements in language
In Proceedings of the 6th International Natural Language
Generation Conference, 2010.

[Raykar et al., 2010]
Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles
Florin, Luca Bogoni, and Linda Moy.
Learning from crowds.
Journal of machine learning research, 110 (4), 2010.

[Krippendorff, 2011]
Klaus Krippendorff.
Computing krippendorff's alpha-reliability.

[Thomson and Reiter, 2020]
Craig Thomson and Ehud Reiter.
A gold standard methodology for evaluating accuracy in data-to-text
arXiv preprint arXiv:2011.03992, 2020.

[Lowe et al., 2017]
Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier,
Yoshua Bengio, and Joelle Pineau.
Towards an automatic turing test: Learning to evaluate dialogue
arXiv preprint arXiv:1708.07149, 2017.

[Reiter and Belz, 2009]
Ehud Reiter and Anja Belz.
An investigation into the validity of some metrics for automatically
evaluating natural language generation systems.
Computational Linguistics, 350 (4):0 529–558,

[Popović, 2015]
Maja Popović.
chrf: character n-gram f-score for automatic mt evaluation.
In Proceedings of the Tenth Workshop on Statistical Machine
Translation, pages 392–395, 2015.

[Levenshtein et al., 1966]
Vladimir I Levenshtein et al.
Binary codes capable of correcting deletions, insertions, and
In Soviet physics doklady, volume 10, pages 707–710. Soviet
Union, 1966.

[Roemmele and Gordon, 2015]
Melissa Roemmele and Andrew S Gordon.
Creative help: A story writing assistant.
In International Conference on Interactive Digital
Storytelling, pages 81–92. Springer, 2015.

[Papineni et al., 2002]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In Proceedings of the 40th annual meeting of the Association
for Computational Linguistics, pages 311–318, 2002.

[Banerjee and Lavie, 2005]
Satanjeev Banerjee and Alon Lavie.
Meteor: An automatic metric for mt evaluation with improved
correlation with human judgments.
In Proceedings of the acl workshop on intrinsic and extrinsic
evaluation measures for machine translation and/or summarization, pages
65–72, 2005.

[Lin, 2004]
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
In Text summarization branches out, pages 74–81, 2004.

[Doddington, 2002]
George Doddington.
Automatic evaluation of machine translation quality using n-gram
co-occurrence statistics.
In Proceedings of the second international conference on Human
Language Technology Research, pages 138–145, 2002.

[Vedantam et al., 2015]
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.
Cider: Consensus-based image description evaluation.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 4566–4575, 2015.

[Mikolov et al., 2013]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their
Advances in neural information processing systems, 26, 2013.

[Lo, 2017]
Chi-kiu Lo.
Meant 2.0: Accurate semantic mt evaluation for any output language.
In Proceedings of the second conference on machine
translation, pages 589–597, 2017.

[Clark et al., 2019]
Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith.
Sentence mover’s similarity: Automatic evaluation for
multi-sentence texts.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 2748–2760, 2019.

[Zhang et al., 2019]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
arXiv preprint arXiv:1904.09675, 2019c.

[Stanojević and Sima’an, 2014]
Miloš Stanojević and Khalil Sima’an.
Beer: Better evaluation as ranking.
In Proceedings of the Ninth Workshop on Statistical Machine
Translation, pages 414–419, 2014.

[Ma et al., 2017]
Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu.
Blend: a novel combined mt metric based on direct
assessment—casict-dcu submission to wmt17 metrics task.
In Proceedings of the second conference on machine
translation, pages 598–603, 2017.

[Sharif et al., 2018]
Naeha Sharif, Mohammed Bennamoun, Lyndon Rhys White, and Syed Afaq Ali Shah.
Learning-based composite metrics for improved caption evaluation.

[Tao et al., 2018]
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan.
Ruber: An unsupervised method for automatic evaluation of open-domain
dialog systems.
In Thirty-Second AAAI Conference on Artificial Intelligence,

[Sinha et al., 2020]
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L
Hamilton, and Joelle Pineau.
Learning an unreferenced metric for online dialogue evaluation.
arXiv preprint arXiv:2005.00583, 2020.

[Shimanaka et al., 2018]
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi.
Ruse: Regressor using sentence embeddings for automatic machine
translation evaluation.
In Proceedings of the Third Conference on Machine Translation:
Shared Task Papers, pages 751–758, 2018.

[Sellam et al., 2020]
Thibault Sellam, Dipanjan Das, and Ankur P Parikh.
Bleurt: Learning robust metrics for text generation.
arXiv preprint arXiv:2004.04696, 2020.

[Serban et al., 2017]
Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau,
Aaron Courville, and Yoshua Bengio.
A hierarchical latent variable encoder-decoder model for generating
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 31, 2017.

[Gutmann and Hyvärinen, 2010]
Michael Gutmann and Aapo Hyvärinen.
Noise-contrastive estimation: A new estimation principle for
unnormalized statistical models.
In Proceedings of the thirteenth international conference on
artificial intelligence and statistics, pages 297–304. JMLR Workshop and
Conference Proceedings, 2010.

[Conneau et al., 2017]
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes.
Supervised learning of universal sentence representations from
natural language inference data.
arXiv preprint arXiv:1705.02364, 2017.

[Logeswaran and Lee, 2018]
Lajanugen Logeswaran and Honglak Lee.
An efficient framework for learning sentence representations.
arXiv preprint arXiv:1803.02893, 2018.

[Cer et al., 2018]
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St
John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al.
Universal sentence encoder.
arXiv preprint arXiv:1803.11175, 2018.

[Vinyals et al., 2015]
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.
Show and tell: A neural image caption generator.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 3156–3164, 2015.

[Xu et al., 2015]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan
Salakhudinov, Rich Zemel, and Yoshua Bengio.
Show, attend and tell: Neural image caption generation with visual
In International conference on machine learning, pages
2048–2057. PMLR, 2015b.

[Lu et al., 2017]
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.
Knowing when to look: Adaptive attention via a visual sentinel for
image captioning.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 375–383, 2017.

[Callison-Burch et al., 2006]
Chris Callison-Burch, Miles Osborne, and Philipp Koehn.
Re-evaluating the role of bleu in machine translation research.
In 11th conference of the european chapter of the association
for computational linguistics, pages 249–256, 2006.

[Galley et al., 2015]
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli,
Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan.
deltableu: A discriminative metric for generation tasks with
intrinsically diverse targets.
arXiv preprint arXiv:1506.06863, 2015.

[Chen et al., 2019]
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
Evaluating question answering evaluation.
In Proceedings of the 2nd Workshop on Machine Reading for
Question Answering, pages 119–124, 2019.

[Ding et al., 2018]
Jie Ding, Vahid Tarokh, and Yuhong Yang.
Model selection techniques: An overview.
IEEE Signal Processing Magazine, 350 (6):0
16–34, 2018.

[Emmert-Streib and Dehmer, 2019]
Frank Emmert-Streib and Matthias Dehmer.
Evaluation of regression models: Model assessment, model selection
and generalization error.
Machine learning and knowledge extraction, 10
(1):0 521–551, 2019.

[Bates, 1995]
Madeleine Bates.
Models of natural language understanding.
Proceedings of the National Academy of Sciences, 920
(22):0 9977–9982, 1995.

[Liu et al., 2021]
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser.
Benchmarking natural language understanding services for building
conversational agents.
In Increasing Naturalness and Flexibility in Spoken Dialogue
Interaction, pages 165–183. Springer, 2021.

[Radford et al., 2018]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.
Improving language understanding by generative pre-training.

[Reiter and Dale, 1997]
Ehud Reiter and Robert Dale.
Building applied natural language generation systems.
Natural Language Engineering, 30 (1):0 57–87,

[Gatt and Krahmer, 2018]
Albert Gatt and Emiel Krahmer.
Survey of the state of the art in natural language generation: Core
tasks, applications and evaluation.
Journal of Artificial Intelligence Research, 61:0
65–170, 2018.

[Cho et al., 2014]
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau,
Fethi Bougares, Holger Schwenk, and Yoshua Bengio.
Learning phrase representations using rnn encoder-decoder for
statistical machine translation.
arXiv preprint arXiv:1406.1078, 2014.

[Sutskever et al., 2014]
Ilya Sutskever, Oriol Vinyals, and Quoc V Le.
Sequence to sequence learning with neural networks.
Advances in neural information processing systems, 27, 2014.

[Bahdanau et al., 2014]
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.
Neural machine translation by jointly learning to align and
arXiv preprint arXiv:1409.0473, 2014.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[Peñas et al., 2012]
Anselmo Peñas, Bernardo Magnini, Pamela Forner, Richard Sutcliffe, Alvaro
Rodrigo, and Danilo Giampiccolo.
Question answering at the cross-language evaluation forum 2003–2010.
Language resources and evaluation, 460 (2):0
177–217, 2012.

[Androutsopoulos et al., 1995]
Ion Androutsopoulos, Graeme D Ritchie, and Peter Thanisch.
Natural language interfaces to databases–an introduction.
Natural language engineering, 10 (1):0 29–81,

[Kolomiyets and Moens, 2011]
Oleksandr Kolomiyets and Marie-Francine Moens.
A survey on question answering technology from an information
retrieval perspective.
Inform Sciences, 1810 (24):0 5412–5434, 2011.

[Green Jr et al., 1961]
Bert F Green Jr, Alice K Wolf, Carol Chomsky, and Kenneth Laughery.
Baseball: an automatic question-answerer.
In Papers presented at the May 9-11, 1961, western joint
IRE-AIEE-ACM computer conference, pages 219–224, 1961.

[Woods, 1973]
William A Woods.
Progress in natural language understanding: an application to lunar
In Proceedings of the June 4-8, 1973, national computer
conference and exposition, pages 441–450, 1973.

[Voorhees et al., 1999]
Ellen M Voorhees et al.
The trec-8 question answering track report.
In Trec, volume 99, pages 77–82, 1999.

[Yih et al., 2015]
Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao.
Semantic parsing via staged query graph generation: Question
answering with knowledge base.
In Proceedings of the Joint Conference of the 53rd Annual
Meeting of the ACL and the 7th International Joint Conference on Natural
Language Processing of the AFNLP, 2015.

[Indurthi et al., 2017]
Sathish Reddy Indurthi, Dinesh Raghu, Mitesh M Khapra, and Sachindra Joshi.
Generating natural language question-answer pairs from a knowledge
graph using a rnn based question generation model.
In Proceedings of the 15th Conference of the European Chapter
of the Association for Computational Linguistics: Volume 1, Long Papers,
pages 376–385, 2017.

[Pan et al., 2019]
Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan.
Recent advances in neural question generation.
arXiv preprint arXiv:1905.08949, 2019.

[Zhang et al., 2021]
Ruqing Zhang, Jiafeng Guo, Lu Chen, Yixing Fan, and Xueqi Cheng.
A review on question generation from natural language text.
ACM Transactions on Information Systems (TOIS), 400
(1):0 1–43, 2021.

[Sukhbaatar et al., 2015]
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.
End-to-end memory networks.
Advances in neural information processing systems, 28, 2015.

[Yue et al., 2017]
Chunyi Yue, Hanqiang Cao, Kun Xiong, Anqi Cui, Haocheng Qin, and Ming Li.
Enhanced question understanding with dynamic memory networks for
textual question answering.
Expert Systems with Applications, 80:0 39–45, 2017.

[Kumar et al., 2016]
Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan
Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher.
Ask me anything: Dynamic memory networks for natural language
In International conference on machine learning, pages
1378–1387. PMLR, 2016.

[Ojokoh and Adebisi, 2018]
Bolanle Ojokoh and Emmanuel Adebisi.
A review of question answering systems.
J Web Eng, 170 (8):0 717–758, 2018.

[Luo et al., 2018]
Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu.
Knowledge base question answering via encoding of complex query
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 2185–2194, 2018.

[Zhu et al., 2020]
Shuguang Zhu, Xiang Cheng, and Sen Su.
Knowledge-based question answering by tree-to-sequence learning.
Neurocomputing, 372:0 64–72, 2020.

[Qu et al., 2018]
Yingqi Qu, Jie Liu, Liangyi Kang, Qinfeng Shi, and Dan Ye.
Question answering over freebase via attentive rnn with similarity
matrix based cnn.
arXiv preprint arXiv:1804.03317, 38, 2018.

[Lan and Jiang, 2020]
Yunshi Lan and Jing Jiang.
Query graph generation for answering multi-hop complex questions from
knowledge bases.
Association for Computational Linguistics, 2020.

[Dimitrakis et al., 2020]
Eleftherios Dimitrakis, Konstantinos Sgontzos, and Yannis Tzitzikas.
A survey on question answering systems over linked data and
Journal of intelligent information systems, 550
(2):0 233–259, 2020.

[Cortes et al., 2020]
Eduardo Cortes, Vinicius Woloszyn, Arne Binder, Tilo Himmelsbach, Dante Barone,
and Sebastian Möller.
An empirical comparison of question classification methods for
question answering systems.
In Proceedings of the 12th Language Resources and Evaluation
Conference, pages 5408–5416, 2020.

[Zhang et al., 2019]
Siheng Zhang, Wensheng Zhang, and Jinghao Niu.
Improving short-text representation in convolutional networks by
dependency parsing.
Knowledge and Information Systems, 610 (1):0
463–484, 2019a.

[Ben Abacha and Demner-Fushman, 2019]
Asma Ben Abacha and Dina Demner-Fushman.
A question-entailment approach to question answering.
BMC bioinformatics, 200 (1):0 1–23, 2019.

[Hu et al., 2017]
Sen Hu, Lei Zou, Jeffrey Xu Yu, Haixun Wang, and Dongyan Zhao.
Answering natural language questions by subgraph matching over
knowledge graphs.
IEEE Trans. Knowl. Data Eng., 300 (5):0
824–837, 2017.

[Lan et al., 2019]
Yunshi Lan, Shuohang Wang, and Jing Jiang.
Knowledge base question answering with a matching-aggregation model
and question-specific contextual relations.
IEEE/ACM Trans. Audio, Speech, Language Process., 270
(10):0 1629–1638, 2019a.

[He et al., 2019]
Junqing He, Mingming Fu, and Manshu Tu.
Applying deep matching networks to chinese medical question
answering: a study and a dataset.
BMC medical informatics and decision making, 190
(2):0 91–100, 2019.

[Zhang et al., 2019]
Xu Zhang, Wenpeng Lu, Fangfang Li, Xueping Peng, and Ruoyu Zhang.
Deep feature fusion model for sentence semantic matching.
Computers, Materials and Continua, 2019b.

[Abbasiantaeb and Momtazi, 2021]
Zahra Abbasiantaeb and Saeedeh Momtazi.
Text-based question answering from information retrieval and deep
neural network perspectives: A survey.
Wiley Interdisciplinary Reviews: Data Mining and Knowledge
Discovery, 110 (6):0 e1412, 2021.

[Lan et al., 2021]
Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong
A survey on complex knowledge base question answering: Methods,
challenges and solutions.
arXiv preprint arXiv:2105.11644, 2021.

[Hao et al., 2022]
Tianyong Hao, Xinxin Li, Yulan He, Fu Lee Wang, and Yingying Qu.
Recent progress in leveraging deep learning methods for question
Neural Computing and Applications, pages 1–19, 2022.

[Lee et al., 2019]
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.
Latent retrieval for weakly supervised open domain question
arXiv preprint arXiv:1906.00300, 2019a.

[Yang et al., 2017]
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W Cohen.
Semi-supervised qa with generative domain-adaptive nets.
arXiv preprint arXiv:1702.02206, 2017.

[Fabbri et al., 2020]
Alexander R Fabbri, Patrick Ng, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang.
Template-based question generation from retrieved sentences for
improved unsupervised question answering.
arXiv preprint arXiv:2004.11892, 2020.

[Bashath et al., 2022]
Samar Bashath, Nadeesha Perera, Shailesh Tripathi, Kalifa Manjang, Matthias
Dehmer, and Frank Emmert Streib.
A data-centric review of deep transfer learning with applications to
text data.
Information Sciences, 585:0 498–528, 2022.

[Alwaneen et al., 2021]
Tahani H Alwaneen, Aqil M Azmi, Hatim A Aboalsamh, Erik Cambria, and Amir
Arabic question answering system: a survey.
Artif Intell Rev, pages 1–47, 2021.

[Mishra and Jain, 2016]
Amit Mishra and Sanjay Kumar Jain.
A survey on question answering systems with classification.
Journal of King Saud University-Computer and Information
Sciences, 280 (3):0 345–361, 2016.

[Diefenbach et al., 2018]
Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret.
Core techniques of question answering systems over knowledge bases: a
Knowledge and Information systems, 550 (3):0
529–569, 2018.

[Ni et al., 2021]
Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga, and Erik Cambria.
Recent advances in deep learning based dialogue systems: A systematic
arXiv preprint arXiv:2105.04387, 2021.

[Wu et al., 2017]
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van
Den Hengel.
Visual question answering: A survey of methods and datasets.
Computer Vision and Image Understanding, 163:0 21–40,

[Srivastava et al., 2020]
Yash Srivastava, Vaishnav Murali, Shiv Ram Dubey, and Snehasis Mukherjee.
Visual question answering using deep learning: A survey and
performance analysis.
In International Conference on Computer Vision and Image
Processing, pages 75–86. Springer, 2020.

[Deriu et al., 2021]
Jan Deriu, Alvaro Rodrigo, Arantxa Otegi, Guillermo Echegoyen, Sophie Rosset,
Eneko Agirre, and Mark Cieliebak.
Survey on evaluation methods for dialogue systems.
Artificial Intelligence Review, 540 (1):0
755–810, 2021.

[Emmert-Streib et al., 2019]
Frank Emmert-Streib, Salisou Moutari, and Matthias Dehmer.
A comprehensive survey of error measures for evaluating binary
decision making in data science.
Wiley Interdisciplinary Reviews: Data Mining and Knowledge
Discovery, 90 (5):0 e1303, 2019.

[Sai et al., 2022]
Ananya B Sai, Akash Kumar Mohankumar, and Mitesh M Khapra.
A survey of evaluation metrics used for nlg systems.
ACM Computing Surveys (CSUR), 550 (2):0 1–39,

[Celikyilmaz et al., 2020]
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao.
Evaluation of text generation: A survey.
arXiv preprint arXiv:2006.14799, 2020.

[Wang et al., 2007]
Mengqiu Wang, Noah A Smith, and Teruko Mitamura.
What is the jeopardy model? a quasi-synchronous grammar for qa.
In Proceedings of the 2007 joint conference on empirical
methods in natural language processing and computational natural language
learning (EMNLP-CoNLL), pages 22–32, 2007.

[Sammut and Banerji, 1986]
Claude Sammut and Ranan B Banerji.
Learning concepts by asking questions.
Machine learning: An artificial intelligence approach,
2:0 167–192, 1986.

[Liu et al., 2017]
Tianyu Liu, Bingzhen Wei, Baobao Chang, and Zhifang Sui.
Large-scale simple question generation by template-based seq2seq
In National CCF Conference on Natural Language Processing and
Chinese Computing, pages 75–87. Springer, 2017.

[Straach and Truemper, 1999]
Janell Straach and Klaus Truemper.
Learning to ask relevant questions.
Artificial Intelligence, 1110 (1-2):0
301–327, 1999.

[Harabagiu et al., 2005]
Sanda Harabagiu, Andrew Hickl, John Lehmann, and Dan Moldovan.
Experiments with interactive question-answering.
In Proceedings of the 43rd annual meeting of the association
for computational linguistics (ACL’05), pages 205–214, 2005.

[Heilman and Smith, 2010]
Michael Heilman and Noah A Smith.
Good question! statistical ranking for question generation.
In Human Language Technologies: The 2010 Annual Conference of
the North American Chapter of the Association for Computational Linguistics,
pages 609–617, 2010.

[Levy and Andrew, 2006]
Roger Levy and Galen Andrew.
Tregex and tsurgeon: Tools for querying and manipulating tree data
In LREC, pages 2231–2234. Citeseer, 2006.

[Dhole and Manning, 2020]
Kaustubh D Dhole and Christopher D Manning.
Syn-qg: Syntactic and shallow semantic rules for question generation.
arXiv preprint arXiv:2004.08694, 2020.

[Yao and Zhang, 2010]
Xuchen Yao and Yi Zhang.
Question generation with minimal recursion semantics.
In Proceedings of QG2010: The Third Workshop on Question
Generation, pages 68–75. Citeseer, 2010.

[Ko et al., 2007]
Jeongwoo Ko, Luo Si, and Eric Nyberg.
A probabilistic framework for answer selection in question answering.
In Human Language Technologies 2007: The Conference of the
North American Chapter of the Association for Computational Linguistics;
Proceedings of the Main Conference, pages 524–531, 2007.

[Zhou et al., 2016]
Xiangyang Zhou, Daxiang Dong, Hua Wu, Shiqi Zhao, Dianhai Yu, Hao Tian, Xuan
Liu, and Rui Yan.
Multi-view response selection for human-computer conversation.
In Proceedings of the 2016 conference on empirical methods in
natural language processing, pages 372–381, 2016.

[Wu et al., 2016]
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li.
Sequential matching network: A new architecture for multi-turn
response selection in retrieval-based chatbots.
arXiv preprint arXiv:1612.01627, 2016.

[Jing et al., 2019]
Li Jing, Caglar Gulcehre, John Peurifoy, Yichen Shen, Max Tegmark, Marin
Soljacic, and Yoshua Bengio.
Gated orthogonal recurrent units: On learning to forget.
Neural computation, 310 (4):0 765–783, 2019.

[Cho et al., 2019]
Woon Sang Cho, Yizhe Zhang, Sudha Rao, Chris Brockett, and Sungjin Lee.
Generating a common question from multiple documents using
multi-source encoder-decoder models.
arXiv preprint arXiv:1910.11483, 2019a.

[Huang et al., 2018]
Heyan Huang, Xiaochi Wei, Liqiang Nie, Xianling Mao, and Xin-Shun Xu.
From question to text: Question-oriented feature attention for answer
ACM Transactions on Information Systems (TOIS), 370
(1):0 1–33, 2018.

[Kim et al., 2018]
Seongsoon Kim, Donghyeon Park, Yonghwa Choi, Kyubum Lee, Byounggun Kim, Minji
Jeon, Jihye Kim, Aik Choon Tan, Jaewoo Kang, et al.
A pilot study of biomedical text comprehension using an
attention-based deep neural reader: Design and experimental analysis.
JMIR medical informatics, 60 (1):0 e8751,

[Zhang et al., 2016]
Liwen Zhang, John Winn, and Ryota Tomioka.
Gaussian attention model and its application to knowledge base
embedding and question answering.
arXiv preprint arXiv:1611.02266, 2016.

[Chen et al., 2017]
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.
Reading wikipedia to answer open-domain questions.
arXiv preprint arXiv:1704.00051, 2017.

[Seo et al., 2016]
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi.
Bidirectional attention flow for machine comprehension.
arXiv preprint arXiv:1611.01603, 2016.

[Devlin et al., 2018]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
arXiv preprint arXiv:1810.04805, 2018.

[Dong et al., 2019]
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao,
Ming Zhou, and Hsiao-Wuen Hon.
Unified language model pre-training for natural language
understanding and generation.
Adv Neur In, 32, 2019.

[Lopez et al., 2021]
Luis Enrico Lopez, Diane Kathryn Cruz, Jan Christian Blaise Cruz, and Charibeth
Simplifying paragraph-level question generation via transformer
language models.
In Pacific Rim International Conference on Artificial
Intelligence, pages 323–334. Springer, 2021.

[Lin et al., 2017]
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.
Focal loss for dense object detection.
In Proceedings of the IEEE international conference on computer
vision, pages 2980–2988, 2017.

[Liu et al., 2019]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.

[Lan et al., 2019]
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and
Radu Soricut.
Albert: A lite bert for self-supervised learning of language
arXiv preprint arXiv:1909.11942, 2019b.

[Sanh et al., 2019]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.
Distilbert, a distilled version of bert: smaller, faster, cheaper and
arXiv preprint arXiv:1910.01108, 2019.

[Roberts et al., 2020]
Adam Roberts, Colin Raffel, and Noam Shazeer.
How much knowledge can you pack into the parameters of a language
arXiv preprint arXiv:2002.08910, 2020.

[Joshi et al., 2017]
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer.
Triviaqa: A large scale distantly supervised challenge dataset for
reading comprehension.
arXiv preprint arXiv:1705.03551, 2017.

[Chuang et al., 2019]
Yung-Sung Chuang, Chi-Liang Liu, Hung-Yi Lee, and Lin-shan Lee.
Speechbert: An audio-and-text jointly learned language model for
end-to-end spoken question answering.
arXiv preprint arXiv:1910.11559, 2019.

[Fayek and Johnson, 2020]
Haytham M Fayek and Justin Johnson.
Temporal reasoning via audio question answering.
IEEE/ACM Transactions on Audio, Speech, and Language
Processing, 28:0 2283–2294, 2020.

[Ji and Grishman, 2011]
Heng Ji and Ralph Grishman.
Knowledge base population: Successful approaches and challenges.
In Proceedings of the 49th annual meeting of the association
for computational linguistics: Human language technologies, pages
1148–1158, 2011.

[Berant and Liang, 2014]
Jonathan Berant and Percy Liang.
Semantic parsing via paraphrasing.
In Proceedings of the 52nd Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425,

[Berant et al., 2013]
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang.
Semantic parsing on freebase from question-answer pairs.
In Proceedings of the 2013 conference on empirical methods in
natural language processing, pages 1533–1544, 2013.

[Yih et al., 2016]
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina
The value of semantic parse labeling for knowledge base question
In Proceedings of the 54th Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Papers), pages 201–206,

[Dubey et al., 2019]
Mohnish Dubey, Debayan Banerjee, Abdelrahman Abdelkawi, and Jens Lehmann.
Lc-quad 2.0: A large dataset for complex question answering over
wikidata and dbpedia.
In International semantic web conference, pages 69–78.
Springer, 2019.

[Zhang et al., 2018]
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song.
Variational reasoning for question answering with knowledge graph.
In Thirty-second AAAI conference on artificial intelligence,

[Bisk et al., 2016]
Yonatan Bisk, Siva Reddy, John Blitzer, Julia Hockenmaier, and Mark Steedman.
Evaluating induced ccg parsers on grounded semantic parsing.
arXiv preprint arXiv:1609.09405, 2016.

[Talmor and Berant, 2018]
Alon Talmor and Jonathan Berant.
The web as a knowledge-base for answering complex questions.
arXiv preprint arXiv:1803.06643, 2018.

[Zhou et al., 2018]
Mantong Zhou, Minlie Huang, and Xiaoyan Zhu.
An interpretable reasoning network for multi-relation question
arXiv preprint arXiv:1801.04726, 2018.

[Bollacker et al., 2008]
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.
Freebase: a collaboratively created graph database for structuring
human knowledge.
In Proceedings of the 2008 ACM SIGMOD international conference
on Management of data, pages 1247–1250, 2008.

[Lehmann et al., 2015]
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas,
Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef,
Sören Auer, et al.
Dbpedia–a large-scale, multilingual knowledge base extracted from
Semantic web, 60 (2):0 167–195, 2015.

[Pellissier Tanon et al., 2016]
Thomas Pellissier Tanon, Denny Vrandečić, Sebastian Schaffert, Thomas
Steiner, and Lydia Pintscher.
From freebase to wikidata: The great migration.
In Proceedings of the 25th international conference on world
wide web, pages 1419–1428, 2016.

[Wilcke et al., 2017]
Xander Wilcke, Peter Bloem, and Victor De Boer.
The knowledge graph as the default data model for learning on
heterogeneous knowledge.
Data Science, 10 (1-2):0 39–57, 2017.

[Vakulenko et al., 2019]
Svitlana Vakulenko, Javier David Fernandez Garcia, Axel Polleres, Maarten
de Rijke, and Michael Cochez.
Message passing for complex question answering over knowledge graphs.
In Proceedings of the 28th acm international conference on
information and knowledge management, pages 1431–1440, 2019.

[Wang et al., 2019]
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua.
Kgat: Knowledge graph attention network for recommendation.
In Proceedings of the 25th ACM SIGKDD international conference
on knowledge discovery & data mining, pages 950–958, 2019.

[Bordes et al., 2015]
Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston.
Large-scale simple question answering with memory networks.
arXiv preprint arXiv:1506.02075, 2015.

[Mihalcea and Csomai, 2007]
Rada Mihalcea and Andras Csomai.
Wikify! linking documents to encyclopedic knowledge.
In Proceedings of the sixteenth ACM conference on Conference on
information and knowledge management, pages 233–242, 2007.

[Radford et al., 2019]
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya
Sutskever, et al.
Language models are unsupervised multitask learners.
OpenAI blog, 10 (8):0 9, 2019.

[Lukovnikov et al., 2019]
Denis Lukovnikov, Asja Fischer, and Jens Lehmann.
Pretrained transformers for simple question answering over knowledge
In International Semantic Web Conference, pages 470–486.
Springer, 2019.

[Zelle and Mooney, 1996]
John M Zelle and Raymond J Mooney.
Learning to parse database queries using inductive logic programming.
In Proceedings of the national conference on artificial
intelligence, pages 1050–1055, 1996.

[Iyer et al., 2017]
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke
Learning a neural semantic parser from user feedback.
arXiv preprint arXiv:1704.08760, 2017.

[Rajpurkar et al., 2016]
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
Squad: 100,000+ questions for machine comprehension of text.
arXiv preprint arXiv:1606.05250, 2016.

[Zhang et al., 2018]
Sheng Zhang, Xin Zhang, Hui Wang, Lixiang Guo, and Shanshan Liu.
Multi-scale attentive interaction networks for chinese medical
question answer selection.
IEEE Access, 6:0 74061–74071, 2018b.

[Lee et al., 2019]
Chia-Hsuan Lee, Hung-yi Lee, Szu-Lin Wu, Chi-Liang Liu, Wei Fang, Juei-Yang
Hsu, and Bo-Hsiang Tseng.
Machine comprehension of spoken content: Toefl listening test and
spoken squad.
IEEE/ACM Trans. Audio, Speech, Language Process., 270
(9):0 1469–1480, 2019b.

[Nguyen et al., 2016]
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan
Majumder, and Li Deng.
Ms marco: A human generated machine reading comprehension dataset.
In CoCo@ NIPS, 2016.

[Yang et al., 2015]
Yi Yang, Wen-tau Yih, and Christopher Meek.
Wikiqa: A challenge dataset for open-domain question answering.
In Proceedings of the 2015 conference on empirical methods in
natural language processing, pages 2013–2018, 2015.

[Rajpurkar et al., 2018]
Pranav Rajpurkar, Robin Jia, and Percy Liang.
Know what you don't know: Unanswerable questions for squad.
arXiv preprint arXiv:1806.03822, 2018.

[Yang et al., 2018]
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan
Salakhutdinov, and Christopher D Manning.
Hotpotqa: A dataset for diverse, explainable multi-hop question
arXiv preprint arXiv:1809.09600, 2018.

[Kwiatkowski et al., 2019]
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur
Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin,
Kenton Lee, et al.
Natural questions: a benchmark for question answering research.
Transactions of the Association for Computational Linguistics,
7:0 453–466, 2019.

[Clark et al., 2020]
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski,
Vitaly Nikolaev, and Jennimaria Palomaki.
Tydi qa: A benchmark for information-seeking question answering in
typologically diverse languages.
Transactions of the Association for Computational Linguistics,
8:0 454–470, 2020.

[Moldovan et al., 1999]
Dan I Moldovan, Sanda M Harabagiu, Marius Pasca, Rada Mihalcea, Richard
Goodrum, Roxana Girju, and Vasile Rus.
Lasso: A tool for surfing the answer net.
In TREC, volume 8, pages 65–73, 1999.

[Voorhees et al., 2001]
Ellen M Voorhees et al.
Overview of the trec 2001 question answering track.
In Trec, pages 42–51, 2001.

[Tan et al., 2017]
Chuanqi Tan, Furu Wei, Qingyu Zhou, Nan Yang, Bowen Du, Weifeng Lv, and Ming
Context-aware answer sentence selection with hierarchical gated
recurrent neural networks.
IEEE/ACM Transactions on Audio, Speech, and Language
Processing, 260 (3):0 540–549, 2017.

[Dai et al., 2016]
Zihang Dai, Lei Li, and Wei Xu.
Cfo: Conditional focused neural question answering with large-scale
knowledge bases.
arXiv preprint arXiv:1606.01994, 2016.

[Trivedi et al., 2017]
Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann.
Lc-quad: A corpus for complex question answering over knowledge
In International Semantic Web Conference, pages 210–218.
Springer, 2017.

[Ferrucci, 2012]
David A Ferrucci.
Introduction to “this is watson”.
IBM Journal of Research and Development, 560
(3.4):0 1–1, 2012.

[Luhn, 1957]
Hans Peter Luhn.
A statistical approach to mechanized encoding and searching of
literary information.
IBM Journal of research and development, 10
(4):0 309–317, 1957.

[Jones, 1972]
Karen Sparck Jones.
A statistical interpretation of term specificity and its application
in retrieval.
Journal of documentation, 1972.

[Kamphuis et al., 2020]
Chris Kamphuis, Arjen P de Vries, Leonid Boytsov, and Jimmy Lin.
Which bm25 do you mean? a large-scale reproducibility study of
scoring variants.
In European Conference on Information Retrieval, pages 28–34.
Springer, 2020.

[Robertson et al., 1995]
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu,
Mike Gatford, et al.
Okapi at trec-3.
Nist Special Publication Sp, 109:0 109, 1995.

[Manning et al., 2010]
Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze.
Introduction to information retrieval.
Nat Lang Eng, 160 (1):0 100–103, 2010.

[Alberti et al., 2019]
Chris Alberti, Kenton Lee, and Michael Collins.
A bert baseline for the natural questions.
arXiv preprint arXiv:1901.08634, 2019.

[Furnas et al., 1987]
George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais.
The vocabulary problem in human-system communication.
Commun. ACM, 300 (11):0 964–971, 1987.

[Deerwester et al., 1990]
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and
Richard Harshman.
Indexing by latent semantic analysis.
Journal of the American society for information science,
410 (6):0 391–407, 1990.

[Mahmoud and Zrigui, 2019]
Adnen Mahmoud and Mounir Zrigui.
Sentence embedding and convolutional neural network for semantic
textual similarity detection in arabic language.
Arabian Journal for Science and Engineering, 440
(11):0 9263–9274, 2019.

[Karpukhin et al., 2020]
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu,
Sergey Edunov, Danqi Chen, and Wen-tau Yih.
Dense passage retrieval for open-domain question answering.
arXiv preprint arXiv:2004.04906, 2020.

[Liu et al., 2016]
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin,
and Joelle Pineau.
How not to evaluate your dialogue system: An empirical study of
unsupervised evaluation metrics for dialogue response generation.
arXiv preprint arXiv:1603.08023, 2016.

[Xu et al., 2015]
Kun Xu, Yansong Feng, Songfang Huang, and Dongyan Zhao.
Question answering via phrasal semantic parsing.
In International conference of the cross-language evaluation
forum for european languages, pages 414–426. Springer, 2015a.

[Sun et al., 2022]
Yawei Sun, Pengwei Li, Gong Cheng, and Yuzhong Qu.
Skeleton parsing for complex question answering over knowledge bases.
J Web Semant, 72:0 100698, 2022.

[Zhang et al., 2022]
Jinhao Zhang, Lizong Zhang, Bei Hui, and Ling Tian.
Improving complex knowledge base question answering via structural
information learning.
Knowledge-Based Systems, page 108252, 2022.

[Dong et al., 2015]
Li Dong, Furu Wei, Ming Zhou, and Ke Xu.
Question answering over freebase with multi-column convolutional
neural networks.
In Proceedings of the 53rd Annual Meeting of the Association
for Computational Linguistics and the 7th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 260–269, 2015.

[Hu et al., 2018]
Sen Hu, Lei Zou, and Xinbo Zhang.
A state-transition framework to answer complex questions over
knowledge base.
In Proceedings of the 2018 conference on empirical methods in
natural language processing, pages 2098–2108, 2018.

[Duan et al., 2017]
Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou.
Question generation for question answering.
In Proceedings of the 2017 conference on empirical methods in
natural language processing, pages 866–874, 2017.

[Reddy et al., 2019]
Siva Reddy, Danqi Chen, and Christopher D Manning.
Coqa: A conversational question answering challenge.
Transactions of the Association for Computational Linguistics,
7:0 249–266, 2019.

[Zhou et al., 2017]
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou.
Neural question generation from text: A preliminary study.
In National CCF Conference on Natural Language Processing and
Chinese Computing, pages 662–671. Springer, 2017.

[Pan et al., 2020]
Youcheng Pan, Baotian Hu, Qingcai Chen, Yang Xiang, and Xiaolong Wang.
Learning to generate diverse questions from keywords.
In ICASSP 2020-2020 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP), pages 8224–8228. IEEE,

[Gao et al., 2018]
Yifan Gao, Lidong Bing, Wang Chen, Michael R Lyu, and Irwin King.
Difficulty controllable generation of reading comprehension
arXiv preprint arXiv:1807.03586, 2018.

[Ali et al., 2010]
Husam Ali, Yllias Chali, and Sadid A Hasan.
Automatic question generation from sentences.
In Actes de la 17e conférence sur le Traitement Automatique
des Langues Naturelles. Articles courts, pages 213–218, 2010.

[Du et al., 2017]
Xinya Du, Junru Shao, and Claire Cardie.
Learning to ask: Neural question generation for reading
arXiv preprint arXiv:1705.00106, 2017.

[Zhao et al., 2018]
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke.
Paragraph-level neural question generation with maxout pointer and
gated self-attention networks.
In Proceedings of the 2018 Conference on Empirical Methods in
Natural Language Processing, pages 3901–3910, 2018.

[Pan et al., 2020]
Liangming Pan, Yuxi Xie, Yansong Feng, Tat-Seng Chua, and Min-Yen Kan.
Semantic graphs for generating deep questions.
arXiv preprint arXiv:2004.12704, 2020b.

[Cho et al., 2019]
Woon Sang Cho, Yizhe Zhang, Sudha Rao, Asli Celikyilmaz, Chenyan Xiong,
Jianfeng Gao, Mengdi Wang, and Bill Dolan.
Contrastive multi-document question generation.
arXiv preprint arXiv:1911.03047, 2019b.

[Song et al., 2018]
Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea.
Leveraging context information for natural question generation.
In Proceedings of the 2018 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 2 (Short Papers), pages 569–574, 2018.

[Hochreiter, 1998]
Sepp Hochreiter.
The vanishing gradient problem during learning recurrent neural nets
and problem solutions.
International Journal of Uncertainty, Fuzziness and
Knowledge-Based Systems, 60 (02):0 107–116, 1998.

[Gers et al., 2000]
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins.
Learning to forget: Continual prediction with lstm.
Neural computation, 120 (10):0 2451–2471,

[Yuan et al., 2017]
Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman,
Sandeep Subramanian, Saizheng Zhang, and Adam Trischler.
Machine comprehension by text-to-text neural question generation.
arXiv preprint arXiv:1705.02012, 2017.

[Zhang and Bansal, 2019]
Shiyue Zhang and Mohit Bansal.
Addressing semantic drift in question generation for semi-supervised
question answering.
arXiv preprint arXiv:1909.06356, 2019.

[Qi et al., 2020]
Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
Zhang, and Ming Zhou.
Prophetnet: Predicting future n-gram for sequence-to-sequence
arXiv preprint arXiv:2001.04063, 2020.

[Lewis et al., 2020]
Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel.
Question and answer test-train overlap in open-domain question
answering datasets.
arXiv preprint arXiv:2008.02637, 2020.

[Trischler et al., 2016]
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni,
Philip Bachman, and Kaheer Suleman.
Newsqa: A machine comprehension dataset.
arXiv preprint arXiv:1611.09830, 2016.

[Dunn et al., 2017]
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and
Kyunghyun Cho.
Searchqa: A new q&a dataset augmented with context from a search
arXiv preprint arXiv:1704.05179, 2017.

[Bajaj et al., 2016]
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu,
Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al.
Ms marco: A human generated machine reading comprehension dataset.
arXiv preprint arXiv:1611.09268, 2016.

[Kočiskỳ et al., 2018]
Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer,
Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette.
The narrativeqa reading comprehension challenge.
Transactions of the Association for Computational Linguistics,
6:0 317–328, 2018.

[He et al., 2017]
Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu,
Yizhong Wang, Hua Wu, Qiaoqiao She, et al.
Dureader: a chinese machine reading comprehension dataset from
real-world applications.
arXiv preprint arXiv:1711.05073, 2017.

[Chen et al., 2018]
Guanliang Chen, Jie Yang, Claudia Hauff, and Geert-Jan Houben.
Learningq: a large-scale dataset for educational question generation.
In Twelfth International AAAI Conference on Web and Social
Media, 2018.

[Choi et al., 2018]
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy
Liang, and Luke Zettlemoyer.
Quac: Question answering in context.
arXiv preprint arXiv:1808.07036, 2018.

[Hill et al., 2015]
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston.
The goldilocks principle: Reading children's books with explicit
memory representations.
arXiv preprint arXiv:1511.02301, 2015.

[Lai et al., 2017]
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.
Race: Large-scale reading comprehension dataset from examinations.
arXiv preprint arXiv:1704.04683, 2017.

[Richardson et al., 2013]
Matthew Richardson, Christopher JC Burges, and Erin Renshaw.
Mctest: A challenge dataset for the open-domain machine comprehension
of text.
In Proceedings of the 2013 conference on empirical methods in
natural language processing, pages 193–203, 2013.

[Van Der Lee et al., 2019]
Chris Van Der Lee, Albert Gatt, Emiel Van Miltenburg, Sander Wubben, and Emiel
Best practices for the human evaluation of automatically generated
In Proceedings of the 12th International Conference on Natural
Language Generation, pages 355–368, 2019.

[Lin and Hovy, 2003]
Chin-Yew Lin and Eduard Hovy.
Automatic evaluation of summaries using n-gram co-occurrence
In Proceedings of the 2003 human language technology conference
of the North American chapter of the association for computational
linguistics, pages 150–157, 2003.

[Likert, 1932]
Rensis Likert.
A technique for the measurement of attitudes.
Archives of psychology, 1932.

[Amidei, 2021]
Jacopo Amidei.
Evaluating the Evaluators: Subjective Bias and Consistency in
Human Evaluation of Natural Language Generation.
Open University (United Kingdom), 2021.

[Belz and Kow, 2010]
Anja Belz and Eric Kow.
Comparing rating scales and preference judgements in language
In Proceedings of the 6th International Natural Language
Generation Conference, 2010.

[Raykar et al., 2010]
Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles
Florin, Luca Bogoni, and Linda Moy.
Learning from crowds.
Journal of machine learning research, 110 (4), 2010.

[Krippendorff, 2011]
Klaus Krippendorff.
Computing krippendorff's alpha-reliability.

[Thomson and Reiter, 2020]
Craig Thomson and Ehud Reiter.
A gold standard methodology for evaluating accuracy in data-to-text
arXiv preprint arXiv:2011.03992, 2020.

[Lowe et al., 2017]
Ryan Lowe, Michael Noseworthy, Iulian V Serban, Nicolas Angelard-Gontier,
Yoshua Bengio, and Joelle Pineau.
Towards an automatic turing test: Learning to evaluate dialogue
arXiv preprint arXiv:1708.07149, 2017.

[Reiter and Belz, 2009]
Ehud Reiter and Anja Belz.
An investigation into the validity of some metrics for automatically
evaluating natural language generation systems.
Computational Linguistics, 350 (4):0 529–558,

[Popović, 2015]
Maja Popović.
chrf: character n-gram f-score for automatic mt evaluation.
In Proceedings of the Tenth Workshop on Statistical Machine
Translation, pages 392–395, 2015.

[Levenshtein et al., 1966]
Vladimir I Levenshtein et al.
Binary codes capable of correcting deletions, insertions, and
In Soviet physics doklady, volume 10, pages 707–710. Soviet
Union, 1966.

[Roemmele and Gordon, 2015]
Melissa Roemmele and Andrew S Gordon.
Creative help: A story writing assistant.
In International Conference on Interactive Digital
Storytelling, pages 81–92. Springer, 2015.

[Papineni et al., 2002]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In Proceedings of the 40th annual meeting of the Association
for Computational Linguistics, pages 311–318, 2002.

[Banerjee and Lavie, 2005]
Satanjeev Banerjee and Alon Lavie.
Meteor: An automatic metric for mt evaluation with improved
correlation with human judgments.
In Proceedings of the acl workshop on intrinsic and extrinsic
evaluation measures for machine translation and/or summarization, pages
65–72, 2005.

[Lin, 2004]
Chin-Yew Lin.
Rouge: A package for automatic evaluation of summaries.
In Text summarization branches out, pages 74–81, 2004.

[Doddington, 2002]
George Doddington.
Automatic evaluation of machine translation quality using n-gram
co-occurrence statistics.
In Proceedings of the second international conference on Human
Language Technology Research, pages 138–145, 2002.

[Vedantam et al., 2015]
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.
Cider: Consensus-based image description evaluation.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 4566–4575, 2015.

[Mikolov et al., 2013]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their
Advances in neural information processing systems, 26, 2013.

[Lo, 2017]
Chi-kiu Lo.
Meant 2.0: Accurate semantic mt evaluation for any output language.
In Proceedings of the second conference on machine
translation, pages 589–597, 2017.

[Clark et al., 2019]
Elizabeth Clark, Asli Celikyilmaz, and Noah A Smith.
Sentence mover’s similarity: Automatic evaluation for
multi-sentence texts.
In Proceedings of the 57th Annual Meeting of the Association
for Computational Linguistics, pages 2748–2760, 2019.

[Zhang et al., 2019]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
arXiv preprint arXiv:1904.09675, 2019c.

[Stanojević and Sima’an, 2014]
Miloš Stanojević and Khalil Sima’an.
Beer: Better evaluation as ranking.
In Proceedings of the Ninth Workshop on Statistical Machine
Translation, pages 414–419, 2014.

[Ma et al., 2017]
Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu.
Blend: a novel combined mt metric based on direct
assessment—casict-dcu submission to wmt17 metrics task.
In Proceedings of the second conference on machine
translation, pages 598–603, 2017.

[Sharif et al., 2018]
Naeha Sharif, Mohammed Bennamoun, Lyndon Rhys White, and Syed Afaq Ali Shah.
Learning-based composite metrics for improved caption evaluation.

[Tao et al., 2018]
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan.
Ruber: An unsupervised method for automatic evaluation of open-domain
dialog systems.
In Thirty-Second AAAI Conference on Artificial Intelligence,

[Sinha et al., 2020]
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L
Hamilton, and Joelle Pineau.
Learning an unreferenced metric for online dialogue evaluation.
arXiv preprint arXiv:2005.00583, 2020.

[Shimanaka et al., 2018]
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi.
Ruse: Regressor using sentence embeddings for automatic machine
translation evaluation.
In Proceedings of the Third Conference on Machine Translation:
Shared Task Papers, pages 751–758, 2018.

[Sellam et al., 2020]
Thibault Sellam, Dipanjan Das, and Ankur P Parikh.
Bleurt: Learning robust metrics for text generation.
arXiv preprint arXiv:2004.04696, 2020.

[Serban et al., 2017]
Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau,
Aaron Courville, and Yoshua Bengio.
A hierarchical latent variable encoder-decoder model for generating
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 31, 2017.

[Gutmann and Hyvärinen, 2010]
Michael Gutmann and Aapo Hyvärinen.
Noise-contrastive estimation: A new estimation principle for
unnormalized statistical models.
In Proceedings of the thirteenth international conference on
artificial intelligence and statistics, pages 297–304. JMLR Workshop and
Conference Proceedings, 2010.

[Conneau et al., 2017]
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes.
Supervised learning of universal sentence representations from
natural language inference data.
arXiv preprint arXiv:1705.02364, 2017.

[Logeswaran and Lee, 2018]
Lajanugen Logeswaran and Honglak Lee.
An efficient framework for learning sentence representations.
arXiv preprint arXiv:1803.02893, 2018.

[Cer et al., 2018]
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St
John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al.
Universal sentence encoder.
arXiv preprint arXiv:1803.11175, 2018.

[Vinyals et al., 2015]
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan.
Show and tell: A neural image caption generator.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 3156–3164, 2015.

[Xu et al., 2015]
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan
Salakhudinov, Rich Zemel, and Yoshua Bengio.
Show, attend and tell: Neural image caption generation with visual
In International conference on machine learning, pages
2048–2057. PMLR, 2015b.

[Lu et al., 2017]
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher.
Knowing when to look: Adaptive attention via a visual sentinel for
image captioning.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pages 375–383, 2017.

[Callison-Burch et al., 2006]
Chris Callison-Burch, Miles Osborne, and Philipp Koehn.
Re-evaluating the role of bleu in machine translation research.
In 11th conference of the european chapter of the association
for computational linguistics, pages 249–256, 2006.

[Galley et al., 2015]
Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli,
Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan.
deltableu: A discriminative metric for generation tasks with
intrinsically diverse targets.
arXiv preprint arXiv:1506.06863, 2015.

[Chen et al., 2019]
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
Evaluating question answering evaluation.
In Proceedings of the 2nd Workshop on Machine Reading for
Question Answering, pages 119–124, 2019.

[Ding et al., 2018]
Jie Ding, Vahid Tarokh, and Yuhong Yang.
Model selection techniques: An overview.
IEEE Signal Processing Magazine, 350 (6):0
16–34, 2018.

[Emmert-Streib and Dehmer, 2019]
Frank Emmert-Streib and Matthias Dehmer.
Evaluation of regression models: Model assessment, model selection
and generalization error.
Machine learning and knowledge extraction, 10
(1):0 521–551, 2019.