hallucination. Lee et al. (2019) and Raunak et al. (2021) suggest addition of
perturbed sentences. Furthermore, perturbation, where the insertions of most
common tokens are placed at the beginning of the sentence, seems to be the
most successful in hallucination mitigation. A disadvantage of this method is
the need to understand different types of hallucinations produced by the model
in order to apply a correct augmentation method. Corpus filtering is a method
of mitigating hallucinations caused by the noise in the dataset by removing
the repetitive and mismatching source and target sequences (Raunak et al.,
2021). Junczys-Dowmunt (2018) implements a cross-entropy data filtering method
for bilingual data, which uses cross-entropy scores calculated for noisy pairs
according to two translation models trained on the clean data. The scores that
suggest dissagreament between sentence pairs from two models are subsequently
penalized.

While (Raunak et al., 2021; Lee et al., 2019) and (Junczys-Dowmunt, 2018)
define noise as mismatched source and target sentences, (Briakou and Carpuat,
2021) analyzes the influence of fine-grained semantic divergences on NMT
outputs. The authors consequently propose a mitigation method for fine-grained
divergences based on semantic factors. The tags are applied to each source and
target sentence to inform about the position of divergent tokens. Factorizing
divergence not only helps to mitigate hallucinations, but improves the overall
performance of the NMT. This shows that tagging small semantic divergences can
provide useful information for the network during training.

####  11.3.2. Modeling and Inference

Overexposure bias is a common problem in NMT, amplified by the teacher-forcing
technique used in sequence-to-sequence models. The models are trained on the
ground truth, but during inference, they attend to the past predictions, which
can be incorrect (Ranzato et al., 2016; Kong et al., 2019). To mitigate this
problem, Wang and Sennrich (2020) propose substituting MLE as a training
objective with minimum risk training (MRT) (Och, 2003). Scheduled sampling is
a classic method of mitigating overexposure bias first proposed by (Bengio et
al., 2015). Based on that method, (Goyal et al., 2017) create a differentiable
approximation to greedy decoding that shows a good performance in the NMT
task. (Xu et al., 2019) propose further improvement of the scheduled sampling
algorithm for NMT by optimizing the probability of source and target word
alignments. This improvement helps to address the issue flexibility in word
order between a source and target language when performing scheduled sampling.

Zhou et al. (2021b) propose a method of improving self-training of NMT based
on hallucination detection. They create hallucination labels (see Section
11.2.2), and then discard losses of tokens predicted as hallucinations, which
is known as token loss truncation. This is similar to the method proposed by
Kang and Hashimoto (2020a), the latter for full sentences in the summarization
task. Furthermore, instead of adjusting losses, Zhou et al. (2021b) mask the
hidden states of the discarded losses in the decoder in a procedure called
decoder HS masking. Experimental results show both a translation quality
improvement in terms of BLEU and also a large reduction in hallucination. The
token loss truncation method shows good results in the low-resource languages
scenario.

Another method to mitigate the impact of noisy datasets is tilted empirical
risk minimization (TERM), a training objective proposed by Li et al. (2020a).
(Lee et al., 2019) mentions that techniques such as dropout, L2E
regularization, and clipping tend to decrease the number of hallucinations.
Lastly, several authors propose methods of improving phrase alignment that are
helpful both in increasing translation accuracy and identifying content that
did not appear in the source translation (Weng et al., 2020; Zhang et al.,
2021b; Garg et al., 2019).

###  11.4. Future Directions in NMT

The future work on hallucinations in NMT is to define hallucinations in a
quantifiable manner; i.e., to specify a cut-off value between translation
error and hallucinated content using a particular metric. Martindale et al.
(2019) propose a threshold between fluency and adequacy which is the closest
to this ideal. They, however, do not concentrate on hallucinated content as
such, and thus fluent but inadequate sentences may not always indicate
hallucinations but also other types of translation errors. Balakrishnan et al.
(2019) mention constrained decoding as a method to mitigate hallucinations in
dialogue systems, but it could also be applied in NMT. (Hokamp and Liu, 2017;
Post and Vilar, 2018; Song et al., 2019; Dinu et al., 2019; Susanto et al.,
2020; Xu and Carpuat, 2021) and (Xu and Carpuat, 2020) use constrained
decoding to incorporate specific terminology into MT, but the above methods
can be repurposed to mitigate hallucinations.

Another direction for future work on hallucinations is improving existing
methods of searching for hallucinatory content, such as the algorithms
proposed by Lee et al. (2019); Feng et al. (2020) and Raunak et al. (2021),
that are computationally expensive (Raunak et al., 2021) or require the
creation of an additional perturbed test-set (Lee et al., 2019). Similarly,
for mitigation of lack of faithfulness and fluency, the method proposed by
Feng et al. (2020) requires the creation of a one-to-many architecture (one
encoder and two decoders), which is also computationally expensive. Future
directions would therefore include simplification of existing hallucination
evaluation methods, applying them to different architectures like CNNs and
transformers, and possibly conducting research on finding simpler
hallucination search methods.

##  12\. Hallucination in Vision-Language Generation

With the vast advancement of the Transformer architecture (Vaswani et al.,
2017; Dosovitskiy et al., 2021) in both CV and NLP, there is a trend to pre-
train large-scale unified vision-language (VL) models (Wang et al., 2022b; Dai
et al., 2022a; Li et al., 2022; Wang et al., 2022c; Alayrac et al., 2022; Wang
et al., 2022a) to perform vision grounded text generation tasks, such as image
captioning and visual question answering (VQA). Generally, there are two
common schemas for vision-language pre-training: 1) pre-train from scratch
with a massive amount of image-text pairs as well as optionally a large text-
only corpus; or 2) initialize model parameters from a large pre-trained LM and
then adapt it to the VL domain with adequate image-text pairs. Either way, the
learned vision and language representations are aligned in the same multimodal
space and the resulting model can be seen as a LM that understands visual
information. Therefore, the hallucination problem is also observed in VL
models due to similar reasons as found in NLG.

In the VL domain, the research on hallucination is still in its very early
stage and how to measure and mitigate hallucination is an open question. In
this section, we first review the hallucination in image captioning as it is
the only VL task that has corresponding previous research works. Then, we
introduce hallucination phenomena found in other VL tasks. Finally, we discuss
potential future research directions on this problem.

###  12.1. Object Hallucination in Image Captioning

##### Definition.

Object hallucination is defined as models generating captions that contain
non-existent or inaccurate objects from the input image. Following tasks in
NLG, we also categorize object hallucination into intrinsic and extrinsic
ones:

Figure 2. Examples of intrinsic and extrinsic object hallucination in image
captioning.

  * •

Intrinsic Object Hallucination: captions contain incorrect or definitely non-
existent objects given the input image. For example in Figure 2, there is no
“mirror” or “football” on top of the chest in the given image.

  * •

Extrinsic Object Hallucination: captions contain objects cannot be verified
their existence from the input image. For example in Figure 2, we cannot
verify whether there are “letters” in the drawer or a “fan” on the roof.

##### Metrics

To automatically measure object hallucination, Rohrbach et al. (2018) propose
the CHAIR (Caption Hallucination Assessment with Image Relevance) metric,
which calculates what proportion of object words generated are actually in the
image according to the ground truth captions. Specifically, there are two
variants of it, which are CHAIRi and CHAIRs defined as follows,

$\displaystyle\textrm{CHAIR}_{i}=\frac{\textrm{\\# \\{hallucinated
objects\\}}}{\textrm{\\# \\{all objects in ground
truth\\}}},\textrm{CHAIR}_{s}=\frac{\textrm{\\# \\{hallucinated
captions\\}}}{\textrm{\\# \\{all captions\\}}}.$

CHAIRi measures per-instance object hallucination, i.e. what fraction of
object instances in each generated caption are hallucinated. CHAIRs measures
per-sentence object hallucination, i.e. what fraction of generated captions
include at least one hallucinated object. For example, to calculate CHAIR
scores for the MSCOCO dataset (Lin et al., 2014), Rohrbach et al. (2018) apply
the 80-object list used in the MSCOCO segmentation challenge Lu et al. (2018)
and find exact matches of object words or phrases in captions.

##### Mitigation

As a research problem that is in its early stage, there are currently a
limited number of approaches proposed to mitigate object hallucination in
image captioning. Biten et al. (2022) hypothesize that the main cause of
object hallucination is the systematic co-occurence of particular object
categories in input images. They propose three simple yet effective ways of
data augmentation to make the co-occurence statistics matrix more uniform to
mitigate object hallucination. Results show that their introduced method can
reduce object hallucination without changing model architectures. From another
perspective, Xiao and Wang (2021) propose an uncertainty-aware beam search
method for decoding and exhibit that reducing uncertainty can lead to less
hallucination. Specifically, a weighted penalty term is added to the beam
search objective to balance between log probability and predictive uncertainty
of the selected word candidates. More recently, Dai et al. (2022b) analyze
object hallucination in VL pre-training and propose a novel pre-training
objective named object masked language modeling to alleviate this problem.

###  12.2. Hallucination in Other VL Tasks

In addition to image captioning, hallucination has also been observed in other
VL tasks and raised as an open research question. For example, in open-ended
visual question answering, Figure 3 (left and right) shows that the model
could generate seem likely answers when we only see the text, however wrong
when given the image. Moreover, Figure 3 (middle) indicates that hallucination
can also be triggered by adversarially prompting an unanswerable question. The
model will imagine an unsupported answer that commonly matches the given
visual scene.

Figure 3. Examples of hallucination in visual question answering (taken from
(Alayrac et al., 2022)). The bold text is the output generated by the model
and the part before it is the input prompt.

###  12.3. Future Directions in VL

For future research on the hallucination problem in VL, we summarize three
promising directions. Firstly, hallucination in VL is still in the early
stage. There is a lack of empirical and theoretical analyses in many tasks,
such as visual storytelling, visual commonsense reasoning, video captioning,
etc. Secondly, more effective evaluation metrics are needed. For example,
although CHAIR can automatically evaluate the degree of object hallucination
in image captioning, it requires a pre-defined list of object categories,
which does not generalize well. Furthermore, for the hallucination types
discussed in Section 12.2, currently there is no automatic metric. Therefore,
we cannot perform quantitative evaluations for them. Thirdly, we believe how
to perform controlled generation (Rebuffel et al., 2022; Dai et al., 2022b)
with visual grounding is a promising direction to mitigate hallucination in
VL.

##  13\. Conclusion

In this survey, we provide the first comprehensive overview of the
hallucination problem in NLG, summarizing existing evaluation metrics,
mitigation methods, and the remaining challenges for future research.
Hallucination is an artifact of neural-based NLG and is of concern because
they appear fluent and can therefore be misleading to users. In some scenarios
and tasks, hallucination can cause harm. We survey various contributors to
hallucination, ranging from noisy data, erroneous parametric knowledge,
incorrect attention mechanism, inappropriate training strategy, to inference
exposure bias, etc. We show that there are two categories of hallucinations,
namely intrinsic hallucination and extrinsic hallucination, and they need to
be treated differently with diverse mitigation strategies. Hallucination is
relatively easy to detect in abstractive summarization and in NMT against the
evidence in the source. For dialogue systems, it is important to balance
diversity vs consistency in dialogue responses. Hallucination in GQA and VL
tasks is detrimental to the performance, but research on mitigation methods is
still very preliminary in these areas. For data-to-text generation,
hallucination arises from the discrepancy between the input and output format.
Most methods to mitigate hallucinations in NMT either aim to reduce dataset
noise or alleviate exposure bias. In the VL domain, models also generate
unfaithful output given the visual scene, and recent works have mainly focused
on the object hallucination problem. There remain many challenges ahead in
identifying and mitigating hallucinations in NLG, and we hope research in this
area can benefit from this survey.

## References

  * (1)
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022\.  Flamingo: a Visual Language Model for Few-Shot Learning.  _ArXiv_ abs/2204.14198 (2022).
  * Albrecht and Hwa (2007) Joshua Albrecht and Rebecca Hwa. 2007.  A Re-examination of Machine Learning Approaches for Sentence-Level MT Evaluation. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_. Association for Computational Linguistics, Prague, Czech Republic, 880–887.  https://aclanthology.org/P07-1111
  * Aralikatte et al. (2021) Rahul Aralikatte, Shashi Narayan, Joshua Maynez, Sascha Rothe, and Ryan McDonald. 2021.  Focus Attention: Promoting Faithfulness and Diversity in Summarization.  _ACL_ (2021).
  * Bai et al. (2009) Xiang Bai, Xinggang Wang, Longin Jan Latecki, Wenyu Liu, and Zhuowen Tu. 2009.  Active Skeleton for Non-rigid Object Detection. In _2009 IEEE 12th International Conference on Computer Vision_. 575–582.  https://doi.org/10.1109/ICCV.2009.5459188
  * Baker and Kanade (2000) S. Baker and T. Kanade. 2000.  Hallucinating Faces. In _Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580)_. 83–88.  https://doi.org/10.1109/AFGR.2000.840616
  * Balakrishnan et al. (2019) Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019\.  Constrained Decoding for Neural NLG from Compositional Representations in Task-Oriented Dialogue. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 831–844.  https://doi.org/10.18653/v1/P19-1080
  * Baly et al. (2018) Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018\.  Predicting Factuality of Reporting and Bias of News Media Sources. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 3528–3539.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020\.  Longformer: The Long-Document Transformer.  _arXiv:2004.05150_ (2020).
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.  _Advances in neural information processing systems_ 28 (2015).
  * Beyer et al. (2021) Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021\.  Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Online, 4164–4173.  https://doi.org/10.18653/v1/2021.naacl-main.328
  * Bi et al. (2019) Bin Bi, Chen Wu, Ming Yan, Wei Wang, Jiangnan Xia, and Chenliang Li. 2019\.  Incorporating External Knowledge into Machine Reading for Generative Question Answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 2521–2530.
  * Białecki et al. (2012) Andrzej Białecki, Robert Muir, Grant Ingersoll, and Lucid Imagination. 2012.  Apache lucene 4. In _SIGIR 2012 workshop on open source information retrieval_. 17.
  * Biten et al. (2022) Ali Furkan Biten, Lluís Gómez, and Dimosthenis Karatzas. 2022.  Let There Be a Clock on the Beach: Reducing Object Hallucination in Image Captioning. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_. 1381–1390.
  * Blom ([n. d.]) Jan Dirk Blom. [n. d.].  _A Dictionary of Hallucinations_.  Springer.
  * Briakou and Carpuat (2021) Eleftheria Briakou and Marine Carpuat. 2021.  Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation.  _CoRR_ abs/2105.15087 (2021).  arXiv:2105.15087 https://arxiv.org/abs/2105.15087
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020\.  Factual Error Correction for Abstractive Summarization Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 6251–6258.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization.  _EMNLP_ (2021).
  * Cao et al. (2018) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018\.  Faithful to the Original: Fact Aware Neural Abstractive Summarization. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Carlini et al. (2020) Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Úlfar Erlingsson, et al. 2020\.  Extracting Training Data from Large Language Models.  (2020).
  * Chen et al. (2021b) Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021b.  Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 5935–5941.
  * Chen et al. (2021a) Wenqing Chen, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. 2021a.  De-Confounded Variational Encoder-Decoder for Logical Table-to-Text Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 5532–5542.
  * Chen et al. (2020) Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020.  Logic2Text: High-Fidelity Natural Language Generation from Logical Forms. In _EMNLP (Findings)_.
  * Chisholm et al. (2017) Andrew Chisholm, Will Radford, and Ben Hachey. 2017\.  Learning to generate one-sentence biographies from Wikidata. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_. 633–642.
  * Crawshaw (2020) Michael Crawshaw. 2020\.  Multi-Task Learning with Deep Neural Networks: A Survey.  _arXiv preprint arXiv:2009.09796_ (2020).
  * Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020.  MuTual: A Dataset for Multi-Turn Dialogue Reasoning. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 1406–1416.
  * Dai et al. (2022a) Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022a.  Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation. In _Findings of the Association for Computational Linguistics: ACL 2022_. Association for Computational Linguistics, Dublin, Ireland, 2383–2395.  https://doi.org/10.18653/v1/2022.findings-acl.187
  * Dai et al. (2022b) Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. 2022b.  Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training.  _ArXiv_ abs/2210.07688 (2022).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dhingra et al. (2019) Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019\.  Handling Divergent Reference Texts when Evaluating Table-to-Text Generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 4884–4895.
  * Dinan et al. (2020) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, et al. 2020\.  The Second Conversational Intelligence Challenge (ConvAI2).  In _The NeurIPS’18 Competition_. Springer, 187–208.
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019\.  Wizard of Wikipedia: Knowledge-Powered Conversational agents.  _ICLR_ (2019).
  * Dinu et al. (2019) Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019.  Training Neural Machine Translation To Apply Terminology Constraints.  _ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_ (6 2019), 3063–3068.  https://doi.org/10.18653/v1/p19-1294
  * Dong et al. (2020) Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020.  Multi-Fact Correction in Abstractive Text Summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 9320–9331.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=YicbFdNTTy
  * Durmus et al. (2020) Esin Durmus, He He, and Mona Diab. 2020.  FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 5055–5070.
  * Dušek et al. (2019) Ondřej Dušek, David M Howcroft, and Verena Rieser. 2019\.  Semantic Noise Matters for Neural Natural Language Generation. In _Proceedings of the 12th International Conference on Natural Language Generation_. 421–426.
  * Dušek and Jurčíček (2016) Ondřej Dušek and Filip Jurčíček. 2016\.  Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Strings. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_. Association for Computational Linguistics, Berlin, Germany, 45–51.  https://doi.org/10.18653/v1/P16-2008
  * Dušek and Kasner (2020) Ondřej Dušek and Zdeněk Kasner. 2020.  Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference. In _Proceedings of the 13th International Conference on Natural Language Generation_. Association for Computational Linguistics, Dublin, Ireland, 131–137.  https://aclanthology.org/2020.inlg-1.19
  * Dziri et al. (2019) Nouha Dziri, Ehsan Kamalloo, Kory Mathewson, and Osmar Zaiane. 2019.  Evaluating Coherence in Dialogue Systems using Entailment. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 3806–3812.  https://doi.org/10.18653/v1/N19-1381
  * Dziri et al. (2021a) Nouha Dziri, Andrea Madotto, Osmar Zaiane, and Avishek Joey Bose. 2021a.  Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding.  _EMNLP_ (2021).
  * Dziri et al. (2021b) Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021b.  Evaluating Groundedness in Dialogue Systems: The BEGIN Benchmark.  _Findings of ACL_ (2021).
  * Eric and Manning (2017) Mihail Eric and Christopher Manning. 2017.  A Copy-Augmented Sequence-to-Sequence Architecture Gives Good Performance on Task-Oriented Dialogue. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_. Association for Computational Linguistics, Valencia, Spain, 468–473.  https://aclanthology.org/E17-2075
  * Etzioni et al. (2008) Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S. Weld. 2008.  Open Information Extraction from the Web.  _Commun. ACM_ 51, 12 (Dec. 2008), 68–74.  https://doi.org/10.1145/1409360.1409378
  * Falke et al. (2019) Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019.  Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 2214–2220.
  * Fan et al. (2019a) Angela Fan, Claire Gardent, Chloé Braud, and Antoine Bordes. 2019a.  Using Local Knowledge Graph Construction to Scale Seq2Seq Models to Multi-Document Inputs. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 4186–4196.
  * Fan et al. (2019b) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019b.  ELI5: Long Form Question Answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 3558–3567.
  * Fawzi et al. (2016) Alhussein Fawzi, Horst Samulowitz, Deepak Turaga, and Pascal Frossard. 2016.  Image Inpainting through Neural Networks Hallucinations. In _2016 IEEE 12th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP)_. Ieee, 1–5.
  * Feng et al. (2020) Yang Feng, Wanying Xie, Shuhao Gu, Chenze Shao, Wen Zhang, Zhengxin Yang, and Dong Yu. 2020.  Modeling Fluency and Faithfulness for Diverse Neural Machine Translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 59–66.
  * Filippova (2020) Katja Filippova. 2020\.  Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_. 864–870.
  * Fish et al. (2009) William Fish et al. 2009\.  _Perception, hallucination, and illusion_.  OUP USA.
  * Gabriel et al. (2021) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021.  GO FIGURE: A Meta Evaluation of Factuality in Summarization. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Association for Computational Linguistics, Online, 478–487.  https://doi.org/10.18653/v1/2021.findings-acl.42
  * Gao et al. (2018) Jianfeng Gao, Michel Galley, and Lihong Li. 2018.  Neural Approaches to Conversational AI. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts_. Association for Computational Linguistics, Melbourne, Australia, 2–7.  https://doi.org/10.18653/v1/P18-5002
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  Creating Training Corpora for NLG Micro-Planning. In _55th annual meeting of the Association for Computational Linguistics (ACL)_.
  * Garg et al. (2019) Sarthak Garg, Stephan Peitz, Udhyakumar Nallasamy, and Matthias Paulik. 2019.  Jointly Learning to Align and Translate with Transformer Models. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 4453–4462.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation.  _Journal of Artificial Intelligence Research_ 61 (2018), 65–170.
  * Ghosal et al. (2021) Deepanway Ghosal, Pengfei Hong, Siqi Shen, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2021\.  CIDER: Commonsense Inference for Dialogue Explanation and Reasoning.  _ACL_ (2021).
  * Ginsca et al. (2015) Alexandru L Ginsca, Adrian Popescu, and Mihai Lupu. 2015\.  Credibility in Information Retrieval.  _Foundations and Trends in Information Retrieval_ 9, 5 (2015), 355–475.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization. In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_. Association for Computational Linguistics, Hong Kong, China, 70–79.  https://doi.org/10.18653/v1/D19-5409
  * Göbel and Rushworth (2004) Silke M Göbel and Matthew FS Rushworth. 2004.  Cognitive Neuroscience: Acting on Numbers.  _Current Biology_ 14, 13 (2004), R517–R519.
  * Goodrich et al. (2019) Ben Goodrich, Vinay Rao, Peter J Liu, and Mohammad Saleh. 2019\.  Assessing the Factual Accuracy of Generated Text. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 166–175.
  * Goyal et al. (2017) Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. 2017\.  Differentiable Scheduled Sampling for Credit Assignment.  _ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_ 2 (4 2017), 366–371.  https://doi.org/10.18653/v1/P17-2058
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating Factuality in Generation with Dependency-level Entailment. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_. 3592–3603.
  * Guan and Huang (2020) Jian Guan and Minlie Huang. 2020.  UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 9157–9166.
  * Gunel et al. (2019) Beliz Gunel, Chenguang Zhu, Michael Zeng, and Xuedong Huang. 2019\.  Mind the facts: Knowledge-Boosted Coherent Abstractive Text Summarization.  _NeurIPS, Knowledge Representation & Reasoning Meets Machine Learning (KR2ML workshop)_ (2019).
  * Gupta et al. (2021) Prakhar Gupta, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021\.  DialFact: A Benchmark for Fact-Checking in Dialogue.  _arXiv preprint arXiv:2110.08222_ (2021).
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018\.  Annotation Artifacts in Natural Language Inference Data. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_. 107–112.
  * Hancock et al. (2019) Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019.  Learning from Dialogue after Deployment: Feed Yourself, Chatbot!. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 3667–3684.  https://doi.org/10.18653/v1/P19-1358
  * He et al. (2021) Tianxing He, Jingzhao Zhang, Zhiming Zhou, and James Glass. 2021\.  Exposure Bias versus Self-Recovery: Are Distortions Really Incremental for Autoregressive Text Generation?. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 5087–5102.
  * Hokamp and Liu (2017) Chris Hokamp and Qun Liu. 2017.  Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search.  _ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers)_ 1 (4 2017), 1535–1546.  https://doi.org/10.18653/v1/P17-1141
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The Curious Case of Neural Text Degeneration. In _International Conference on Learning Representations_.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022\.  TRUE: Re-evaluating Factual Consistency Evaluation. In _Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering_. 161–175.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021\.  Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering.  _EMNLP_ (2021).
  * Huang et al. (2020a) Luyang Huang, Lingfei Wu, and Lu Wang. 2020a.  Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward.  _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ (2020).
  * Huang et al. (2020b) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020b.  Challenges in Building Intelligent Open-domain Dialog Systems.  _ACM Transactions on Information Systems (TOIS)_ 38, 3 (2020), 1–32.
  * Huang et al. (2021) Yichong Huang, Xiachong Feng, Xiaocheng Feng, and Bing Qin. 2021\.  The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey.  _arXiv preprint arXiv:2104.14839_ (2021).
  * Hussein et al. (2017) Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017.  Imitation Learning: A Survey of Learning Methods.  _ACM Comput. Surv._ 50, 2, Article 21 (apr 2017), 35 pages.  https://doi.org/10.1145/3054912
  * Ji et al. (2022) Ziwei Ji, Yan Xu, I-Tsun Cheng, Samuel Cahyawijaya, Rita Frieske, Etsuko Ishii, Min Zeng, Andrea Madotto, and Pascale Fung. 2022.  VScript: Controllable Script Generation with Visual Presentation.  arXiv:2203.00314
  * Junczys-Dowmunt (2018) Marcin Junczys-Dowmunt. 2018\.  Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora. In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_. Association for Computational Linguistics, Belgium, Brussels, 888–895.  https://doi.org/10.18653/v1/W18-6478
  * Jurafsky and Marin (2019) Daniel Jurafsky and James H. Marin. 2019.  _Speech and Language Processing_.  Draft of October 16th, 2019, Website: https://web.stanford.edu/~jurafsky/slp3/26.pdf, Chapter 26.
  * Kang and Hashimoto (2020a) Daniel Kang and Tatsunori Hashimoto. 2020a.  Improved Natural Language Generation via Loss Truncation.  (4 2020), 718–731.  https://arxiv.org/abs/2004.14589v2
  * Kang and Hashimoto (2020b) Daniel Kang and Tatsunori B Hashimoto. 2020b.  Improved Natural Language Generation via Loss Truncation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 718–731.
  * Kayhan et al. (2021) Osman Semih Kayhan, Bart Vredebregt, and Jan C van Gemert. 2021\.  Hallucination In Object Detection—A Study In Visual Part VERIFICATION. In _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2234–2238.
  * Khashabi et al. (2021) Daniel Khashabi, Amos Ng, Tushar Khot, Ashish Sabharwal, Hannaneh Hajishirzi, and Chris Callison-Burch. 2021.  GooAQ: Open Question Answering with Diverse Answer Types. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 421–433.  https://doi.org/10.18653/v1/2021.findings-emnlp.38
  * Koehn and Knowles (2017a) Philipp Koehn and Rebecca Knowles. 2017a.  Six Challenges for Neural Machine Translation. In _First Workshop on Neural Machine Translation_. Association for Computational Linguistics, 28–39.
  * Koehn and Knowles (2017b) Philipp Koehn and Rebecca Knowles. 2017b.  Six Challenges for Neural Machine Translation.  (2017), 28–39.  http://www.statmt.org/wmt17/
  * Kong et al. (2019) Xiang Kong, Zhaopeng Tu, Shuming Shi, Eduard Hovy, and Tong Zhang. 2019.  Neural Machine Translation with Adequacy-Oriented Learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 6618–6625.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to Progress in Long-form Question Answering. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 4940–4957.
  * Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020.  Evaluating the Factual Consistency of Abstractive Text Summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 9332–9346.
  * Kukich (1983) Karen Kukich. 1983\.  Design of a Knowledge-based Report Generator. In _21st Annual Meeting of the Association for Computational Linguistics_. 145–150.
  * Kulikov et al. (2019) Ilia Kulikov, Alexander H. Miller, Kyunghyun Cho, and Jason Weston. 2019.  Importance of Search and Evaluation Strategies in Neural Dialogue Modeling. In _Proceedings of the 12th International Conference on Natural Language Generation, INLG 2019, Tokyo, Japan, October 29 - November 1, 2019_ , Kees van Deemter, Chenghua Lin, and Hiroya Takamura (Eds.). Association for Computational Linguistics, 76–87.  https://doi.org/10.18653/v1/W19-8609
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019\.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 452–466.
  * Laban et al. (2022) Philippe Laban, Tobias Schnabel, Paul N Bennett, and Marti A Hearst. 2022.  SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization.  _Transactions of the Association for Computational Linguistics_ 10 (2022), 163–177.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016\.  Neural Text Generation from Structured Data with Application to the Biography Domain.  _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ (2016).
  * Lee et al. (2019) Katherine Lee, Orhan Firat, Ashish Agarwal, Clara Fannjiang, and David Sussillo. 2019.  Hallucinations in Neural Machine Translation.  _ICLR_ (2019).
  * Lee et al. (2021) Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021.  Deduplicating Training Data Makes Language Models Better.  _arXiv preprint arXiv:2107.06499_ (2021).
  * Lee et al. (2020) Nayeon Lee, Belinda Z Li, Sinong Wang, Wen-Tau Yih, Hao Ma, and Madian Khabsa. 2020\.  Language Models as Fact Checkers?  _ACL 2020_ (2020), 36\.
  * Lee et al. (2022) Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2022\.  Factuality Enhanced Language Models for Open-Ended Text Generation.  _arXiv preprint arXiv:2206.04624_ (2022).
  * Lee et al. ([n. d.]) Nayeon Lee, Chien-Sheng Wu, and Pascale Fung. [n. d.].  Improving Large-Scale Fact-Checking using Decomposable Attention Models and Lexical Tagging.  ([n. d.]).
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 7871–7880.
  * Li et al. (2021b) Bohan Li, Yutai Hou, and Wanxiang Che. 2021b.  Data Augmentation Approaches in Natural Language Processing: A Survey.  _arXiv preprint arXiv:2110.01852_ (2021).
  * Li et al. (2021a) Chenliang Li, Bin Bi, Ming Yan, Wei Wang, and Songfang Huang. 2021a.  Addressing Semantic Drift in Generative Question Answering with Auxiliary Extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_. 942–947.
  * Li et al. (2018) Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018\.  Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization. In _Proceedings of the 27th International Conference on Computational Linguistics_. Association for Computational Linguistics, Santa Fe, New Mexico, USA, 1430–1441.  https://aclanthology.org/C18-1121
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016\.  A Persona-Based Neural Conversation Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 994–1003.  https://doi.org/10.18653/v1/P16-1094
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022\.  BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In _ICML_.
  * Li et al. (2020b) Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020b.  Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 4715–4728.  https://doi.org/10.18653/v1/2020.acl-main.428
  * Li et al. (2020a) Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. 2020a.  Tilted Empirical Risk Minimization. In _International Conference on Learning Representations_.
  * Li et al. (2020c) Yangming Li, Kaisheng Yao, Libo Qin, Wanxiang Che, Xiaolong Li, and Ting Liu. 2020c.  Slot-consistent NLG for Task-oriented Dialogue Systems with Iterative Rectification Network. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 97–106.  https://doi.org/10.18653/v1/2020.acl-main.10
  * Lin (2004) Chin-Yew Lin. 2004\.  Rouge: A Package for Automatic Evaluation of Summaries. In _Text summarization branches out_. 74–81.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021\.  TruthfulQA: Measuring How Models Mimic Human Falsehoods.  _arXiv preprint arXiv:2109.07958_ (2021).
  * Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014.  Microsoft COCO: Common Objects in Context.  _ArXiv_ abs/1405.0312 (2014).
  * Liu et al. (2007) Ce Liu, Heung-Yeung Shum, and William T Freeman. 2007.  Face Hallucination: Theory and Practice.  _International Journal of Computer Vision_ 75, 1 (2007), 115–134.
  * Liu et al. (2021a) Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021a.  A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation.  _arXiv preprint arXiv:2104.08704_ (2021).
  * Liu et al. (2021b) Tianyu Liu, Xin Zheng, Baobao Chang, and Zhifang Sui. 2021b.  Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 13415–13423.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021\.  Entity-Based Knowledge Conflicts in Question Answering. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 7052–7063.
  * Lu et al. (2018) Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2018\.  Neural Baby Talk. In _CVPR_.
  * Ma et al. (2020) Yukun Ma, Khanh Linh Nguyen, Frank Z Xing, and Erik Cambria. 2020.  A Survey on Empathetic Dialogue Systems.  _Information Fusion_ 64 (2020), 50–70.
  * Macpherson and Platchias (2013) Fiona Macpherson and Dimitris Platchias. 2013.  _Hallucination: Philosophy and psychology_.  MIT Press.
  * Madotto et al. (2020a) Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata, Yan Xu, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020a.  Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 2372–2394.  https://doi.org/10.18653/v1/2020.findings-emnlp.215
  * Madotto et al. (2019) Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019.  Personalizing dialogue agents via meta-learning. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 5454–5459.
  * Madotto et al. (2020b) Andrea Madotto, Zihan Liu, Zhaojiang Lin, and Pascale Fung. 2020b.  Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems.  arXiv:2008.06239 [cs.CL]
  * Madotto et al. (2018) Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2018\.  Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Melbourne, Australia, 1468–1478.  https://doi.org/10.18653/v1/P18-1136
  * Magdy and Wanas (2010) Amr Magdy and Nayer Wanas. 2010.  Web-based Statistical Fact Checking of Textual Documents. In _Proceedings of the 2nd international workshop on Search and mining user-generated contents_. 103–110.
  * Martindale et al. (2019) Marianna J. Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019.  Identifying Fluently Inadequate Output in Neural and Statistical Machine Translation. In _MTSummit_.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On Faithfulness and Factuality in Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 1906–1919.
  * Mazaré et al. (2018) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018.  Training Millions of Personalized Dialogue Agents. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 2775–2779.  https://doi.org/10.18653/v1/D18-1298
  * McKeown (1992) Kathleen McKeown. 1992\.  _Text Generation_.  Cambridge University Press.
  * Mesgar et al. (2021) Mohsen Mesgar, Edwin Simpson, and Iryna Gurevych. 2021\.  Improving Factual Consistency Between a Response and Persona Facts. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. Association for Computational Linguistics, Online, 549–562.  https://doi.org/10.18653/v1/2021.eacl-main.44
  * Metzler et al. (2021) Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork. 2021\.  Rethinking Search: Making Experts out of Dilettantes.  _arXiv preprint arXiv:2105.02274_ (2021).
  * Mielke et al. (2020) Sabrina J Mielke, Arthur Szlam, Y-Lan Boureau, and Emily Dinan. 2020.  Linguistic calibration through metacognition: aligning dialogue agent responses with expected correctness.  _arXiv preprint arXiv:2012.14983_ (2020).
  * Mishra et al. (2021) Anshuman Mishra, Dhruvesh Patel, Aparna Vijayakumar, Xiang Lorraine Li, Pavan Kapanipathi, and Kartik Talamadupula. 2021.  Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 1322–1336.
  * Müller et al. (2020) Mathias Müller, Annette Rios, and Rico Sennrich. 2020\.  Domain Robustness in Neural Machine Translation. In _14th Conference of the Association for Machine Translation in the Americas_. Association for Machine Translation in the Americas, AMTA, 151–164.
  * Nakano et al. (2021) Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021\.  WebGPT: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ (2021).
  * Nan et al. (2021) Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. 2021\.  Entity-level Factual Consistency of Abstractive Text Summarization. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. 2727–2733.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset. In _CoCo@ NIPS_.
  * Nie et al. (2018) Feng Nie, Jinpeng Wang, Jin-Ge Yao, Rong Pan, and Chin-Yew Lin. 2018.  Operation-guided Neural Networks for High Fidelity Data-To-Text Generation. In _EMNLP_.
  * Nie et al. (2019) Feng Nie, Jin-Ge Yao, Jinpeng Wang, Rong Pan, and Chin-Yew Lin. 2019.  A Simple Recipe towards Reducing Hallucination in Neural Surface Realisation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 2673–2679.  https://doi.org/10.18653/v1/P19-1256
  * Och (2003) Franz Josef Och. 2003\.  Minimum Error Rate Training in Statistical Machine Translation. In _Proceedings of the 41st annual meeting of the Association for Computational Linguistics_. 160–167.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021\.  Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 4812–4829.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A Controlled Table-To-Text Generation Dataset. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 1173–1186.
  * Parthasarathi et al. (2021) Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, and Adina Williams. 2021.  Sometimes We Want Ungrammatical Translations. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 3205–3227.  https://doi.org/10.18653/v1/2021.findings-emnlp.275
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019\.  Language Models as Knowledge Bases?. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 2463–2473.  https://doi.org/10.18653/v1/D19-1250
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018\.  Hypothesis Only Baselines in Natural Language Inference. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_. 180–191.
  * Popat et al. (2016) Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2016.  Credibility Assessment of Textual Claims on the Web. In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_. 2173–2178.
  * Popat et al. (2018) Kashyap Popat, Subhabrata Mukherjee, Andrew Yates, and Gerhard Weikum. 2018.  DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Brussels, Belgium, 22–32.  https://doi.org/10.18653/v1/D18-1003
  * Post and Vilar (2018) Matt Post and David Vilar. 2018.  Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation.  _NAACL HLT 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ 1 (4 2018), 1314–1324.  https://doi.org/10.18653/v1/n18-1119
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019\.  Data-to-text generation with content selection and planning. In _Proceedings of the AAAI conference on artificial intelligence_.
  * Puduppully and Lapata (2021) Ratish Puduppully and Mirella Lapata. 2021.  Data-to-text generation with macro planning.  _Transactions of the Association for Computational Linguistics_ (2021).
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  _OpenAI Blog_ 1, 8 (2019), 9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.  http://jmlr.org/papers/v21/20-074.html
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence Level Training with Recurrent Neural Networks.  _ICLR_ (2016).
  * Rashkin et al. (2021) Hannah Rashkin, David Reitter, Gaurav Singh Tomar, and Dipanjan Das. 2021.  Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 704–718.  https://doi.org/10.18653/v1/2021.acl-long.58
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021\.  The Curious Case of Hallucinations in Neural Machine Translation.  (4 2021), 1172–1183.  https://arxiv.org/abs/2104.06683v1
  * Rebuffel et al. (2022) Clément Rebuffel, Marco Roberti, Laure Soulier, Geoffrey Scoutheeten, Rossella Cancelliere, and Patrick Gallinari. 2022\.  Controlling hallucinations at word level in data-to-text generation.  _Data Mining and Knowledge Discovery_ 36, 1 (2022), 318–354.
  * Rebuffel et al. (2021) Clément Rebuffel, Thomas Scialom, Laure Soulier, Benjamin Piwowarski, Sylvain Lamprier, Jacopo Staiano, Geoffrey Scoutheeten, and Patrick Gallinari. 2021.  Data-QuestEval: A Reference-less Metric for Data-to-Text Semantic Evaluation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Reiter (2018) Ehud Reiter. 2018\.  A Structured Review of the Validity of BLEU.  _Computational Linguistics_ 44, 3 (Sept. 2018), 393–401.  https://doi.org/10.1162/coli_a_00322
  * Reiter and Dale (1997) Ehud Reiter and Robert Dale. 1997.  Building Applied Natural Language Generation Systems.  _Natural Language Engineering_ 3, 1 (1997), 57–87.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020\.  How Much Knowledge Can You Pack Into the Parameters of a Language Model?. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 5418–5426.  https://doi.org/10.18653/v1/2020.emnlp-main.437
  * Rohrbach et al. (2018) Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018.  Object Hallucination in Image Captioning. In _EMNLP_.
  * Roller et al. (2020) Stephen Roller, Y-Lan Boureau, Jason Weston, Antoine Bordes, Emily Dinan, Angela Fan, David Gunning, Da Ju, Margaret Li, Spencer Poff, et al. 2020\.  Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions.  _arXiv preprint arXiv:2006.12442_ (2020).
  * Roller et al. (2021) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, et al. 2021\.  Recipes for Building an Open-Domain Chatbot. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. 300–325.
  * Sabet et al. (2020) Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020\.  SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. 1627–1643.
  * Santhanam et al. (2021) Sashank Santhanam, Behnam Hedayatnia, Spandana Gella, Aishwarya Padmakumar, Seokhwan Kim, Yang Liu, and Dilek Hakkani-Tur. 2021.  Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generation.  _arXiv preprint arXiv:2110.05456_ (2021).
  * Scialom et al. (2021) Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021\.  Questeval: Summarization asks for fact-based evaluation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017\.  Get To The Point: Summarization with Pointer-Generator Networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, 1073–1083.  https://doi.org/10.18653/v1/P17-1099
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning Robust Metrics for Text Generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 7881–7892.
  * Shen et al. (2021) Lei Shen, Haolan Zhan, Xin Shen, Hongshen Chen, Xiaofang Zhao, and Xiaodan Zhu. 2021\.  Identifying Untrustworthy Samples: Data Filtering for Open-domain Dialogues with Bayesian Optimization. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 1598–1608.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval Augmentation Reduces Hallucination in Conversation.  _EMNLP_ (2021).
  * Song et al. (2020b) Haoyu Song, Wei-Nan Zhang, Jingwen Hu, and Ting Liu. 2020b.  Generating Persona Consistent Dialogues by Exploiting Natural Language Inference.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34, 05 (Apr. 2020), 8878–8885.  https://doi.org/10.1609/aaai.v34i05.6417
  * Song et al. (2020a) Kaiqiang Song, Logan Lebanoff, Qipeng Guo, Xipeng Qiu, Xiangyang Xue, Chen Li, Dong Yu, and Fei Liu. 2020a.  Joint Parsing and Generation for Abstractive Summarization. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 8894–8901.
  * Song et al. (2019) Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019.  Code-Switching for Enhancing NMT with Pre-Specified Translation.  _NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ 1 (4 2019), 449–459.  https://arxiv.org/abs/1904.09107v4
  * Su et al. (2022) Dan Su, Xiaoguang Li, Jindi Zhang, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022.  Read before Generate! Faithful Long Form Question Answering with Machine Reading.  arXiv:2203.00343 [cs.CL]
  * Su et al. (2020) Hui Su, Xiaoyu Shen, Sanqiang Zhao, Zhou Xiao, Pengwei Hu, Cheng Niu, and Jie Zhou. 2020.  Diversifying Dialogue Generation with Non-Conversational Text. In _58th Annual Meeting of the Association for Computational Linguistics_. ACL, 7087–7097.
  * Su et al. (2021) Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021.  Plan-then-Generate: Controlled Data-to-Text Generation via Planning.  _Findings of EMNLP_ (2021).
  * Suadaa et al. (2021) Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. 2021\.  Towards Table-to-text Generation with Numerical Reasoning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 1451–1465.
  * Sun (2010) Yanli Sun. 2010\.  Mining the Correlation between Human and Automatic Evaluation at Sentence Level.. In _LREC_.
  * Susanto et al. (2020) Raymond Hendy Susanto, Shamil Chollampatt, and Liling Tan. 2020\.  Lexically Constrained Neural Machine Translation with Levenshtein Transformer.  (7 2020), 3536–3543.  https://doi.org/10.18653/V1/2020.ACL-MAIN.325
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014\.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ 27 (2014).
  * Tang et al. (2021) Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. 2021\.  CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning.  _arXiv preprint arXiv:2112.08713_ (2021).
  * Thawani et al. (2021) Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021.  Representing Numbers in NLP: a Survey and a Vision. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 644–656.
  * Thomson and Reiter (2020) Craig Thomson and Ehud Reiter. 2020.  A Gold Standard Methodology for Evaluating Accuracy in Data-To-Text Systems. In _Proceedings of the 13th International Conference on Natural Language Generation_. 158–168.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a Large-scale Dataset for Fact Extraction and VERification. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 809–819.  https://doi.org/10.18653/v1/N18-1074
  * Thorne et al. (2019) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019.  Evaluating adversarial attacks against multiple fact verification systems. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, 2944–2953.  https://doi.org/10.18653/v1/D19-1292
  * Tian et al. (2020) Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P. Parikh. 2020\.  Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation.  arXiv:1910.08684 [cs.CL]
  * Tran and Nguyen (2017) Van-Khanh Tran and Le-Minh Nguyen. 2017.  Natural Language Generation for Spoken Dialogue System using RNN Encoder-Decoder Networks. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_. Association for Computational Linguistics, Vancouver, Canada, 442–451.  https://doi.org/10.18653/v1/K17-1044
  * Tu et al. (2017) Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017.  Neural Machine Translation with Reconstruction. In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.  Modeling Coverage for Neural Machine Translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 76–85.
  * Uc-Cetina et al. (2021) Victor Uc-Cetina, Nicolas Navarro-Guerrero, Anabel Martin-Gonzalez, Cornelius Weber, and Stefan Wermter. 2021\.  Survey on reinforcement learning for language processing.  _arXiv preprint arXiv:2104.05565_ (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. 5998–6008.  https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015.  A Neural Conversational Model.  _ICML Deep Learning Workshop_ (2015).
  * Wang et al. (2020a) Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a.  Asking and Answering Questions to Evaluate the Factual Consistency of Summaries.  _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ (2020).
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang and Sennrich (2020) Chaojun Wang and Rico Sennrich. 2020.  On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation. In _2020 Annual Conference of the Association for Computational Linguistics_. Association for Computational Linguistics (ACL), 3544–3552.
  * Wang (2019) Hongmin Wang. 2019\.  Revisiting Challenges in Data-to-Text Generation with Fact Grounding. In _Proceedings of the 12th International Conference on Natural Language Generation_. 311–322.
  * Wang et al. (2021a) Peng Wang, Junyang Lin, An Yang, Chang Zhou, Yichang Zhang, Jingren Zhou, and Hongxia Yang. 2021a.  Sketch and Refine: Towards Faithful and Informative Table-to-Text Generation.  _ACL_ (2021).
  * Wang et al. (2022b) Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b.  OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework.  _CoRR_ abs/2202.03052 (2022).
  * Wang et al. (2022a) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. 2022a.  Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks.  _ArXiv_ abs/2208.10442 (2022).
  * Wang et al. (2021b) Xu Wang, Hainan Zhang, Shuai Zhao, Yanyan Zou, Hongshen Chen, Zhuoye Ding, Bo Cheng, and Yanyan Lan. 2021b.  FCM: A Fine-grained Comparison Model for Multi-turn Dialogue Reasoning.  _EMNLP Findings_ (2021).
  * Wang et al. (2020b) Zhenyi Wang, Xiaoyang Wang, Bang An, Dong Yu, and Changyou Chen. 2020b.  Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraints. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 1072–1086.
  * Wang et al. (2022c) Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022c.  SimVLM: Simple Visual Language Model Pretraining with Weak Supervision.  _ArXiv_ abs/2108.10904 (2022).
  * Welleck et al. (2019a) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019a.  Neural Text Generation With Unlikelihood Training. In _International Conference on Learning Representations_.
  * Welleck et al. (2019b) Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019b.  Dialogue Natural Language Inference. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 3731–3741.  https://doi.org/10.18653/v1/P19-1363
  * Wen et al. (2015a) Tsung-Hsien Wen, Milica Gašić, Dongho Kim, Nikola Mrkšić, Pei-Hao Su, David Vandyke, and Steve Young. 2015a.  Stochastic Language Generation in Dialogue using Recurrent Neural Networks with Convolutional Sentence Reranking. In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_. Association for Computational Linguistics, Prague, Czech Republic, 275–284.  https://doi.org/10.18653/v1/W15-4639
  * Wen et al. (2015b) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-hao Su, David Vandyke, and Steve J Young. 2015b.  Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems. In _EMNLP_.
  * Weng et al. (2020) Rongxiang Weng, Heng Yu, Xiangpeng Wei, and Weihua Luo. 2020\.  Towards Enhancing Faithfulness for Neural Machine Translation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 2675–2684.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018\.  A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 1112–1122.  https://doi.org/10.18653/v1/N18-1101
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017\.  Challenges in Data-to-Document Generation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Copenhagen, Denmark, 2253–2263.  https://doi.org/10.18653/v1/D17-1239
  * Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019\.  TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents.  _CoRR_ abs/1901.08149 (2019).  arXiv:1901.08149 http://arxiv.org/abs/1901.08149
  * Wu et al. (2019) Chien-Sheng Wu, Richard Socher, and Caiming Xiong. 2019\.  Global-to-local Memory Pointer Networks for Task-Oriented Dialogue. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=ryxnHhRqFm
  * Wu et al. (2021) Zeqiu Wu, Michel Galley, Chris Brockett, Yizhe Zhang, Xiang Gao, Chris Quirk, Rik Koncel-Kedziorski, Jianfeng Gao, Hannaneh Hajishirzi, Mari Ostendorf, et al. 2021\.  A Controllable Model of Grounded Response Generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 14085–14093.
  * Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021.  On Hallucination and Predictive Uncertainty in Conditional Language Generation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. 2734–2744.
  * Xu et al. (2021c) Jing Xu, Arthur D. Szlam, and Jason Weston. 2021c.  Beyond Goldfish Memory: Long-Term Open-Domain Conversation.  _ArXiv_ abs/2107.07567 (2021).
  * Xu and Carpuat (2020) Weijia Xu and Marine Carpuat. 2020.  EDITOR: an Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints.  _Transactions of the Association for Computational Linguistics_ 9 (11 2020), 311–328.  https://doi.org/10.1162/tacl_a_00368d3/2021.
  * Xu and Carpuat (2021) Weijia Xu and Marine Carpuat. 2021.  Rule-based Morphological Inflection Improves Neural Terminology Translation.  (9 2021), 5902–5914.  https://doi.org/10.18653/v1/2021.emnlp-main.477
  * Xu et al. (2019) Weijia Xu, Xing Niu, and Marine Carpuat. 2019.  Differentiable Sampling with Flexible Reference Word Order for Neural Machine Translation.  _NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference_ 1 (4 2019), 2047–2053.  https://doi.org/10.18653/v1/n19-1207
  * Xu et al. (2021a) Xinnuo Xu, Ondřej Dušek, Verena Rieser, and Ioannis Konstas. 2021a.  AGGGEN: Ordering and Aggregating while Generating.  _roceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL2021)_ (2021).
  * Xu et al. (2021b) Yan Xu, Etsuko Ishii, Samuel Cahyawijaya, Zihan Liu, Genta Indra Winata, Andrea Madotto, Dan Su, and Pascale Fung. 2021b.  Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters.  _arXiv preprint arXiv:2105.06232_ (2021).
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018\.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 2369–2380.
  * Yavuz et al. (2019) Semih Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tur. 2019.  DEEPCOPY: Grounded Response Generation with Hierarchical Pointer Networks. In _Proceedings of SIGdial_.
  * Yin et al. (2016) Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016\.  Neural generative question answering. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_. 2972–2978.
  * Yoneda et al. (2018) Takuma Yoneda, Jeff Mitchell, Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018\.  UCL Machine Reading Group: Four Factor Framework For Fact Finding (HexaF). In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_. Association for Computational Linguistics, Brussels, Belgium, 97–102.  https://doi.org/10.18653/v1/W18-5515
  * Yu et al. (2021) Tiezheng Yu, Zihan Liu, and Pascale Fung. 2021.  AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 5892–5904.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big Bird: Transformers for Longer Sequences. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 17283–17297.  https://proceedings.neurips.cc/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf
  * Zemlyanskiy and Sha (2018) Yury Zemlyanskiy and Fei Sha. 2018.  Aiming to Know You Better Perhaps Makes Me a More Engaging Dialogue Partner. In _Proceedings of the 22nd Conference on Computational Natural Language Learning_. Association for Computational Linguistics, Brussels, Belgium, 551–561.  https://doi.org/10.18653/v1/K18-1053
  * Zhang et al. (2021a) Chen Zhang, Grandee Lee, Luis Fernando D’Haro, and Haizhou Li. 2021a.  D-score: Holistic dialogue evaluation without reference.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 2502–2516.
  * Zhang et al. (2019b) Hongguang Zhang, Jing Zhang, and Piotr Koniusz. 2019b.  Few-Shot Learning via Saliency-Guided Hallucination of Samples. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 2770–2779.  https://doi.org/10.1109/CVPR.2019.00288
  * Zhang et al. (2021b) Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, and Yang Liu. 2021b.  Neural Machine Translation with Explicit Phrase Alignment.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ 29 (2021), 1001–1010.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018\.  Personalizing Dialogue Agents: I have a dog, do you have pets too?. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 2204–2213.
  * Zhang et al. (2019a) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019a.  BERTScore: Evaluating Text Generation with BERT. In _International Conference on Learning Representations_.
  * Zhang et al. (2020b) Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020b.  Do Language Embeddings capture Scales?. In _Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_. 292–299.
  * Zhang et al. (2020a) Yuhao Zhang, Derek Merck, Emily Tsai, Christopher D Manning, and Curtis Langlotz. 2020a.  Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 5108–5120.
  * Zhang et al. (2020c) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020c.  DIALOGPT : Large-Scale Generative Pre-training for Conversational Response Generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 270–278.  https://doi.org/10.18653/v1/2020.acl-demos.30
  * Zhao et al. (2021) Jing Zhao, Junwei Bao, Yifan Wang, Yongwei Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2021.  RoR: Read-over-Read for Long Document Machine Reading Comprehension. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 1862–1872.  https://aclanthology.org/2021.findings-emnlp.160
  * Zhao et al. (2020) Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020\.  Reducing Quantity Hallucinations in Abstractive Summarization. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 2237–2249.  https://doi.org/10.18653/v1/2020.findings-emnlp.203
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. 2021\.  QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 5905–5921.
  * Zhou et al. (2019) Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. 2019\.  Density Matching for Bilingual Word Embedding. In _NAACL_.
  * Zhou et al. (2021b) Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021b.  Detecting Hallucinated Content in Conditional Neural Sequence Generation. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. 1393–1404.
  * Zhou et al. (2018) Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018\.  A Dataset for Document Grounded Conversations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. 708–713.
  * Zhou et al. (2021a) Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. 2021a.  Think Before You Speak: Using Self-talk to Generate Implicit Commonsense Knowledge for Response Generation.  _arXiv preprint arXiv:2110.08501_ (2021).
  * Zhu et al. (2021) Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2021.  Enhancing Factual Consistency of Abstractive Summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 718–733.