# Faithfulness in Natural Language Generation: A Systematic Survey of
Analysis, Evaluation and Optimization Methods

Wei Li1, Wenhao Wu2, Moye Chen1, Jiachen Liu1, Xinyan Xiao1, Hua Wu1
1Baidu Inc., Beijing, China
2Key Laboratory of Computational Linguistics, MOE, Peking University
<EMAIL_ADDRESS>

###### Abstract

Natural Language Generation (NLG) has made great progress in recent years due
to the development of deep learning techniques such as pre-trained language
models. This advancement has resulted in more fluent, coherent and even
properties controllable (e.g. stylistic, sentiment, length etc.) generation,
naturally leading to development in downstream tasks such as abstractive
summarization, dialogue generation, machine translation, and data-to-text
generation. However, the faithfulness problem that the generated text usually
contains unfaithful or non-factual information has become the biggest
challenge, which makes the performance of text generation unsatisfactory for
practical applications in many real-world scenarios. Many studies on analysis,
evaluation, and optimization methods for faithfulness problems have been
proposed for various tasks, but have not been organized, compared and
discussed in a combined manner. In this survey, we provide a systematic
overview of the research progress on the faithfulness problem of NLG,
including problem analysis, evaluation metrics and optimization methods. We
organize the evaluation and optimization methods for different tasks into a
unified taxonomy to facilitate comparison and learning across tasks. Several
research trends are discussed further.

Figure 1: Four aspects of the NLG challenge. Faithfulness has become the
biggest challenge in modern natural language generation.

###### Contents

  1. 1 Introduction
    1. 1.1 Developing of NLG
    2. 1.2 The Faithfulness Problem
    3. 1.3 Structure of This Survey
  2. 2 Problem Analysis
    1. 2.1 Definition and Categorization
    2. 2.2 Challenges and Issues
    3. 2.3 Cause Analysis
  3. 3 Automatic Evaluation Metrics
    1. 3.1 Entailment-based Metrics
    2. 3.2 QA-based Metrics
    3. 3.3 Fact-based Metrics
      1. 3.3.1 Entity-based
      2. 3.3.2 Ngram-based
      3. 3.3.3 Relation-based
    4. 3.4 Other Metrics
    5. 3.5 Meta Evaluation
  4. 4 Optimization Methods
    1. 4.1 Faithfulness in Abstractive Summarization
      1. 4.1.1 Factual Guidance
      2. 4.1.2 Auxiliary Tasks
      3. 4.1.3 Learning Methods
      4. 4.1.4 Post-Editing
      5. 4.1.5 Constrained Decoding
      6. 4.1.6 Other Methods
    2. 4.2 Faithfulness in Dialogue Generation
      1. 4.2.1 Factual Guidance
      2. 4.2.2 Auxiliary Tasks
      3. 4.2.3 Learning Methods
      4. 4.2.4 Constrained Decoding
      5. 4.2.5 Post-Editing
      6. 4.2.6 Other Methods
    3. 4.3 Faithfulness in Machine Translation
      1. 4.3.1 Auxiliary Tasks
      2. 4.3.2 Learning Methods
      3. 4.3.3 Constrained Decoding
      4. 4.3.4 Other Methods
    4. 4.4 Faithfulness in Data-to-Text Generation
      1. 4.4.1 Factual Guidance
      2. 4.4.2 Auxilary Tasks
      3. 4.4.3 Learning methods
      4. 4.4.4 Constrained Decoding
      5. 4.4.5 Other Methods
    5. 4.5 Faithfulness in Other NLG Tasks
      1. 4.5.1 Factual Language Model
      2. 4.5.2 Factuality Detection
      3. 4.5.3 Constrained Text Generation
      4. 4.5.4 Image Caption
  5. 5 Discussion
    1. 5.1 Fine-grained and General Evaluation
    2. 5.2 Reasoning-based Optimization
  6. 6 Conclusion

##  1 Introduction

Natural Language Generation (NLG) is the process of producing a natural
language text from a textual or non-textual input in order to meet specified
communicative goals (Gatt and Krahmer, 2018). The input of NLG varies with
different task settings, however, the output is always readable natural
language text. According to the type of input, the tasks of NLG can be mainly
categorized into: text-to-text generation, data-to-text generation, and
multimodality-to-text generation.

The text-to-text generation tasks take existing texts as input, and
automatically produce a new, coherent text as output. The most common
applications include: text summarization (Allahyari et al., 2017), dialogue
generation (Li et al., 2016b), machine translation (Koehn, 2009), question
generation (Du et al., 2017), paraphrase generation (Li et al., 2017) etc. The
data-to-text generation tasks automatically generate text from numerical or
structured data such as table, key-value lists, and tuples. The example
applications include: table-to-text generation (Liu et al., 2018b), KG-to-text
generation (Ke et al., 2021), meaning-to-text generation (e.g. AMR-to-text)
(Song et al., 2018) etc. The multimodality-to-text generation tasks transfer
the semantics in multimodal input such as images or videos, into natural
language texts. Typical tasks include image caption (Vinyals et al., 2015),
visual storytelling (Huang et al., 2016), and video summarization (Ma et al.,
2002).

From the perspective of input-output information transformation, the tasks of
NLG can be divided into open-ended language generation and non-open-ended
language generation. Open-ended language generation tasks refer to tasks that
the input is incomplete and the output semantics are not contained by the
input. For example, story generation is a classical open-ended language
generation task, which tends to generate a complete story based on some
leading sentences or keywords. Obviously, the model needs to create new
information to completing storyline planning and generating meaningful
stories. One of the greatest characteristics of the open-ended language
generation tasks is that the information mapping between input and output is
usually one-to-many. The same input can produce many outputs with different
meanings.

By contrast, for non-open-ended language generation tasks, the input usually
provides complete or even more information for the output. Machine translation
is one typical non-open-ended language generation task where the input
provides complete semantics for the output. Paraphrase generation can be
regarded as an equivalent transformation of information, where the input and
output semantics are exactly the same, but the language expression is
different. In text summarization, input usually provides more information than
output, so the summarization model needs to select salient information to
produce summary output.

Table 1 lists some common NLG tasks as well as their characteristics.

Table 1: Categories of common natural language generation tasks. Tasks | Category | Information Mapping
---|---|---
Text Summarization | Text-to-Text | Non-open-ended
Machine Translation | Text-to-Text | Non-open-ended
Sentence Simplication | Text-to-Text | Non-open-ended
Paraphrase Generation | Text-to-Text | Non-open-ended
Dialogue Generation | Text-to-Text | Open-ended
Question Generation | Text-to-Text | Non-open-ended
Story Generation | Text-to-Text | Open-ended
Essay Generation | Text-to-Text | Open-ended
News Generation | Text-to-Text | Open-ended
Poetry Generation | Text-to-Text | Open-ended
Table-to-Text Generation | Data-to-Text | Non-open-ended
AMR-to-Text Generation | Data-to-Text | Non-open-ended
Image Caption | Multimodality-to-Text | Non-open-ended
Video Caption | Multimodality-to-Text | Non-open-ended
Visual Storytelling | Multimodality-to-Text | Open-ended

###  1.1 Developing of NLG

Table 2: Four paradigms in natural language generation. Paradigm |  Engineering |  Main Problems
---|---|---
Template-based |  Manual Rules (Content Planning, Sentence Planning, Text Realization) |  Fluency, Informativeness
Statistical-based |  Statistic Language Model (e.g. N-gram, Smoothing, Perplexity) |  Fluency, Informativeness
Neural-based |  Neural Architecture (e.g. RNN, LSTM, CNN, Transformer) |  Controllability, Faithfulness
Pretraining-based |  Pretraining Objectives (e.g. BERT, T5, GPT3, BART, CTRL) |  Faithfulness

The research on NLG has a long history, starting from 1950s. The developing of
NLG approaches can be mainly divided into four stages: template-based,
statistical-based, neural-based and pretraining-based, as shown in Table 2.

  * •

Template-based. The earliest natural language generation system adopted the
method of rules and templates to design different modules for text generation,
which reflected the linguistic knowledge of vocabulary, grammar, syntax and
even pragmatics designed by many experts. They usually consists of several
different components including content planning, sentence planning and text
realization, each performing a specified function.

  * •

Statistical-based. Statistical language model further proposes a new idea of
language modeling from the perspective of probability and statistics, which
encodes the dependency between vocabulary and context in conditional
probability. N-gram language model is the most popular statistical language
model, which is usually coupled with template-based methods for re-ranking and
selecting fluent generated texts.

  * •

Neural-based. With the development of deep learning, the neural-based end-to-
end methods have gradually occupied a dominant position, which can better
model the statistical co-occurrence relationship between vocabulary and
context through end-to-end training, thus significantly improves the
performance of text generation. Various neural architectures have been
explored for NLG, such as Recurrent Neural Network (RNN) (Graves, 2013;
Zaremba et al., 2014), Convolutional Neural Network (CNN) (Kalchbrenner et
al., 2014) and self-attention Transformer network (Vaswani et al., 2017).

  * •

Pretraining-based. Most recently, the pre-trained language generation model
based on the Transformer architecture can better capture the linguistic
knowledge of vocabulary, syntax and semantics, which greatly promotes the
development of natural language generation. The rise of pre-trained language
models (Devlin et al., 2018; Liu et al., 2019c; Brown et al., 2020) has led to
strong text generation models for applications including text summarization
(Zhang et al., 2020b; Liu and Lapata, 2019; Dong et al., 2019), dialogue
generation (Zhang et al., 2019; Bao et al., 2020), data-to-text generation
(Chen et al., 2020b), and machine translation (Liu et al., 2020). However,
while these models generate fluent and grammatical text, they are prone to
making factual errors that contradict the input text (Cao et al., 2017).

##### Challenges and Issues

NLG faces four main challenges and issues at different development stages:
fluency, informativeness, controllability and faithfulness, as shown in Figure
1.

  * •

The fluency problem refers to whether the generated text is fluent,
grammatical, and coherent.

  * •

The informativeness problem refers to that the model generates redundant,
meaningless, and general content, and the generated text is significantly
insufficient in informativeness, diversity, and specificity.

  * •

The controllability problem means that the generated text cannot satisfy the
pre-given constraints, such as text style, attributes and content.

  * •

The faithfulness problem means that the generated content is inconsistent with
the input information, has hallucinations or non-factual information.

Traditional template-based methods can usually generate reliable and faithful
texts, but limited by the diversity and generality of rules, the generated
texts usually face the problems of fluency and informativeness. Benefiting
from end-to-end training on large corpus, the neural-based methods can
generate fluent and informative texts. However, due to the introduction of the
probability sampling mechanism, they need to sample from the probability
distribution estimated by the model each time. Considering that the vocabulary
is very large, generally in the order of $1000\sim 50000$, the probability
distribution inevitably contains a large number of long-tail words with low
probability of occurrence, coupled with the randomness of probability sampling
itself, the controllability and faithfulness of the neural-based NLG model is
particularly serious. In the pre-training era, through self-supervised
training on large-scale unlabeled corpora, the model generated text is
outstanding in terms of fluency, informativeness and even controllability, but
it still cannot solve the faithfulness problem.

###  1.2 The Faithfulness Problem

The faithfulness problem has become the biggest challenge in NLG, which
largely limits the applicability of NLG algorithms in practical scenarios. For
example, the researches on abstractive text summarization show that about 30%
of summaries generated by state-of-the-art models have faithfulness issues
(Cao et al., 2017; Pagnoni et al., 2021; Kryściński et al., 2019; Falke et
al., 2019). This brings serious problems to the credibility and usability of
abstractive summarization systems.

Table 3: Examples of unfaithful errors for several common NLG tasks. Red color denotes factual errors. Tasks |  Source |  Output
---|---|---
Abstractive Summarization |  The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started. |  The first vaccine for Ebola was rejected in 2019. Scientists say a vaccine for Ebola is unlikely to be ready this year.
Dialogue Generation |  Persona: I have two cats. I work as a teacher at a middle school. My favorite color is yellow. I dislike country music. Dialogue: hi, do you have any pets? |  I do not have any pets. Do you play any sports ?
Machine Translation |  迈克周四去书店。(Michael goes to the bookstore on Thursday.) |  Mike happily goes to the bookstore on Thursday with his friend.
Table-to-Text Generation |  Name: Frank Lino; Caption: FBI surveillance photo; Birth date: October 30, 1938; Birth place: Gravesend, Brooklyn, New York, United States; |  Frank Lino (born October 30, 1938 in Brooklyn) is an American criminal defense attorney.

The faithfulness problem is widely existed in nearly all NLG tasks, such as
text summarization, dialogue generation, machine translation and table-to-text
generation. Unfaithful examples of several popular tasks are shown in Table 3.
For the non-open-ended NLG tasks and open-ended NLG tasks, the definitions of
faithfulness problem are different.

##### Faithfulness in Non-open-ended NLG

For non-open NLG tasks, NLG models generate text based on the input which
provides complete or even more information for the output text. The
faithfulness problem in non-open NLG tasks is defined as whether the generated
content is factually consistent with the input information, often referred to
as factual consistency. For example, faithfulness in text summarization is
whether the generated summary is factually consistent with and faithful to the
input document. If the summary has hallucinations that not contained by the
input document, then it’s unfaithful to the input document. Similarly,
faithfulness in machine translation is whether the translation is consistent
with and faithful to the original language.

##### Faithfulness in Open-ended NLG

For Open-ended NLG tasks, the model needs to leverage knowledge in Knowledge
Graph or corpus to create new content that not contained by the input. The
faithfulness problem in open-ended NLG tasks is defined as whether the
generated content is factually consistent with the world knowledge or
commonsense, often referred to as factual correctness. For example, the
faithfulness in news article generation is whether the facts in the generated
article actually exists or happen in the real world. One relevant research
topic is fake news detection (Shu et al., 2017; Zhang and Ghorbani, 2020).

To address the faithfulness problem, a lot of automatic faithfulness
evaluation metrics and meta evaluations for these metrics have been proposed.
Besides, much effort has been devoted to optimizing faithfulness for different
NLG tasks. The framework of existing researches is demonstrated in Figure 2.
Since the research on faithfulness mainly focuses on non-open-ended tasks,
such as text summarization, machine translation, knowledge-grounded dialogue
generation and data-to-text generation, this paper mainly studies the
faithfulness (i.e. factual consistency) in non-open-ended tasks. We conduct a
comprehensive survey of existing researches on faithfulness, including problem
analysis, evaluation metrics, and optimization approaches.

Figure 2: The research framework on the faithfulness problem.

forked edges, for tree= grow’=0, draw, align=c, font=, rounded corners ,
[Faithfulness, for tree=fill=black!10,draw=black [Problem
Analysis 2, for tree=fill=purple!10,draw=purple,text width=60 [Definition&
Categorization [FRANK[132], NPH[39], etc., for
tree=fill=purple!10,draw=purple!10,text width=120] ] [Challenges
&Issues [Model Analysis [132; 18], etc., for
tree=fill=purple!10,draw=purple!10,text width=120] [Evaluation Problem [86;
43], for tree=fill=purple!10,draw=purple!10,text width=120] [Annotation
Problem [45; 132], for tree=fill=purple!10,draw=purple!10,text width=120] ]
[Cause Analysis [Data Divergence [117; 32; 184], for
tree=fill=purple!10,draw=purple!10,text width=120] [Exposure Bais [171; 117],
etc., for tree=fill=purple!10,draw=purple!10,text width=120 ] [Poor
Representation [114], etc., for tree=fill=purple!10,draw=purple!10,text
width=120 ] ] ] [Evaluation
Metrics 3, for tree=fill=orange!10,draw=orange [Meta Evaluation, for tree=text
width=75 [FRANK[132], SUMMAc[88], BEGIN[40], etc., for
tree=fill=orange!10,draw=orange!10, text width=180] ] [NLI-based Metrics, for
tree=text width=75 [DAE[60], FactCC[65], DialogNLI[178], etc., for
tree=fill=orange!10,draw=orange!10, text width=180] ] [QA-based Metrics, for
tree=text width=75 [QAGS[170], FEQA[65], $Q^{2}$[69], QUALS[127] etc., for
tree=fill=orange!10,draw=orange!10, text width=180] ] [Fact-based Metrics, for
tree=text width=75 [Entity, for tree=fill=orange!10,draw=orange, text width=40
[EntityAlign[128], SimAlign[149], for tree=fill=orange!10,draw=orange!10, text
width=125] ] [N-gram, for tree=fill=orange!10,draw=orange, text width=40
[PARENT[32], PARENT-T[175], for tree=fill=orange!10,draw=orange!10, text
width=125] ] [Relation, for tree=fill=orange!10,draw=orange, text width=40
[TripleAlign[65], ArcsAlign[60], for tree=fill=orange!10,draw=orange!10, text
width=125] ] ] [Other Metrics, for tree=text width=75 [BARTScore[190],
COCO[187], TokenCLS[205], for tree=fill=orange!10,draw=orange!10, text
width=185] ] ] [Optimization
Methods 4, for tree=fill=blue!10,draw=blue [Factual
Guidance, for tree=text width=60 [Abstractive
Summarization, for tree=text width=60 [Keyword[36], Sentence[160],
Relation[19], for tree=fill=blue!10,draw=blue!10, text width=160] ] [Dialogue
Generation, for tree=text width=60 [Inplicit[95], Extractive[57],
Retrived[33], for tree=fill=blue!10,draw=blue!10, text width=160] ] [Data-to-
Text
Generation, for tree=text width=65 [SANA [173], Segment[152], Entity [107],
for tree=fill=blue!10,draw=blue!10, text width=160] ] ] [Auxilary Tasks , for
tree=fill=yellow!10,draw=yellow [Abstractive
Summarization, for tree=text width=60 [Entailment[94; 45], QA[127],
Others[200], for tree=fill=yellow!10,draw=yellow!10, text width=160] ]
[Dialogue
Generation, for tree=text width=60 [NLI-reranking [178], NLI-RL [156; 119],
for tree=fill=yellow!10,draw=yellow!10, text width=160] ] [Data-to-Text
Generation, for tree=text width=65 [EntityMatching [175], FocusAttn[106], for
tree=fill=yellow!10,draw=yellow!10, text width=160] ] [Machine
Translation, for tree=text width=65 [FENMT [181], WordAlignment[194; 46; 54],
for tree=fill=yellow!10,draw=yellow!10, text width=160] ] ] [Post-Editing ,
for tree=fill=green!10,draw=green [Abstractive
Summarization, for tree=text width=65 [SpanFact[35], FactCorrect[16],
ContrastSel[23], for tree=fill=green!10,draw=green!10, text width=160] ]
[Dialogue
Generation, for tree=text width=65 [GDR [157], NeuralPathHunter [39], for
tree=fill=green!10,draw=green!10, text width=160] ] ] ] ]

forked edges, for tree= grow’=0, draw, align=c, font=, rounded corners ,
highlight/.style= thick, font= [Optimization
Methods 4, for tree=fill=blue!10,draw=blue,text width=70 [Learning
Methods , for tree=fill=blue!10,draw=blue [Abstractive
Summarization, for tree=text width=60 [CLIFF[18], CO2Sum[65], CLAPS[91], for
tree=fill=blue!10,draw=blue!10, text width=160] ] [Dialogue
Generation, for tree=text width=60 [Unlikelihood learning [97; 148], for
tree=fill=blue!10,draw=blue!10, text width=160] ] [Machine
Translation, for tree=text width=65 [MRT [171], Adversarial[11; 26; 11; 79;
27], for tree=fill=blue!10,draw=blue!10, text width=160] ] ] [Constrained
Decoding , for tree=fill=pink!10,draw=pink [Abstractive
Summarization, for tree=text width=60 [CAS[116], FocusAttn[3], POINTER[199],
for tree=fill=pink!10,draw=pink!10, text width=160] ] [Dialogue
Generation, for tree=text width=60 [ Balakrishnan et al. [6], Nye et al.
[131], for tree=fill=pink!10,draw=pink!10, text width=160] ] [Machine
Translation, for tree=text width=65 [GBS [67], CBS[2], DBA[135], for
tree=fill=pink!10,draw=pink!10, text width=110] ] [Data-to-Text
Generation, for tree=text width=65 [ConfidentDecoding [163], for
tree=fill=pink!10,draw=pink!10, text width=110] ] ] [Other Methods , for
tree=fill=purple!10,draw=purple [Abstractive
Summarization, for tree=text width=60 [MoFE [28], HERMAN[202], ENTFA[17], for
tree=fill=purple!10,draw=purple!10, text width=160] ] [Machine
Translation, for tree=text width=60 [Feng et al. [46, Weng et al. [180], Tu et
al. [164], for tree=fill=purple!10,draw=purple!10, text width=160] ] [Data-to-
text
Generation, for tree=text width=60 [Nie et al. [129], Wang [172], for
tree=fill=purple!10,draw=purple!10, text width=160] ] [Dialogue
Generation, for tree=text width=60 [Kim et al. [82], for
tree=fill=purple!10,draw=purple!10, text width=160] ] ] ]

Figure 3: The content typology of the survey.

###  1.3 Structure of This Survey

The content typology of this survey is shown in Figure 3.

In Section 2, we give a systematic analysis on the faithfulness problem in
NLG, including categorization of unfaithful errors, manual annotations,
challenges for evaluating and optimizing faithfulness, cause analysis, and
relations with other aspects.

In Section 3, we organize the various evaluation metrics proposed for
faithfulness evaluation, and combine the meta-evaluations for these metrics to
facilitate future research on faithfulness evaluations.

In Section 4, we summarize different optimization methods from both the
perspective of tasks and methodology, and detail their relative advantages.

##  2 Problem Analysis

In general, the task of natural language generation (NLG) targets at finding
an optimal sequence $y_{<T+1}=(y_{1},y_{2},\ldots,y_{T})$ that satisfies:

$\displaystyle y_{<T+1}=$
$\displaystyle\mathop{\arg\max}_{y_{<T+1}\in\mathcal{S}}logP_{\theta}(y_{<T+1}|x)$
(1)
$\displaystyle\mathop{\arg\max}_{y_{<T+1}\in\mathcal{S}}\sum_{t=1}^{T}logP_{\theta}(y_{t}|y_{<t},x)$

where $T$ represents the number of tokens of the generated sequence,
$\mathcal{S}$ represents a set containing all possible sequences, and
$P_{\theta}(y_{t}|y_{<t},x)$ is the conditional probability of the next token
$y_{t}$ based on its previous tokens $y_{<t}=(y_{1},y_{2},\ldots,y_{t-1})$ and
the source sequence $x$ with model parameters $\theta$.

###  2.1 Definition and Categorization

We define the output sequence as being unfaithful if it has a span
$y_{i},...,y_{j}$, that is not supported by the input sequence $x$. The
faithfulness issues, i.e. factual inconsistent with source sequence, can be
divided into two categories:

  * •

Intrinsic Error: the fact that is contradicted to the source sequence $x$ due
to synthesizing content using information present in $x$, which is also
referred to “intrinsic hallucination” in Maynez et al. (2020).

  * •

Extrinsic Error: the fact that is neither supported nor contradicted by the
source, which is also referred to “extrinsic hallucination” in Maynez et al.
(2020).

Frank (Pagnoni et al., 2021) defines a fine-grained typology of factual errors
for text summarization, which is theoretically grounded in frame semantics
(Fillmore et al., 1976; Palmer et al., 2005) and linguistic discourse analysis
(Brown et al., 1983). It can also be applied to other non-open-ended NLG
tasks, such as dialogue generation, machine translation and table-to-text
generation.

Table 4: Hierarchical typology of unfaithful errors. Examples of text summarization are shown. The source input is the same as the first line in Table 3. Categorization |  Examples
---|---
Intrinsic Error | Semantic Frame Errors |  Predicate Error (PredE) |  The Ebola vaccine was rejected by the FDA in 2019.
Entity Error (EntE) |  Scientists say a vaccine for Ebola is unlikely to be ready this year.
Circumstance Error (CircE) |  The first vaccine for Ebola was approved by the FDA in 2014.
Discourse Errors |  Co-reference Error (CorefE) |  The first vaccine for Ebola was approved in 2019. They say a vaccine for COVID-19 is unlikely to be ready this year.
Discourse Link Error (LinkE) |  To produce the vaccine, scientists have to show successful human trials, then sequence the DNA of the virus.
Extrinsic Error | Factual |  China has already started clinical trials of the COVID-19 vaccine.
Non-Factual |  China didn’t start clinical trials of the COVID-19 vaccine.

The fine-grained categories of factual errors mainly include:

  1. 1.

Semantic Frame Errors capture factual errors in frame semantic and its core
and non-core frame elements, including:

     * •

Predicate Error (PredE) denotes errors where the predicate is inconsistent
with the source text;

     * •

Entity Error (EntE) denotes errors where the primary arguments (like entities)
of the predicate are wrong or have the wrong attributes;

     * •

Circumstance Error (CircE) denotes errors where the arguments and predicates
interact (e.g. location, time, manner, direction, modality) are wrong.

  2. 2.

Discourse Errors capture erroneous links between discourse segments,
including:

     * •

Co-reference Error (CorefE) denotes errors where pronouns and other types of
references to previously mentioned entities either are incorrect or have no
clear antecedents;

     * •

Discourse Link Error (LinkE) denotes incorrect discourse link between
different statements.

  3. 3.

Content Verifiable Errors capture erroneous information that cannot be
verified against the source text, which are mainly caused by:

     * •

Out of Article Error (OutE) denotes information that cannot be deduced by from
the original text (the same as extrinsic hallucinations (Maynez et al.,
2020));

     * •

Grammatical Error (GramE) denotes not well formed statements that make their
meaning incomprehensible or ambiguous and cannot be verified against the
source.

Despite all extrinsic errors are assumed incorrect, Cao et al. (2021) and
Maynez et al. (2020) find that much hallucinated content is factual, namely
consistent with world knowledge. Factual hallucinations refer to content that
is verifiable by world knowledge but not inferable from source text. For
example, in text summarization, they find that more than half of the
hallucinated entities are factual with respect to the source document and
world knowledge. These factual hallucinations can be beneficial in a summary
by providing useful background information. Thus, the extrinsic errors or
OutEs can be further categorized into factual hallucinations and non-factual
hallucinations.

Combining these definitions and categorization, we define a more thorough
hierarchical typology of the faithfulness problems, as shown in Table 4. This
typology provides us with the means to categorize the types of errors made by
generation models, helping us gain deeper insights than simply categorizing
content as faithful or unfaithful.

###  2.2 Challenges and Issues

##### Model Analysis

A lot of researches make annotations with different granularity to analyze the
faithfulness performance of existing language generation models. The results
show that even the most powerful pertaining models suffer from serious
unfaithful problems. Take the abstractive summarization task for example, the
annotation results of the ratio of unfaithful summaries generated by several
popular models including T5 (Raffel et al., 2019), BART (Lewis et al., 2019)
and PEGASUS (Zhang et al., 2020b), are shown in Table 5. The annotation
results are combined from Pagnoni et al. (2021) and Cao and Wang (2021).

Table 5: The ratio of unfaithful summaries annotated by human for different systems. System |  XSum |  CNN/DM
---|---|---
TransS2S |  96.9% |  74.8%
BERTSum |  83.7% |  27.2%
T5 |  82.0% |  26.7%
BART |  66.7% |  24.7%
PEGASUS |  60.7% |  13.3%

The above results show that all systems generate over 60% unfaithful summaries
on the XSum dataset. Also, on the CNN/DM dataset, T5 and BART generate over
20% unfaithful summaries. On the one hand, the above results show the severity
of the faithfulness problem of current models, and on the other hand, it also
shows that the impact of different datasets is also very large. We will
analyze the influence of dataset in Section 2.3.

##### Evaluation

Common automatic evaluation metrics for text generation based on n-gram
overlap – BLEU, ROUGE, and METEOR (Papineni et al., 2002; Lin, 2004; Banerjee
and Lavie, 2005) – are insufficient to measure the faithfulness of the
generated text. Kryściński et al. (2019) and Fabbri et al. (2021a) find that
they have low correlation with human judgements of factuality, as shown in
Table 6. So, a lot of new evaluation methods are proposed to evaluate the
faithfulness of generated text for different tasks. We will describe them in
Section 3.

Table 6: The Person and Spearman correlation between different n-gram based metrics and human annotation of faithfulness on CNN/DM dataset, XSum dataset and their combination. | All data | CNN/DM | XSum
---|---|---|---
Metrics | Person | Spearman | Person | Spearman | Person | Spearman
$\rho$ | p-val | $\gamma$ | p-val | $\rho$ | p-val | $\gamma$ | p-val | $\rho$ | p-val | $\gamma$ | p-val
BLEU | 0.10 | 0.00 | 0.07 | 0.00 | 0.08 | 0.01 | 0.08 | 0.01 | 0.14 | 0.00 | 0.20 | 0.00
METEOR | 0.14 | 0.00 | 0.11 | 0.00 | 0.12 | 0.00 | 0.10 | 0.00 | 0.15 | 0.00 | 0.10 | 0.00
Rouge-1 | 0.14 | 0.00 | 0.10 | 0.00 | 0.12 | 0.00 | 0.10 | 0.00 | 0.15 | 0.00 | 0.09 | 0.01
Rouge-2 | 0.12 | 0.00 | 0.08 | 0.00 | 0.08 | 0.00 | 0.07 | 0.01 | 0.17 | 0.00 | 0.14 | 0.00
Rouge-L | 0.13 | 0.00 | 0.09 | 0.00 | 0.11 | 0.00 | 0.09 | 0.00 | 0.16 | 0.00 | 0.10 | 0.00

##### Annotation

Faithfulness annotation of NLG models is very difficult. Most existing work
consider faithfulness as a binary concept, annotating generated text as
faithful or unfaithful (Maynez et al., 2020). However, Falke et al. (2019)
showed relatively low crowd–expert agreement, indicating the presence of
subjectivity in the annotation process. Pagnoni et al. (2021) annotated the
faithfulness of summarization systems in a more fine-grained manner, however,
the inter-annotator agreement is also low. They collect human annotations from
three independent annotators. The inter-annotator agreement in terms of Fleiss
Kappa $\kappa$ (Fleiss, 1971) is 0.58 for faithful or not, and 0.39 for
specific unfaithful error types shown in Table 4, which all indicate low
inter-annotator agreement. Tang et al. (2021) compared the reliability of
ranking and rating-based human annotations of faithfulness in summarization
models and found that ranking-based Best-Worst Scaling annotations are largely
reliable than rating-based annotations.

###  2.3 Cause Analysis

Many factors can affect the faithfulness of model-generated results, such as
dataset, training method, and model expressiveness.

##### Data divergence between source and reference.

The divergence between source and reference is one of the main reason for
extrinsic hallucinations during generation. For example, in text
summarization, summaries were usually written by journalists as introductions
to the news articles they precede. These summaries, therefore, often have true
additional information not found in the document. Such divergence issue
between source and target is not uncommon in conditional text generation
(Kryściński et al., 2019; Wiseman et al., 2017; Dhingra et al., 2019). The
divergence may be a product of heuristic data collection, or it may be
inevitable due to the nature of some NLG tasks, such as table-to-text
generation and dialogue generation.

Existing models are usually agnostic to the source-reference divergence,
making them vulnerable to hallucinations. Thus, models can generate texts that
are not consistent with the input, yet would likely have reasonable model log-
likelihood. This is the main reason why the same model performs differently on
different datasets, such as the difference in summarization performance on the
XSum dataset and the CNN/DM dataset. The XSum dataset is collected
heuristically by simply taking the introductory sentence prefacing each
article as its reference summary, so reference summaries often contain
hallucinations. Maynez et al. (2020) reported that 76.9% of reference
summaries contained unfaithful content. In contrast, the reference summaries
of the CNN/DM datasets are all human-written with less hallucinations.
Therefore, the faithfulness of the summarization model on the CNN/DM dataset
is much better than that on the XSum dataset.

##### Exposure bias between training and inference.

Wang and Sennrich (2020) state that exposure bias (Ranzato et al., 2015), a
discrepancy between training and inference, is partially to blame for
hallucinations. Specifically, the standard teacher-forcing training algorithm
(Williams and Zipser, 1989) used by most existing work can lead to a
discrepancy between what the model sees during training and test time,
resulting in degenerate outputs with factual hallucinations (Maynez et al.,
2020). Furthermore, the model is also only optimized to maximize the log-
likelihood of the reference summary at the word-level, which does not
necessarily reward models for being faithful.

##### Poor text representation.

A model with poor input text representation will fail to do document level
inference, often required for abstraction and generation, and will be
vulnerable to such errors. For example, in text summarization, the percentage
of system summaries with intrinsic hallucination was much higher than in gold
summaries. This phenomenon particularly revealed the models’ tendency to
misrepresent information in the document due to the lack of document-level
understanding and inference. To improve text representation, it is a common
practice to leverage large pre-trained models for downstream NLG tasks. Pre-
training can improve text generation, due to its exposure to vast amount of
text through pretraining, allowing it to integrate background knowledge with
generation. However, Longpre et al. (2021) have discovered that such models
usually over-rely on the parametric knowledge learned from large scale corpus
over the provided input. Also, the dominant language model usually prompts the
decoder generates common words to make sure outputs are fluent.

##  3 Automatic Evaluation Metrics

Recently, there has been wide empirical success in text summarization, machine
translation, dialogue response generation, and other text generation tasks.
For evaluation, these models generally rely on metrics like ROUGE (Recall-
Oriented Understudy for Gisting Evaluation) (Lin, 2004), BLEU (Bilingual
Evaluation Understudy) (Papineni et al., 2002) and METEOR (Metric for
Evaluation of Translation with Explicit ORdering) (Banerjee and Lavie, 2005)
that measure locally constrained n-gram overlap. However, these metrics
cannnot evaluate the faithfulness of generated text.

Recently, much work focus on evaluating the factual consistency of generated
text and propose various new metrics for different NLG tasks. We categorize
these metrics into 4 types: Entailment-based, QA-based, Fact-based, and
Others, as shown in Table 7.

Table 7: Categorization of Evaluation Metrics. Categories | Methods | Target Tasks
---|---|---
Entailment-based | DAE (Goyal and Durrett, 2021) | Summarization, Paraphrasing
RankNLI (Falke et al., 2019) | Summarization
SummaC (Laban et al., 2021b) | Summarization
DialogNLI (Welleck et al., 2019c) | Dialogue Generation
FactCC (Kryściński et al., 2019) | Summarization
RCDG (Song et al., 2020c) | Dialogue Generation
KvPI(Song et al., 2020a) | Dialogue Generation
CI-ToD (Qin et al., 2021) | Dialogue Generation
| DECODE (Nie et al., 2021) | Dialogue Generation
| SentenceNLI (Mishra et al., 2021) | Summarization
QA-based | QAGS (Wang et al., 2020a) | Summarization
FEQA (Durmus et al., 2020) | Summarization
QAFactEval (Fabbri et al., 2021a) | Summarization
QuestEval (Scialom et al., 2021) | Summarization
$Q^{2}$ (Honovich et al., 2021) | Dialogue Generation
QUALS (Nan et al., 2021a) | Summarization
Fact-based | SimAlign (Sabet et al., 2020) | Machine Translation
EntityAlign (Nan et al., 2021b) | Summarization
TripleAlign (Goodrich et al., 2019) | Summarization
PARENT (Dhingra et al., 2019) | Table-to-Text
PARENT-T (Wang et al., 2020b) | Table-to-Text
Others | COCO (Xie et al., 2021) | Summarization
TokenLevelCLS (Zhou et al., 2021) | Machine Translation, Summarization
BARTScore (Yuan et al., 2021) | 16 NLG tasks
| ShannonScore (Egan et al., 2021) | Summarization

###  3.1 Entailment-based Metrics

Figure 4: The framework of entailment-based metrics.

One of the most popular methods is to apply NLI (Natural Language Inference)
to access the faithfulness of generated texts, that is whether the generated
text is entailed, neutral, or conflicting with a given input, as shown in
Figure 4. The basic hypothesis is that the content of generated texts should
be entailed by or at least not conflict with the source text. Though an NLI
model usually predicts three different scores for entailment, neutral, and
contradiction, most work only utilize entailment score to evaluate
faithfulness. Formally, given a source text $x$ as a premise, a generated text
$y$ as a hypothesis, an NLI model $\mathcal{N}$ predicts the entailment score
as $\mathcal{N}(x,y)$. The larger the $\mathcal{N}(x,y)$ is, the more faithful
$y$ given $x$. For evaluating the proposed metrics, most works report their
correlations with human judgements, while some other works, especially
entailment-based metrics, also propose ranking-based downstream tasks to
demonstrate performances. We will also introduce these ranking tasks in the
following.

##### Sentence-level NLI

Traditional NLI tasks predict entailment scores between sentences. However, in
the text generation scenario, the input text $x$ takes various forms and often
contains multiple sentences that severely challenge the application of NLI.
Earlier attempts directly apply NLI classifiers to access the factual
consistency between input text $x$ and output text $y$. They study how NLI
models trained on traditional NLI datasets like MNLI (Williams et al., 2018)
perform. Falke et al. (2019) proposed to aggregate entailment scores between
sentences of $x$ and $y$ to calculate the faithfulness score between $x$ and
$y$, namely RankNLI. Given sentences $s_{y}\in y$, $s_{x}\in x$, RankNLI
formalizes faithfulness between $y$ and $x$ as:

$\frac{1}{|y|}\sum_{s_{y}\in y}\max_{s_{x}\in x}\mathcal{N}(s_{x},s_{y})$ (2)

They found that while entailment prediction should help with this problem,
out-of-the-box NLI models performed poorly on this task. Falke et al. (2019)
also proposes a summary re-ranking task to evaluate the performance of
RankNLI. In this task, a better metric should help the summarization model to
select more faithful summaries during the reranking process of beam search.
They further analyze how different architectures of the NLI model
$\mathcal{N}$, such as ESIM (Chen et al., 2017), BERT (Devlin et al., 2018),
affect the summary ranking task.

Maynez et al. (2020) applied a much simpler strategy by directly using NLI
models trained on MNLI to predict entailment score $\mathcal{N}(x,y)$.
Barrantes et al. (2020) further found that applying ANLI dataset instead of
MNLI dataset in Falke et al. (2019) to train the NLI classifier is more
suitable for faithfulness evaluation. SummaC (Laban et al., 2021a)
comprehensively revisits sentence-level NLI for accessing faithfulness. They
apply a CNN module to aggregate the entailment score matrix between document
and summary sentences, and demonstrate the potential of sentence-level NLI on
various benchmarks.

##### Annotation-based

The major problem of sentence-level NLI metrics is that they are inconsistent
with their downstream tasks, which often require the evaluator to predict
paragraph-level entailment scores. Some work attempted to directly train an
NLI classifier between source text $x$ and target output $y$. A
straightforward solution is to annotate a certain scale of samples for
training the classifier. In text summarization, Aralikatte et al. (2021) and
Gehrmann et al. (2021) finetuned NLI classifier on hundreds of (around 500)
manual annotated samples for faithfulness evaluation and reported a good
performance of this simple metric.

In dialog generation, Welleck et al. (2019b) constructed a Dialog NLI dataset
(DialogNLI) for factual consistency evaluation. To save human labor, they
annotate the relation triples of dialogue sentences instead. Based on relation
triples, they inference the NLI labels by certain rules. Welleck et al.
(2019b) also propose an utterance ranking task, which is often applied to
evaluate the factual consistency of a dialogue model. In this task, given
history utterances $u_{<t}$, a dialogue model is asked to select the next
utterance $u_{t}$ with the lowest perplexities from a set of candidate
utterances $\mathcal{U}$:

$u_{t}=\arg\min_{u\in\mathcal{U}}-\log p(u|u_{<t}),\quad
where\quad\mathcal{U}=(\mathcal{U}^{+},\mathcal{U}^{-},\mathcal{U}^{rand})$
(3)

where the generation probability $p$ is calculated by the dialogue model,
$\mathcal{U}^{+},\mathcal{U}^{-},\mathcal{U}^{rand}$ represent the set of
utterances that are entailed with $u_{<t}$, conflicted with $u_{<t}$ or
randomly selected, respectively. The higher probability the model selects a
$u_{t}$ from $\mathcal{U}^{+}$, the better factual consistency it has.

Song et al. (2020a) proposed a human-annotated dataset, namely Key-value
Profile Identification (KvPI), with single-turn conversations and
corresponding attribute profiles. They further labeled NLI relations between
each conversation and structured profile. With the NLI labels, they trained a
classifier to predict entailment relations between structured attributes and
generated utterances. Qin et al. (2021) proposed a human-annotated dataset CI-
ToD, which incorporates NLI labels between various types of inputs including
dialogue history, user query and the corresponding knowledge base. They
propose a uniform model to assess all the factual consistency relations above.
Nie et al. (2021) proposed a multi-domain dataset DECODE for evaluating
consistency of dialogue. DECODE balances human-written contradicting dialogues
with an equal number of non-contradicting dialogues from several public
datasets. NLI models trained on DialogNLI or DECODE can be used for assessing
the faithfulness of dialogues. Gupta et al. (2021) proposed a benchmark
DialFact to evaluate and check knowledge errors in open domain dialog
generation. They proposed a fact-checking pipeline for this benchmark, in
which an NLI component is applied to check whether the generated utterance is
supported by the retrieved evidences.

##### Weak Supervision

Because it is difficult to annotate a large-scale document-level NLI dataset,
several recent works apply weakly supervised methods for training. Some work
applied data augmentation methods to construct synthetic datasets. FactCC
(Kryściński et al., 2019) constructed positive entailment samples by different
heuristic methods for weak supervised training. For constructing various
positive samples, they swap, deletes or insert textual units like entities,
pronouns, numbers, etc. To better explain this metric, FactCC also provides an
additional version, FactCCX, which highlights spans of evidence in source
documents. Dziri et al. (2021b) introduced a new benchmark BEGIN for
evaluating factual consistency of knowledge grounded dialogue systems. For the
evaluation metric, they also proposed a NLI classifier by constructing
adversarial samples similar to FactCC. SetenceNLI (Mishra et al., 2021)
suggested that the major bottleneck in the utility of NLI models is that
traditional NLI datasets do not exhibit long premises. To solve this problem,
they convert multiple-choice reading comprehension datasets into two-class NLI
datasets using data augmentation methods. In addition to similar rule-based
methods, they also applied text generation models to generate higher-quality
samples.

##### Fine-grained Prediction

Some works utilize fine-grained features to assess faithfulness. They convert
direct predictions of $\mathcal{N}(x,y)$ into sequentially labeled fine-
grained features in the generated text. DAE (Goyal and Durrett, 2020) applies
a new formulation of entailment that decomposes it at the level of dependency
arcs. Instead of making decisions at the sentence level, DAE sequentially
predicts the entailment scores of each dependency arc in the generated
sentence and aggregates them to obtain the final faithfulness score. Goyal and
Durrett (2021) further explores the difference in error distribution between
synthetic and human written summaries, and investigates how fine-grained
supervision information can benefit faithfulness evaluation.

###  3.2 QA-based Metrics

Figure 5: The framework of QA-based metrics.

Because assessing faithfulness requires logical inference over factual
information, it is natural to utilize the reasoning ability of Question
Answering (QA) models. Several recent works proposed QA-based factual
evaluation metrics. As shown in Figure 5, these metrics often include two
components: a question generation (QG) module and a QA module. The core idea
of these metrics is to predict the matching score between source answers (key
information units from source text) and target answers (key information units
from generated text). The overall procedure of these metrics are summarized as
following:

  1. 1.

Answer Selection: Extract information units from the generated text, which is
viewed as target answers.

  2. 2.

Question Generation: Conditioned upon the selected target answers, the QG
module generates questions using the generated text as context.

  3. 3.

Question Answering: The QA module answers the questions with the source text
as context to retrieve source answers.

  4. 4.

Answer Alignment Evaluation: Calculate the matching score between source and
target answers by a answer alignment metric to output the final evaluation
score.

QAGS (Wang et al., 2020a) and FEQA (Durmus et al., 2020) are the earliest QA-
based factual evaluation metrics. These two metrics share similar model
architectures and processing procedures introduced above. In the procedure 1,
QAGS extracted n-grams as the information units for target answers while FEQA
extracted entities. In procedure 2-4, they both applied BERT-based QA modules,
BART-based QG modules and token-level F1 as answer alignment metrics.

Several QA-based metrics followed the framework of QAGS and FEQA with moderate
modifications. QuestEval Scialom et al. (2021) extended this framework by
adding an extra procedure to measure the recall-oriented performance. The
additional procedure generated question-answer pairs from the source document
and answered the questions from the generated text. In contrast to QuestEval,
QUALS Nan et al. (2021a) simplified the above procedure 1-3 by only one neural
language model (QAGen). QUALS employs QAGen as proposed in (Shakeri et al.,
2020), to generate both the questions and answers from the generated text. In
particular, given a summary $y$, QAGen outputs a question-answer (q-a) pair
jointly, separated by a special token <a>. Let $LL_{y}(q,a)$ be the average
log likelihood of generating the q-a pair from the given summary $y$. Then
given the input document $x$, QUALS simply evaluates the average log
likelihood of the QAGen model producing the same q-a pairs, denoted as
$LL_{x}(q,a)$. Formally, given a summary $y$ and input document $x$, QUALS
score is computed as follows:

$\displaystyle QUALS(x,y)=\frac{1}{M}\sum_{(q,a)\in
y}(LL_{x}(q,a)-LL_{y}(q,a))$ (4)

where $M$ is the number of q-a pairs selected on the summary $y$. This
simplification largely decreases the computational time and memory of the
original QAGS.

QAFactEval Fabbri et al. (2021a) conducted extensive comparisons of QA-based
metrics and demonstrated that carefully choosing the components of a QA-based
metric is critical to performance. The optimized settings of QAFactEval in
each procedure are listed in the following:

  1. 1.

Select NP chunks as the textual units as target answers;

  2. 2.

Apply BART-QA2D (Demszky et al., 2018) for the QG module and filter low
quality generated questions;

  3. 3.

Apply Electra-large (Clark et al., 2020) for QA;

  4. 4.

Apply LERC (Chen et al., 2020a) score as the answer alignment metric.

With these carefully selected settings, Fabbri et al. (2021a) boosted the
performance of QA-based metric to a new level.

In addition to the factual metrics in text summarization listed above,
Honovich et al. (2021) proposed a QA-based metric $Q^{2}$ for evaluating
factual consistency in open-domain dialogue generation. They utilized the
entailment score predicted by an NLI model as the alignment metric for answer
spans.

###  3.3 Fact-based Metrics

The most intuitive way to evaluating faithfulness is to count the fact overlap
between generated text and source document, as shown in Figure 6. Facts can be
represented in different forms, such as entities, n-grams and relation triples
(subject, relation, object).

Factual inconsistency can occur at either the entity or the relation level. At
the entity level, a model generated text may contain named entities that never
appeared in the source document. At the relation level, the entities indeed
exist in the source document but the relations between them are not in the
source document.

Figure 6: The framework of fact alignment-based metrics.

####  3.3.1 Entity-based

##### EntityAlign

Nan et al. (2021b) proposed an entity-based metrics that rely on off-the shelf
tools to perform Named-Entity Recognition(NER). Let $N(x)$ and $N(y)$ denote
the number of named-entities in the source (input document) and target
(generated text), respectively. $N(y\cap x)$ denotes the number of entities
found in the generated text that can find a match in the source document. If a
named entity in the generated text consists of multiple words, it is
considered a match as long as any n-gram of the named-entity can be found in
the source document. The degree of faithfulness with respect to the source
text is quantified: $\textbf{prec}=\mathcal{N}(y\cap x)/\mathcal{N}(y)$.

##### SimAlign

For machine translation, Sabet et al. (2020) proposed to leverage multilingual
word embeddings – both static and contextualized – for word alignment between
source language and translated language.

####  3.3.2 Ngram-based

##### PARENT

For table-to-text generation task, Dhingra et al. (2019) modeled facts as
n-grams, and developed a metric PARENT (Precision And Recall of Entailed
Ngrams from the Table) which aligns n-grams from the reference and generated
texts to the semi-structured data before computing their precision and recall.
When computing precision, PARENT effectively uses a union of the reference and
the table, to reward correct information missing from the reference. When
computing recall, it uses an intersection of the reference and the table, to
ignore extra incorrect information in the reference. The union and
intersection are computed with the help of an entailment model to decide if a
text n-gram is entailed by the table. The entailed precision and recall are
combined into an F-score to give the PARENT metric for one instance. The
system-level PARENT score for a model is the average of instance-level PARENT
scores across the evaluation set.

##### PARENT-T

PARENT-T (Wang et al., 2020b) is a table-focused version of PARENT. When
computing precision, PARENT-T considers an n-gram to be correct if it has a
high probability of being entailed by the table. PARENT-T uses the word
overlap model for computing entailment probability. For recall, PARENT-T only
computes it against table to ensure that texts that mention more information
from the table get higher scores. The system-level PARENT-T score for a model
is the average of instance-level PARENT-T scores across the evaluation set.

####  3.3.3 Relation-based

##### TripleAlign

Facts are usually represented by relation triples (subject, relation, object),
where the subject has a relation to the object. To extract triples, Goodrich
et al. (2019) first try to use OpenIE tool (Yates et al., 2007). However,
OpenIE extracts triples with an unspecified schema instead of a fixed schema.
In unspecific schema extraction, relation is extracted from the text between
subject and object. In fixed schema extraction, a relation is predicted from a
pre-defined relations set, which could be viewed as a classification task.
Unspecific schema extraction makes the extracted triples hard to compare with
each other. To resolve this problem, Goodrich et al. (2019) change to use
relation extraction tools with fixed schema, which helps extracted triples
easier to compare.

###  3.4 Other Metrics

Recently, there are some work evaluate faithfulness of text generation from
other perspectives.

##### BARTScore

Yuan et al. (2021) conceptualize the evaluation of generated text as a text
generation problem, modeled using pre-trained sequence-to-sequence models,
directly evaluating text through the lens of its probability of being
generated from or generating other textual inputs and outputs. The general
idea is that models trained to convert the generated text to/from a reference
output or the source text will achieve higher scores when the generated text
is better. They operationalize this idea using BART (Lewis et al., 2019), an
encoder-decoder based pre-trained model, and propose a metric BARTScore with a
number of variants that can be flexibly applied in an unsupervised fashion to
evaluation of text from different perspectives (e.g. informativeness,
fluency,or factuality).

$\displaystyle
BARTScore=\sum_{t=1}^{m}w_{t}logp(\textbf{y}_{t}|\textbf{y}_{<t},\textbf{x},\theta)$
(5)

To evaluate faithfulness, they propose to compute the probability from source
document to hypothesis $p(y|x,\theta)$. This direction measures how likely it
is that the hypothesis $y$ could be generated based on the source text $x$.

##### TokenLevelCLS

Zhou et al. (2021) propose a general-purpose method for token level
hallucination detection for conditional sequence generation tasks. Given the
source input $x$, they first formulate the task of token-level hallucination
detection as a sequence labeling problem where a binary label is predicted at
each position of the generated text. They train a model with synthetic
training data in the form of $((x,y),L_{y})$ where $L_{y}$ are the labels at
every position of $y$ that indicate if each word is a hallucinated one or not.
They leverage the BART model (Lewis et al., 2019) to mapping a corrupted
sentence back to the original text it was derived from, without providing it
any access to the source sentence, thereby encouraging it to insert new
content as needed to ensure fluency. Then, they finetune a pre-trained
language model (LM) on the synthetic data to help detect the token level
hallucinations in various conditional sequence generation tasks.

##### CoCo

CoCo (Xie et al., 2021) is proposed to evaluate the faithfulness of summarized
texts via counterfactual estimation. They point out that the effect of
language prior can be blamed to cause factual inconsistency. The intuition is
that when texts are generated more relying on the source document rather than
the language prior, they should be more likely to be faithful w.r.t. the
source documents. They adopt the probabilities of the tokens of evaluated
summaries to implement the automatic evaluation metric. Specifically, given
the source document $x$ and model-generated summary $y$, several key tokens
$y^{\prime}$ are first selected from $y$, then the source document $x$ is
masked according $y^{\prime}$ to produce a masked version $x^{\prime}$. $x$
and $x^{\prime}$ are feed into the same scoring model respectively to generate
the probability of each token in $y^{\prime}$, i.e., $P(y_{i}|x,y_{<i})$ and
$P(y_{i}|x^{\prime},y_{<i})$, $\forall y_{i}\in y^{\prime}$. The CoCo value is
defined as:

$\displaystyle CoCo=\frac{1}{|y^{\prime}|}\sum_{y_{i}\in
y^{\prime}}P(y_{i}|x,y_{<i})-P(y_{i}|x^{\prime},y_{<i})$ (6)

##### ShannonScore

Egan et al. (2021) proposed a reference-free metric ShannonScore to evaluate
the quality of generated summary via Shannon Game. The Shannon information
content of event $E$ with the probability $p(E)$ of happening is defined as
$I(E)=-\log p(E)$. The ShannonScore performs the Shannon Game with a language
model such as GPT-2 (Radford et al., 2019). The main assumption of this metric
is that if $y$ is a satisfactory summary of $x$, then $I(x|y)<I(x)$, as
documents that have little to do with the summary should be much less likely
than documents that are relevant to the summary after conditioning the
language model. Thus they define an Information Difference metric of summary
quality as:

$\displaystyle ID(x,y)=I(x)-I(x|y)$ (7)

They further define the ShannonScore as the normalized Information Difference:

$\displaystyle s(x,y)=\frac{I(x)-I(x|y)}{I(x)-I(x|x)}$ (8)

where $I(x|x)$ is the lower bound of $I(x|y)$.

###  3.5 Meta Evaluation

As there are a lot of new metrics proposed to evaluate the faithfulness of
language generation, they perform differently on different tasks. To verify
the effectiveness of the above metrics, these work usually report the
correlations between their own metrics and human-annotated factual consistency
scores. However, it is difficult to compare each metric by the correlations as
the diversity of annotating settings in different works and disagreement among
different annotators. To directly compare the effectiveness of different kinds
of faithfulness metrics, several work (Gabriel et al., 2021b; Koto et al.,
2021) proposed benchmarks to conduct meta-evaluations of faithfulness metrics.
For example, the popular benchmarks of evaluating the faithfulness metrics for
abstractive summarization include FRANK (Pagnoni et al., 2021), SUMMAc (Laban
et al., 2021b), QAGS (Wang et al., 2020a), FEQA (Durmus et al., 2020) and CoCo
(Xie et al., 2021). Table 8 combines these benchmarks, showing the Pearson
correlations between different types of faithfulness evaluation metrics and
human annotations. To facilitate reliable evaluation metrics for grounded
dialogue generation, Dziri et al. (2021b) also proposed a benchmark BEGIN for
evaluation of grounded dialogue generation systems.

Table 8: Meta evaluations on several popular benchmarks. The results are Pearson correlations between different types of faithfulness evaluation metrics and human annotations. Metrics | FRANK benchmark | QAGS benchmark | FEQA benchmark | CoCo benchmark
---|---|---|---|---
| CNN/DM | XSum | CNN/DM | XSum | CNN/DM | XSum | CNN/DM | XSum | SUMMEVAL
N-gram based | ROUGE-1 | 0.12 | 0.15 | 0.29 | 0.13 | 0.12 | -0.03 | 0.29 | 0.13 | 0.20
ROUGE-2 | 0.08 | 0.17 | 0.18 | 0.09 | 0.13 | -0.06 | 0.18 | 0.09 | 0.17
ROUGE-L | 0.11 | 0.16 | 0.24 | 0.09 | 0.13 | -0.06 | 0.23 | 0.08 | 0.19
BLEU | 0.08 | 0.14 | 0.21 | 0.06 | 0.12 | -0.07 | 0.18 | 0.03 | 0.11
METEOR | 0.12 | 0.15 | 0.27 | 0.10 | - | - | 0.26 | 0.11 | 0.17
BERTScore | 0.02 | -0.04 | 0.28 | 0.03 | 0.11 | 0.10 | 0.37 | 0.11 | 0.19
Entailment-based | ENT | - | - | - | - | 0.03 | -0.06 | - | - | -
DAE | 0.25 | 0.04 | - | - |  | - | - | - | -
FactCC | 0.36 | 0.07 | - | - | - | - | - | - | -
QA-based | FEQA | -0.01 | 0.02 | - | - | 0.32 | 0.26 | - | - | -
QAGS | 0.13 | -0.02 | 0.55 | 0.17 | - | - | 0.31 | 0.15 | 0.18
QuestEval | - | - | - | 0.33 | - | - | 0.49 | 0.07 | 0.37
Others | OpenIE | 0.16 | 0.00 | - | - | 0.09 | 0.02 | - | - | -
CoCo | - | - | - | - | - | - | 0.59 | 0.24 | 0.42

Table 8 shows that the meta-evaluation results in different benchmarks differ
greatly. The same metric can also performs very different on different
datasets. Overall, the QA-based metrics achieve better performance on most
datasets and benchmarks. However, the correlations with human evaluations are
not more than 0.6. Therefore, the faithfulness evaluation remains an open
question of exploration.

##  4 Optimization Methods

A lot of optimization methods for the faithfulness problem have been proposed
for different tasks, including abstractive summarization, dialogue generation,
data-to-text generation and machine translation. However, most of these
methods are general for different tasks. They can be categorized as factual
guidance, auxiliary tasks, learning methods, post-editing, constrained
decoding and others. In the following, we will organize the optimization
methods for each task into the above categories to facilitate comparing and
learning of different methods, as shown in Figure 3.

###  4.1 Faithfulness in Abstractive Summarization

The faithfulness problem has attracted more and more attention in abstractive
summarization. A lot of mitigation methods have been studied, many of which
also can be applied to other NLG tasks. In the following, we will introduce
the main types of optimization methods for abstractive summarization.

####  4.1.1 Factual Guidance

Factual guidance is an intuitive and effective method for boosting
faithfulness and informativeness in summarization tasks. Guidance can be
defined as some signals which are fed into the model as additional inputs to
the source document. Within this framework, the crucial points are what kind
of information we need to feed into the model and how to feed it. Figure 7
shows a simple seq2seq based factual guidance framework in which two encoders
process origin source and extra guidance signals respectively, then a decoder
generates final summaries considering the hidden states of both two encoders.
Here, the guidance signals could be keywords, important sentences or other
structures such as relations or semantic graphs. According to the types of
guidance signals, factual encoders could be a Transformer network (for signal
of sequence structure) or a Graph Attention Network (for signal of graph
structure) Veličković et al. (2017). GSum Dou et al. (2021) is such a general
and extensible framework that can take different kinds of external guidances
as extra inputs to mitigate the unfaithful problems. We follow the basic
classifications of guidance signals of GSum and then make an extension by
supplying some different but effective ones. We divide guidance signals into
three types: keywords, sentences and relations.

Figure 7: The framework of factual guidance. Usually, we use an oracle method
to select guidance during training and use automatically extracted guidance at
test time.

##### Keyword Guidance

Keywords reflect the crucial information of the source text in a simplest way.
They help the summarization models to focus on the most important parts of the
source text and result in less factual errors. Li et al. (2018a) propose a Key
Information Guide Network which encodes the keywords into the key information
representation, to guide the process of generation. Firstly, they extract
keywords from the text by using TextRank algorithm, then encode keywords by
Bi-RNN network, and guide the generation process by cooperating keyword
representations in both attention mechanism and the pointer mechanism. Saito
et al. (2020) further combine pre-trained seq2seq model with token-level
saliency models called CIT, in which a saliency model (Transfomer encoder with
feed-forward layer) produces a score for each token in order to select
important ones which are denoted as $K$. Then a combined text
$\hat{X}=concat(K,X)$ is given to the seq2seq model as the input.

##### Sentence Guidance

Keywords convey limited information of the source text, so some works turn to
sentence-level guidance which contains more abundant information including
keywords and the connections among them. Cao et al. (2018) propose Re3Sum
which retrieves existing summary sentences as candidate templates, and then
uses an extended seq2seq framework to jointly conduct template reranking and
template-aware summary generation. Specifically, both the source text $X$ and
the soft template $R$ are converted into hidden states with a RNN encoder. In
the Rerank module, they measure the saliency of $R$ according to its hidden
state relevance to $X$. In the Rewrite module, a RNN decoder combines the
hidden states of $X$ and $R$ to generate a summary $Y$. Song et al. (2020d)
exploit PORL-HG model following the extract-then-rewrite famework. PORL-HG
firstly selects some attractive sentences from the article by an extractor,
then rewrites these sentences by a seq2seq-based abstractor. The model
combines the extractor with the abstractor by a reinforcement learning network
which regards the popularity score and ROUGE scores as rewards to make sure
generated headlines are both attractive and faithful.

##### Relation Guidance

Dou et al. (2021) argue that if we utilize full sentence as guidance signals,
it may contain much unnecessary and irrelevant information which is not
crucial in a summary and could distract the model from focusing on the actual
important parts of the source text. To address this problem, some works use
relation information in the form of relational triples as factual guidance.
Cao et al. (2017) leverage open information extraction and dependency parsing
techniques to extract actual fact descriptions from the source text. They
propose a dual-attention seq2seq framework to force the generation conditioned
on both the source text and the extracted fact descriptions. Huang et al.
(2020) present ASGARD framework which enhances the regular document encoder
with an independent graph-structured encoder which improves upon Graph
Attention Networks (Veličković et al., 2017) to maintain the global context
and local characteristics of entities. They utilize <subject, predicate,
object> triples extracted by OpenIE to construct a knowledge graph, then use
the hidden states of input tokens represented by RoBERTa (Liu et al., 2019c)
to initialize the graph nodes. During decoding, both the representations of
source tokens and graph nodes are incorporated into each generation step via
cross-attention mechanism. In this way, the knowledge graph can be used as an
extra factual guidance during summary generation.

Most works use OpenIE to extract relations from source documents, then
represent them as graph structures to improve seq2seq models. However, these
OpenIE-based graphs only contain sparse relations between partial words, which
cannot cover the overall semantic meaning of the source article. Wu et al.
(2021a) propose BASS which firstly introduce an unified semantic graph to
enhance the performance of multi-document summarization. To construct the
semantic graph, they extract phrases and their relations from sentences by a
two-stage merging in which tokens are firstly merged into phrases based on
dependency parsing trees, then co-referent phrases are merged into graph nodes
according to co-reference chains. Finally, the model encodes graph structures
both in encoding and decoding processes, by applying the graph adjacent matrix
as self-attention mask and using an graph-propagate attention mechanism to
guide the decoding process.

####  4.1.2 Auxiliary Tasks

Figure 8: The framework of auxiliary task-based methods via multi-task
learning, reinforcement learning and re-ranking.

Unlike guidance methods which improve factual consistency explicitly,
auxiliary task-based methods combine extra tasks which are correlative with
factual correctness to boost the performance of summarization systems in an
implicit way . There are three widely used frameworks that can easily involve
auxiliary task into summarization task, which are reinforcement learning (RL)
framework, multi-task learning framework and re-ranking framework, as shown in
Figure 8. In the RL framework, it is common to design a score model for
generated summaries to obtain a reward which will optimize the factual
consistency of summarization models. As for the multi-task framework, a task-
specific layer will be stacked over the shared-weight encoder. In this way,
the summarization model and auxiliary model share the same semantic
representations but have different learning objectives. The related auxiliary
task can be seen as a supplement to the summarization task and will improve
the performance of the original summarization system. As for the re-ranking
framework, it firstly generates several candidate summaries, then a score
model based on auxiliary tasks produces a score for each candidate, and
finally the best one is selected as the summary. In the following, we will
describe several common auxiliary tasks to improve faithfulness of abstractive
summarization.

##### Entailment Task

Natural Language Inference (NLI), in which a hypothesis sentence is classified
as either entailed by, neutral or contradicting a premise sentence. Previous
works (Li et al., 2018b; Falke et al., 2019; Barrantes et al., 2020; Fabbri et
al., 2021b; Laban et al., 2021b) have proved that NLI tasks can improve
faithfulness of summarization models. It can be incorporated into
summarization models by multi-task learning, or acting as a RL reward, or
utilized to re-ranking summary candidates.

Li et al. (2018b) is the first work which incorporates entailment knowledge
into abstractive summarization. They argue that a correct summary is semantic
entailed by the source document. They propose an entailment-aware encoder
under a multi-task learning framework, and an entailment-aware decoder under
an RL framework with entailment rewards. In particular, they use shared weight
encoders trained on both the summarization task (i.e. encoder+decoder) and the
entailment task (i.e. encoder+classifier). Entailment prediction is regarded
as an auxiliary task for summary generation. When decoding, they treat the
entailment score as a special reward and combine the reward with a maximum
likelihood training process by RL.

Following the idea that all information in a summary should be entailed by the
source document, Falke et al. (2019) propose a re-ranking approach to select
summaries with less unfaithful errors by entailment prediction models. They
design a score function mentioned in Equation 2 to measure the entailment
score of a generated summary $y$ given its source document $x$. The candidate
summary with the highest score $\sigma(y)$ is selected as the model output
after reranking. Barrantes et al. (2020) follow this idea and make a further
step by applying the adversarial NLI dataset to train the NLI model. More
accurate NLI model has more potential of selecting faithful summaries. Fabbri
et al. (2021a) propose query-based summarization model which apply NLI score
as one of the reinforcement learning rewards to improve factual coinsistency.

##### Question answering Task

Generating factual consistent summaries not only needs the overall
understanding of source text but also the discrimination between crucial and
useless parts. Thus, it is a natural way to check a summarization model’s
comprehension and distinction abilities by a QA model. The QA-based methods
mainly calculate a QA score by measuring the overlap degree of answers
extracted from source text and from the generated summaries, then use the QA
score as the reward in the RL framework or the reranking framework. The key
procedures of QA-based tasks are mentioned in the Section 3.2. Following this
idea, Nan et al. (2021a) incorporate a QA model (Equation 4) into the seq2seq
architecture by a novel contrastive learning method. They firstly produce some
candidate summaries, then sort them into positive samples and negative samples
according to the QA score, finally improve faithfulness of models through
contrastive learning over them (specifically introduced in Section 4.1.3).

##### Other Tasks

Zhang et al. (2020d) develop a concise framework to quantify factual
correctness of a generated summary using an information extraction model. They
take a structured vector $v$ to represent facts in the reference summaries.
Each dimension of vector $v$ is a binary variable which describes whether an
event or an entity is present or not in the text.

$v=f(y)=(v_{1},\dots,v_{m})$ (9)

Given the reference summary fact vector $v$ and generated summary fact vector
$\hat{v}$, a factual accuracy score s can be computed as:

$s(\hat{v},v)=\frac{\sum_{i=1}^{m}1[v_{i}=\hat{v_{i}}]}{m}$ (10)

Finally, they combine factual score with rouge score as reward via a
reinforcement learning framework.

Nan et al. (2021b) propose a series of simple but effective entity-level
methods to improve factual consistency of abstractive summarization, including
data filtering, multi-task learning, and joint entity and summary generation.
For data filtering, they first apply Spacy NER (Honnibal and Montani, 2017) on
reference summary to identify all named-entities. If any entities cannot find
a match in the source document, they consider this sample as a noisy data and
discard the sentence that contains the entity from the ground truth summary to
make sure that there is no hallucination in the dataset. They also add a
classification layer after the encoder of BART to identify summary-worth
entities. As for decoding, they train the BART model to first generate the
sequence of summary-worth entities and then the summary so that the salient
named-entities can be incorporated into the cross-attention of the decoder.

####  4.1.3 Learning Methods

##### Contrastive Learning

Cao and Wang (2021) observed that the commonly used maximum likelihood
training method showed weak ability of distinguishing references from
incorrect generations. Therefore, a potential solution is to design new
learning objectives to improve the preference of factual summaries over
inconsistent ones. Contrastive learning(CL) is such a paradigm which is first
proposed in visual tasks and recently utilized in many NLP tasks. The main
idea of contrastive learning shown in Figure 9 is to learn representations of
similar samples staying close to each other, while dissimilar ones keeping
away. The key point of CL is how to generate positive and negative samples. In
visual tasks, it is common to construct positive samples by rotating,
resizing, distorting the origin picture, and consider other images as negative
samples. In this section, we will introduce several effective methods to
construct positive and negative samples in summarization task, and how to
involve them into the contrastive learning framework.

Figure 9: Contrastive learning framework.

Cao and Wang (2021) designe a task-specific contrastive learning formulation
(CLIFF) that teaches a summarizer to expand the margin between factually
consistent summaries and incorrect peers. CLIFF uses three methods to
construct positive samples, including paraphrasing with synonym substitution,
randomly replacing words, and back-translation. As for negative samples,
previous works often treat other samples in the same batch as negative ones.
However, Cao and Wang (2021) argue that such negative samples are easy to
distinguish because they are totally different from positive ones. It will be
more effective to construct negative samples by making a small but crucial
change based on the original references, so that the model can focus on the
real important parts of the source text and enhance the ability of
differentiating factual and non-factual summaries. Following this idea, CLIFF
designs four strategies to create negative samples:

  * •

Entity swap imitates intrinsic errors: swapping named entities in the
references with other randomly selected entities of the same entity type in
the source text.

  * •

Mask-and-fill with BART: replacing each named entity in a reference with a
[MASK], then let BART generates new entities.

  * •

Source-conditioned regeneration: for each entity in the reference, feeding the
text before it along with the origin source into BART, then combining the text
before the entity with the generated text as a negative sample.

  * •

System generation: selecting system generated summaries with low probability
as negative samples.

After constructing positive samples (denoted as $P$) and negative samples
(denoted as $N$), CLIFF optimizes the contrastive learning objective in
Equation 11 and combines it with typical cross-entropy loss to form the final
training objective shown in Equation 12, where $h_{i}$, $h_{j}$, $h_{k}$ are
representations for summary $y_{i}$, $y_{j}$, $y_{k}$. $sim$ calculates the
cosine similarity between summary representations.

$\begin{split}L_{CL}=-\frac{1}{\dbinom{\left|P\right|}{2}}\sum_{y_{i},y_{j}\in
P,y_{i}\neq y_{j}}log\frac{exp(sim(h_{i},h_{j})/\tau)}{\sum_{y_{k}\in P\cup
N,y_{k}\neq y_{i}}exp(sim(h_{i},h_{k})/\tau)}\end{split}$ (11)

$L=L_{CE}+\lambda L_{CL}$ (12)

Except for entity-level replacement, Liu et al. (2021c) make a further step by
switching the sentiment of some sentences by adding negation words or
replacing opposite meaning words to generate more diverse negative samples.

Liu et al. (2021b) argue that previous works (Cao and Wang, 2021; Liu et al.,
2021c) mainly focus on entity faithfulness which is not equal to summary
faithfulness. Thus, they propose a contrastive summarization framework CO2Sum
and a span-level negative samples construction method LFN based on pre-trained
language model. Specifically, they delete or disturb factual fragments in
sentences and observe the language model probability of predicting the context
based on these sentences in an iterative way to distinguish which fragments
are important to the source text. After detecting most influential factual
spans, they replace the fragment in the gold summary with embedding-similar
article words to construct negative samples. They involve contrastive learning
in both encoder and decoder. In the encoding procedure, they apply contrastive
learning between source text and summaries by making the representations of
the article and the ground truth summary closer, and make that of the article
and the factual inconsistent summaries apart. The CL loss $L_{Enc}$ for the
encoder is similar to Cao and Wang (2021). As for the decoder, CO2Sum applies
contrastive learning between summaries with a max-margin loss (Yang et al.,
2019) $L_{Dec}$ to force the model to increase the decoding probabilities of
ground truth summaries while decrease the decoding probabilities of negative
summaries. The margin loss $L_{Dec}$ and the final training objective $L$ are
shown as following.

$L_{Dec}=max\left\\{\frac{1}{R}\sum_{i\in
R}(P_{s}(T_{neg},i)-P_{s}(T_{gold},i))+\eta,0\right\\}$ (13)
$L=L_{CE}+\lambda_{Enc}L_{Enc}+\lambda_{Dec}L_{Dec}$ (14)

where $R$ means replaced positions with inconsistent facts, $T_{neg}$ and
$T_{neg}$ denote the negative summary and ground truth summary respectively,
$P_{s}(T,i)$ denotes the generation probability of the $i$-th position in
sequence $T$. $\lambda_{Enc}$ and $\lambda_{Dec}$ denote the loss weights of
the constrastive learning objectives in the encoder side and decoder side,
respectively.

Most works construct negative samples by simply replacing some non-target
sequences. Lee et al. (2021) argue that these explicit negative samples are
suboptimal, since they are easily distinguishable from the correct output,
especially when models are pre-trained with large corpus. Within the simple
and explicit sample construction framework, models barely learn nothing. Thus,
they propose a principled method called CLAPS to construct positive and
negative samples implicitly by adding perturbations to the input sequence. To
generate a negative example, they add a small perturbation (hard sample) to
the hidden representation of target sequence, then minimize its conditional
likelihood. As for positive examples, they adding a large perturbations while
enforcing the model to have a high conditional likelihood. This will yield a
negative example that is very close to the original representation of target
sequence in the embedding space but is largely dissimilar in the semantics,
while the generated positive example is far away from the original input
sequence but has the same semantic as the target sequence. It could generate
hard examples which the model might be difficult to discriminate, helping it
learn more meaningful representations.

####  4.1.4 Post-Editing

Above methods require modification of model structures or extra sample
construction processes to improve factual consistency, which may affect the
informativeness (e.g. ROUGE scores) of summary results. Post-editing based
methods improve factual consistency by adding an corrector to system-generated
summaries. They consider generated summaries as drafts, and correct factual
errors to form the final summaries. This process is quite similar to the human
writing process, where people write a first draft, then review and edit it to
make it better. Figure 10 shows the general framework of post-editing methods.

Figure 10: The post-editing framework.

Dong et al. (2020) propose SpanFact, a suite of two factual correction models
that leverage knowledge learned from question answering models to correct
system-generated summaries through span selection and correction. SpanFact
takes into account entity-level corrections and make them iteratively.
Specifically, assume that the system summary has $N$ entities. At time step
$i$, they mask the $i$-th entity and use this masked sequence as a query to
the QA model. The QA model will replace the wrong entities with the correct
ones based on the source document. The corrected entity will then form an
updated summary for use in the next step. Human evaluation demonstrates that
SpanFact is able to correct about 26% unfaithful summaries, while barely
destroying any otherwise correct summaries. Cao et al. (2020) simplify the
post-editing procedures by directly training a seq2seq rewrite model on
artificial unfaithful summaries as a corrector. They create a weakly-
supervised training dataset based on the text transformations following
Kryściński et al. (2019) which replace entities, numbers, numerals and
pronouns in source documents with other tokens of the same type. The goal of
the corrector is to generate correct summaries based on the unfaithful
summaries and source documents.

As a standalone module, post-editing methods have been shown to be effective
in improving the faithfulness of abstractive summarization systems while
preserving their informativeness. However, it’s more of an indirect solution
than a fundamental solution to factual inconsistencies.

####  4.1.5 Constrained Decoding

Lexically constrained or guided decoding is a modification of beam search that
enforces the inclusion of pre-specified words and phrases in the output. This
is a general way to control specific tokens in the generated output without
modifying the model structure or additional training data.

Mao et al. (2020) propose CAS (Constrained Abstractive Summarization) to
improve the factual consistency of summarization systems by constructing
constrained token sets during dynamic beam search decoding. It only allows the
generation process to end when all constraints are met. They focus on entities
and noun phrases and select these types of words that are not present in the
summaries generated by the unconstrained system to form constrained sets.
Therefore, the model will generate more correct and faithful tokens during the
inference process, effectively improving the faithfulness of abstractive
summarization. Aralikatte et al. (2021) introduce the Foucs Attention
Mechanism (FAME) for the transformer-based seq2seq architecture. FAME combines
a standard contextual representation with a dynamic source-conditioned lexical
bias layer, which encourages the decoder to actively generate tokens that are
faithful to the input document.

####  4.1.6 Other Methods

Zhao et al. (2020a) propose HERMAN which learns to recognize and verify
quantity entities in candidate summaries, in order to re-rank the candidate
summaries to select the one whose quantity terms are supported by the original
text. During the training process, they use a BiLSTM-CRF decoder as a
verification model to tag sequence labels and finally predict an overall label
that indicates whether the output summary is faithful to the source input or
not. At the test time, the same verification model is applied to rerank the
candidate summaries, then select the best one with less hallucinations.

Gabriel et al. (2021a) proposed Co-opNet, a generator-discriminator framework
to do fact-checking for text generation. In this framework, the generator
outputs a series of candidate summaries. Then the discriminator scores the
factuality of these summaries using one of the following objectives: the
overlap between the introduction of a scientific article and the predicted
evidence spans in summaries, the ordering of predicted discourse roles, the
coverage of predicted discourse roles, or the likelilood of adjacency between
generated sentences. The best summary is selected by combining the scores of
the generator and the discriminator.

Cao et al. (2021) propose an interesting method to detect factual errors by
using the prior and posterior predicted probabilities of each token. They
assume that if an entity is a factual error, giving the source should not
provide more evidence for it, resulting in only small changes in the
probabilities between the prior (i.e. without source) and the posterior (i.e.
given source) language models. Based on this assumption, they use prior and
posterior probabilities as key features of a classifier to predict the
factuality of entities.

###  4.2 Faithfulness in Dialogue Generation

Recently, the area of dialogue generation has made significant progress with
end-to-end neural networks and large-scale pre-training (Roller et al., 2021;
Bao et al., 2020). However, a long standing problem, faithfulness, still
challenges current best dialog systems and attracts an increasing amount of
attention. In general, the generated utterance should be faithful to its
history utterances (Vinyals and Le, 2015). Different from other generation
tasks like text summarization, various forms of dialog generation tasks
include a diversity of background or knowledge inputs, with which the
generated utterances should also be consist. In Table 9, we summarize
different forms of inputs that have been studied in dialogue faithfulness. The
optimization methods for dialogue faithfulness are similar to abstractive
summarization, which consists of six types of methods. Some of them can also
be utilized in summarization tasks.

Table 9: Different types of source that dialogue generation models should be faithful to in different tasks. Source Type | Methods
---|---
History Dialogue | DialogNLI (Welleck et al., 2019b), Gao et al. (2019)
Arun et al. (2020), Ghazvininejad et al. (2018)
DECODE (Nie et al., 2021), CI-ToD (Qin et al., 2021)
TransferTransfo (Mesgar et al., 2021), UL (Li et al., 2020a)
Blender (Roller et al., 2021), Balakrishnan et al. (2019)
Nye et al. (2021), Kim et al. (2020)
Persona Facts |  Li et al. (2016a), Zhang et al. (2018)
DialogNLI (Welleck et al., 2019b), DECODE (Nie et al., 2021)
KvBERT (Song et al., 2020a), RCDG (Song et al., 2020c)
TransferTransfo (Mesgar et al., 2021), UL (Li et al., 2020a)
GDR (Song et al., 2020b), Kim et al. (2020)
Unstructured Knowledge |  Rashkin et al. (2021), Wu et al. (2021b)
(e.g. Wikipedia Documents) |  Dinan et al. (2019), Shuster et al. (2021)
Structured Knowledge | KvBERT (Song et al., 2020a), CI-ToD (Qin et al., 2021)
(e.g. Knowledge Graph) | NPH (Dziri et al., 2021a)
User Query | CI-ToD (Qin et al., 2021)
(i.e. task-oriented dialogue) |

####  4.2.1 Factual Guidance

Figure 11: The framework of three types of factual guidance in dialogue
generation.

Several works utilized various guidance information to improve factual
consistency of dialogues. These methods incorporate relevant guidance
information into the training or inference process of dialogue models. As
shown in Figure 7, we categorize theses guidance into three types: implicit
guidance, which is the guidance in vector representations; extracted guidance,
which is the relevant textual information extracted from source inputs;
retrieved guidance, which is information retrieved from open-domain knowledge.

##### Implicit Guidance

Implicit guidance is usually representations that are automatically learned
before or during the training process of a dialogue model. Li et al. (2016a)
inject implicit speaker information into a LSTM-based model to improve the
personality consistency. In each generation step of their model, they fuse the
embedding of the speaker into the text encoder. Zhang et al. (2018) present
the PERSONA-CHAT dataset which provides persona profiles for each speaker.
They also propose a memory-augmented dialogue system where persona profiles
were saved and updated in memory. Guided by the profile memory, the generated
dialogues are more consistent in personality. Gao et al. (2019) combine dialog
generation with style-transfer for a more stylized and context-relevant
chatbot. They fuse conversation modeling and non-parallel style transfer
method by sharing a structured latent space to guide the decoding process.

##### Extractive Guidance

Extractive guidance is often the important information extracted from the
source input, which helps the model focus on the important parts of the input.
Ghazvininejad et al. (2018) propose a knowledge-grounded conversation model,
which extracts factual sentences from history dialog and utilizes them as
factual guidance in the decoding process. Arun et al. (2020) extract tree-
based meaning representations to improve the faithfulness of generated
responses for task-oriented dialog systems. The extract structural knowledge
efficiently guided the model to generate correct information. Rashkin et al.
(2021) utilize control codes to encourage the model to generate responses that
are faithful to the provided evidence. They apply three types of control
codes, including entailment, objective voice and lexical precision, which are
calculated during data pre-processing. Wu et al. (2021b) further construct
fine-grained control codes by using lexical phrases as factual guidance. Based
on these phrases, the generated responses are more relevant and faithful to
the input.

##### Retrieved Guidance

Retrieved guidance is usually from external knowledge. Dinan et al. (2019)
create an open-domain dialogue dataset Wow, where each topic in the
conversation is connected to Wikipedia articles. Then they design a memory
network based dialog system which is enhanced by retrieved knowledge from
Wikipedia. Augmented by knowledge guidance, their model is able to generate
more precise responses. Shuster et al. (2021) propose a retrieval-augmented
neural architectures, in which dialogues are generated grounded on retrieved
knowledge. Specially, they apply a learnable retriever and designed a fine-
grained interactions between history dialogue and knowledge.

####  4.2.2 Auxiliary Tasks

Many work utilize the auxiliary task of Natural Language Inference (NLI) to
improve the factual consistency of dialogue systems. The main approaches
include leveraging entailment scores to rerank candidate texts, or treating
entailment as a reward for reinforcement learning, which are similar to the
entailment-based methods for text summarization (described in Section 4.1.2).

##### Reranking-based

Several works apply entailment score predicted by an NLI model to the
reranking process for selecting more faithful generated text. As discussed in
Section 3.1, several works (Welleck et al., 2019b; Song et al., 2020a; Qin et
al., 2021; Nie et al., 2021) propose their factual evaluation metrics based on
entailment. They utilize entailment scores predicted by their proposed metrics
in the re-ranking process to improve faithfulness of dialogue models. For
example, in Welleck et al. (2019b), given a persona $P$, previous utterances
$u_{<=t}$, and the dialogue model outputs the score of a next-utterance
candidate $s^{i}_{t+1}$, the new score $s_{t+1}^{re-rank}$ after incorporating
NLI relation is:

$s_{t+1}^{re-rank}=s_{t+1}+\lambda s_{t+1}^{contradict}$ (15)

where $s_{t+1}^{contradict}$ is the highest contradiction score between
$s^{i}_{t+1}$ and persona sentences in $P$, hyper-parameter $\lambda$ controls
the NLI model’s influence in re-ranking.

##### Reinforcement Learning

Another type of methods incorporate entailment scores as a part of the
reinforcement learning (RL) rewards, similar to Figure 8. Song et al. (2020c)
propose a RL-based model RCDG for generating persona consistent dialogues.
Similar to the architecture of GANs (Generation Adversarial Neural Networks),
RCG is composed of a generator and two evaluators to estimate the quality and
consistency of generated utterances, respectively. The consistency evaluation
is based on an NLI classifier to compute the entailment score. Mesgar et al.
(2021) also propose an RL-based model TransferTransfo-RL for improving
consistency between generated responses and personas. Differently,
TransferTransfo-RL take the advantage of Actor-Critic (Mnih et al., 2016)
learning approach, which also utilizes the entailment score as reward.

####  4.2.3 Learning Methods

As unfaithful generation relates to the deficiencies of training strategy,
several works improve factual consistency of dialogue models by refining the
training procedures.

Li et al. (2020a) extend unlikelihood training to address various problems in
generating dialogues, including over copying, repetitions, overuse frequent
words, and factual inconsistency. Besides training with common maximum
likelihood estimation (MLE), they apply unlikelihood loss (UL) to alleviate
these problems. In the time step $i$ during training, given an input-output
pair $(x,y)$, a dialogue model $p_{\theta}$ and a set $\mathcal{C}$ containing
sentences contradicting with $y$ which the model should avoid to generate, the
UL is defined as:

$\mathcal{L}_{UL}(p_{\theta},\mathcal{C},x,y)=-\sum_{t=1}^{T}\sum_{y_{c}\in{C}}\beta(y_{c})(1-p_{\theta}(y_{c}|x,y_{<t}))$
(16)

where $\beta(y_{c})$ is the weighting parameter for every $y_{c}\in{C}$.
Roller et al. (2021) followed this method for training a large-scale chatbot.

####  4.2.4 Constrained Decoding

Some works also focus on designing decoding strategies to improve consistency.
Especially, these works apply constrained decoding during inference. These
methods usually require the generated utterance to be semantically consistent
with inputs based on certain semantic structure.

Balakrishnan et al. (2019) apply a tree-structured meaning representations
(MR) in dialog systems. Comparing to common flat MR, which is a flat list of
key-value pairs, their MR is able to represent more fine-grained relations.
Based on the proposed MR, they design a constrained decoding strategy on beam
search which requires the MR of generated text not conflicting with the input.
Similarly, Nye et al. (2021) propose a dual-system approach for faithful text
generation, where the “system 1” is a common model for generation, and the
“system 2” constrains and controls the generated sentences to be factually
correct. The essence of this method is that it applies GPT-3 (Brown et al.,
2020) to parse text into clear and correct logical symbols that are easy for
“system 2” to check. During decoding, “system 2” selects correct candidates
that are faithful to the given context.

####  4.2.5 Post-Editing

Several works focus on refining the generated dialogues without modifying the
original model. These works mainly design an extra refining module to correct
the factual errors in the generated dialogues. These models usually consist of
three steps: generate, delete, and rewrite, similar to the summarization task
shown in Figure 10. In the first step, the dialogue model normally generates
utterances. In the second step, the rewrite module removes the incorrect
contexts in the generated utterances, and then the third step rewrites them to
the correct contexts.

Song et al. (2020b) propose a post-editing based dialogue model, GDR,
following the three steps introduced above. After the normal text generation
procedure “Generate” in the first stage, GDR identifies and deletes conflict
words in the second stage “Delete”. Then GDR recovers the deleted words by a
generation module in the last stage “Rewrite”. Through the three stages above,
GDR refines factual errors in the generated utterance. Dziri et al. (2021a)
apply the similar strategy on knowledge grounded dialogue system. Different
from GDR, their refining module NPH needs to rewrite based on knowledge graph.
After deleting potential incorrect entities in the generated text, NPH based
on graph neural network retrieves correct entities in the grounded knowledge
graph for refining.

####  4.2.6 Other Methods

Kim et al. (2020) propose a dialogue system based on Rational Speech Act
framework (Frank and Goodman, 2012), which enforces dialogue agents to refrain
from uttering contradiction. The proposed model endows dialogue agents with
public self-consciousness, helping them maintain consistency across each
generation step by reflecting the distribution of imagined listeners across
roles.

###  4.3 Faithfulness in Machine Translation

Neural machine translation (NMT) has achieved great success due to the ability
to generate high-quality sentences. Compared with human translations, one of
the drawbacks of current NMT is that translations are not usually faithful to
the input, e.g., omitting information or generating unrelated fragments, which
inevitably decreases the overall quality, especially for human readers. The
optimization methods for faithfulness in machine translation mainly include:
incorporating auxiliary tasks like word alignment (4.3.1), improving learning
methods like minimum risk training (4.3.2), utilizing constrained decoding
methods like grid beam search (4.3.3), etc.

####  4.3.1 Auxiliary Tasks

Wang et al. (2020b) propose a multi-task learning paradigm with two auxilary
tasks, including marked language model task and word alignment task, for
building a faithfulness enhanced NMT (named FENMT). On the encoder side, FENMT
employs a masked language model (MLM) task (Devlin et al., 2018) to infer the
input words didn’t be correctly translated. This task can enhance the ability
of modeling the whole input sentence and give the decoder accurate and
complete representations. On the decoder side, FENMT further uses a word
alignment task to improve the alignment accuracy of the encoder decoder cross-
attention to help the decoder to capture correct contextual representation.
Furthermore, along with the NMT objective, an auxiliary max-margin objective
based on contrastive learning is introduced in all decoding timesteps which
prompts the decoder to translate fluent and faithful sentences.

To improve the ability of the decoder, Tu et al. (2017) propose to introduce a
reconstruction loss to make translation can reconstruct the input sentence.
Kong et al. (2019) propose to use a coverage difference ratio metric as a
reward to train NMT. Zhang et al. (2021); Feng et al. (2020); Garg et al.
(2019) propose to introduce word alignment information in Transformer to
improve translation accuracy.

####  4.3.2 Learning Methods

##### Minimum Risk Training

Wang and Sennrich (2020) hypothesise that exposure bias (Ranzato et al.,
2015), a discrepancy between training and inference, is partially to blame for
hallucinations, and that training with Minimum Risk Training, which avoids
exposure bias, can mitigate this. Minimum Risk Training (MRT) is a sequence
level objective that avoids this problem. Specifically, the objective function
of MRT is the expected loss (risk) with respect to the posterior distribution:

$\displaystyle\mathcal{R(\theta)}=\sum_{(x,y)\in D}\sum_{\widetilde{y}\in
Y(x)}P(\widetilde{y}|x;\theta)\triangle(\widetilde{y};y)$ (17)

in which the loss $\triangle(\widetilde{y};y)$ indicates the discrepancy
between the gold translation $y$ and the model prediction $\widetilde{y}$. Due
to the intractable search space, the posterior distribution $Y(x)$ is
approximated by a subspace $S(x)$ by sampling a certain number of candidate
translations, and normalizing:

$\displaystyle\widetilde{P}(\widetilde{y}|x;\theta,\alpha)=\frac{P(\widetilde{y}|x;\theta)^{\alpha}}{\sum_{y^{\prime}\in\mathcal{S}(x)}P(y^{\prime}|x;\theta)^{\alpha}}$
(18)

where $\alpha$ is a hyper-parameter to control the sharpness of the subspace.
Random sampling was used to generate candidate translations, and the reference
translation was not added to the subspace. They find that Minimum Risk
Training, which does not suffer from exposure bias, reduces the number of
hallucinations substantially, and makes beam search with large beams more
stable.

##### Adversarial Learning

NMT models are still susceptible to input sentence perturbations and tend to
produce hallucinatory outputs in the presence of some source perturbations
(Lee et al., 2018). For example, Belinkov and Bisk (2017) find that NMT models
can be immensely brittle to small perturbations applied to the inputs. Even if
these perturbations are not strong enough to alter the meaning of an input
sentence, they can nevertheless result in different and often incorrect
translations. Belinkov and Bisk (2017) and Karpukhin et al. (2019) study how
to use some synthetic noise and/or natural noise. Cheng et al. (2018) propose
adversarial stability training to improve the robustness on arbitrary noise
type including feature-level and word-level noise. Liu et al. (2018a) examine
the homophonic noise for Chinese translation.

Formally, a set of adversarial examples $Z(x;y)$ is generated with respect to
a training sample $(x;y)$ by solving an optimization problem:

$\displaystyle\\{x^{\prime}|\mathcal{R}(x^{\prime},x)\leq\epsilon,argmax_{x^{\prime}}J(x^{\prime},y;\theta)\\}$
(19)

where $J(.)$ measures the possibility of a sample being adversarial, and
$R(x^{\prime};x)$ captures the degree of imperceptibility for a perturbation.
Although it is difficult to give a precise definition of the degree of
imperceptibility $R(x^{\prime};x)$, $l_{\infty}$ is usually used to bound the
perturbations in image classification (Goodfellow et al., 2014).

Following Goodfellow et al. (2014); Miyato et al. (2016); Ebrahimi et al.
(2017), the white-box method to generate adversarial example stightly guided
by the training loss. Given a parallel sentence pair (x; y), a set of
adversarial examples A(x; y) specific to the NMT model are generated by:

$\displaystyle\\{x^{\prime}|\mathcal{R}(x^{\prime},x)\leq\epsilon,argmax_{x^{\prime}}-\log
P(y|x^{\prime};\theta)\\}$ (20)

where they use the negative log translation probability to estimate $J(.)$.
The formula constructs adversarial examples that are expected to distort the
current prediction and retain semantic similarity bounded by $\mathcal{R}$.

Cheng et al. (2019) propose a gradient-based adversarial learning approach,
called AdvGen, to construct adversarial examples and use these examples to
both attack as well as defend the NMT model to improving the robustness of NMT
models, which consists of two parts: (1) attack the translation model with
adversarial source examples; (2) defend the translation model with adversarial
target inputs to improve its robustness against the adversarial source inputs.
For the generation of adversarial inputs, they propose a gradient-based method
to craft adversarial examples informed by the translation loss over the clean
inputs.

##### Robust Learning on Noisy Corpus

Corpus-level noise in NMT parallel corpora tends to produce significant
hallucinatory patterns (Raunak et al., 2021). NMT models also have a
propensity to hallucinate more frequently under out-of-domain inputs (Müller
et al., 2019). Several techniques can be used to improve learning robustness
to corpus-level noise in NMT. Kang and Hashimoto (2020) propose a loss
truncation method to reduce the impact of noisy references in sequence-to-
sequence training. Li et al. (2020b) propose a modification of expected risk
minimization (ERM), namely Tilted-ERM, to reduce the effect of outliers during
training. Corpus-level noise filtering that incorporating heuristics or
filters (Zhang et al., 2020a; Junczys-Dowmunt, 2018) to remove invalid source-
target pairs is also effective in reducing NMT hallucinations.

####  4.3.3 Constrained Decoding

One interesting method is lexically constrained decoding, a modification to
beam search that allows the user to specify words and phrases that must appear
in the system output. Three algorithms have been proposed for this: grid beam
search (Hokamp and Liu, 2017), constrained beam search (Anderson et al., 2017)
and dynamic beam allocation (Post and Vilar, 2018). These papers showed that
these algorithms do a good job automatically placing constraints and improving
results in tasks such as simulated post-editing, domain adaptation, and
caption generation.

Grid Beam Search (GBS), an algorithm which extends beam search to allow the
inclusion of pre-specified lexical constraints while still taking advantage of
the distribution learned from training data. The algorithm can be used with
any model that generates a sequence $\widehat{y}={y_{0},\ldots,y_{T}}$, by
maximizing $p(y|x)=\prod_{t}p(y_{t}|x;{y_{0},\ldots,y_{t-1}})$. Lexical
constraints take the form of phrases or words that must be present in the
output sequence. This is a very general way to incorporate additional
knowledge into a model’s output without requiring any modification of the
model parameters or training data, thus can improve faithfulness of
translation output.

The computational complexities of grid beam search (Hokamp and Liu, 2017) are
linear and constrained beam search are exponential in the number of
constraints. Post and Vilar (2018) present a more efficient algorithm for
lexically constrained decoding with a complexity of $O(1)$ in the number of
constraints.

####  4.3.4 Other Methods

Some work design special mechanism for using source representation more
effectively. In the RNN-based NMT, Tu et al. (2016) and Mi et al. (2016)
propose a coverage mechanism to improve the accuracy of translation outputs.
Weng et al. (2020a) propose to model global representation in the source side
to improve the source representation. Zheng et al. (2019) propose a capsule
based module to control the source representation dynamically in the decoding
process. Feng et al. (2020) propose a faithfulness part to optimize the
contextual representation before feeding into the decoder. Weng et al. (2017)
propose a bag-of-words loss to constrain decoding process.

###  4.4 Faithfulness in Data-to-Text Generation

Data-to-text generation (or table-to-text generation) has been widely studied
for decades. Recently, deep neural networks have been successfully adopted in
this task. However, the problem of unfaithfulness in data-to-text methods
remains a significant challenge (especially in long-form text generation). In
the data-to-document generation dataset (e.g., Rotowire (Wiseman et al., 2017)
and MLB (Puduppully et al., 2019)), obvious gaps exist in the record
generation (RG) precision between recent methods and faithful templated-based
baselines. Moreover, the issue of generating unfaithful text remains a serious
problem for noisy datasets constructed automatically (e.g., WikiBio (Lebret et
al., 2016) and WikiPerson (Wang et al., 2018)). In WikiBio, almost two-thirds
of the training instances contain unfaithful descriptions (Dhingra et al.,
2019).

####  4.4.1 Factual Guidance

Compared to other text generation tasks, factual information is highly
structured and exists explicitly in data-to-text tasks. Therefore the two-
stage method, which plans the subset of input data to be described and then
generates text from the plan, is popular in this research field. If a method
explicitly considers the issue of faithfulness in the plan-to-text generation
step, we categorize it into factual guidance optimization methods.

Wang et al. (2021) propose a two-stage table-to-text generation method SANA.
SANA firstly constructs a skeleton using an autoregressive pointer network to
select contents from the source table. SANA expands the skeleton to the final
output in the second stage, considering the source table with an edit-based
non-autoregressive model. Therefore, SANA could generate more faithful because
the input of the second stage is directly extracted from the input table as
strong factual guidance.

Shen et al. (2020) model the table-to-text task as a segment-by-segment
generation precedure and every segment is generated in two-stage. Firstly,
proper data records are selected as factual guidance. Secondly, text
corresponding to the plan is generated by paying attention only to the
selected input data records.

####  4.4.2 Auxilary Tasks

Liu et al. (2021a) extend the two-stage table-to-text generator to an
augmented plan-based method. They create a pseudo training corpus for the
plan-to-text phase, which covers all entities in the target description. In
this way, there is no hallucinated entity in the training phase of the plan-
to-text generating. Therefore the noise in the original corpus does not affect
the plan-to-text generating step.

####  4.4.3 Learning methods

Neural sequence-to-sequence learning-based data-to-text models are often
trained by maximum likelihood loss optimization. However, this kind of model
suffers from the noise of the training data and leads to unfaithful output
(also called divergence).

Wang et al. (2020b) extend the maximum likelihood loss of attention-based
Transformer with two additional losses. The first one is a latent matching
disagreement loss which measures the distance between embeddings of the input
table and the output text. The second one is an entity matching optimal-
transport loss to measure the entity matching of the table and the output.
Reinforcement learning (RL) based methods could directly optimize the
faithfulness of data-to-text generation.

Liu et al. (2019b) propose a force attention method to encourage the model to
focus on uncovered data records. Then they adopt RL for information richness
to generate more faithful descriptions for the input data. Rebuffel et al.
(2020) propose an RL-based approach relying on the PARENT metric to reduce the
issue of unfaithfulness.

####  4.4.4 Constrained Decoding

Tian et al. (2019) propose a confident decoding method to detect and avoid
unfaithful generation in the decoder. For each decoder position, a confidence
score consists of attention and a dedicated language model that only gives
higher scores to common words. Therefore, the confidence score could be
utilized to detect the generating of a word conveying source information
without paying enough attention to the source data. Tian et al. (2019) believe
that lower confidence scores indicate higher risks of generating unfaithful
text. In the training time, a variational Bayes training framework is designed
to ensure the model generates high confidence results. In the inference time,
tokens generated with low confidence scores would be marked and skipped.

Filippova (2020) treats faithful data-to-text as a controllable text
generation problem. In the training corpus, a prefix measuring the amount of
noise is appended to the input sequence. Although data-to-text models are
trained without modification, we still categorize this method into constrained
decoding because the controlling prefix guides the decoding. Rebuffel et al.
(2022) extend the controllable text generation method to the fine-grained
level. Word alignment labels are calculated through dependency parsing, and
the labels guide the proposed weighted multi-branch neural decoder.

####  4.4.5 Other Methods

Considering that noise in the training corpus is critical to the faithfulness
of data-to-text generation, pre-processing datasets is a direct and reasonable
method. Nie et al. (2019) propose a neural data refinement method to reduce
unaligned noise from original datasets. Wang (2019) observe that only 60% of
the output contents in RotoWire could be grounded to the input records. They
purify and enlarge the original dataset to a new RotoWire-FG dataset.

###  4.5 Faithfulness in Other NLG Tasks

Faithfulness is a common problem in NLG tasks. There are also some researches
on faithfulness for other tasks, such as image caption, image-to-text
radiology report generation and general factual language model. The methods
for improving faithfulness include: factual guidance by knowledge graph,
constrained decoding, and incorporating auxiliary tasks (e.g. entailment,
word/entity alignment), etc.

####  4.5.1 Factual Language Model

Some work design language models that are conditioned on external, structured
knowledge source to generate factual text. Logan IV et al. (2019) introduce
the knowledge graph language model (KGLM), a neural language model with
mechanisms for selecting and copying information from an external knowledge
graph. The KGLM maintains a dynamically growing local knowledge graph, a
subset of the knowledge graph that contains entities that have already been
mentioned in the text, and their related entities. When generating entity
tokens, the model either decides to render a new entity that is absent from
the local graph, thereby growing the local knowledge graph, or to render a
fact from the local graph. When rendering, the model combines the standard
vocabulary with tokens available in the knowledge graph, thus supporting
numbers, dates, and other rare tokens.

Figure 12: An illustration example of the factual language model (Logan IV et
al., 2019). It maintains a dynamically growing localized knowledge graph
containing facts that are (possibly) conveyed in the sentence above. The graph
is built by iteratively linking each detected entity to Wikidata, then adding
any relations to previously mentioned entities.

An example is shown in Figure 12. Initially, the graph is empty and the model
uses the entity “Super Mario Land” to render the first three tokens, thus
adding it and its relations to the local knowledge graph. After generating the
next two tokens (“is”, “a”) using the standard language model, the model
selects “Super Mario Land” as the parent entity, “Publication Date” as the
relation to render, and copies one of the tokens of the date entity as the
token (“1989” in this case). The factual completion capabilities of KGLM,
which predicts the next word after a factual sentence (e.g., “Barack is
married to”), is significantly more accurate. KGLM is able to generate
accurate facts for rare entities, and can be controlled via modifications on
the knowledge graph.

####  4.5.2 Factuality Detection

Factuality detection is an important task in practical applications. Hansen et
al. (2020) build an ensemble learner that predicts news headline factuality
using only eye-tracking measurements as they find that false headlines receive
statistically significantly less visual attention than true headlines. Meng et
al. (2020) propose a gradient-based adversarial training on transformer
networks to the task of detecting check-worthy claims. Zhong et al. (2020)
propose a graph-based reasoning approach utilizing factual structure of text
for deepfake detection. Zellers et al. (2020) propose a controllable text
generation model Grover to defend against fake news, which can generate fake
news that are more trustworthy than human-written disinformation.
Counterintuitively, the best defense against Grover turns out to be Grover
itself, with 92% accuracy versus 73% from best discriminators. As an important
factuality detection task, fake news detection has been widely explored in the
literature (Shu et al., 2017, 2019; Jain and Kasbe, 2018; Reis et al., 2019).

####  4.5.3 Constrained Text Generation

Generating text under specific lexical constraints is challenging, which can
help generate more faithful and factual texts. Constrained text generation
broadly falls into two categories, depending on whether inclusion of specified
keywords in the output is mandatory.

In soft-constrained generation (Qin et al., 2019; Tang et al., 2019), keyword-
text pairs are typically first constructed (sometimes along with other
conditioning information), and a conditional text generation model is trained
to capture their co-occurrence, so that the model learns to incorporate the
constrained keywords into the generated text. While soft constrained models
are easy to design, keywords are apt to be lost during generation, especially
when multiple keywords must be included, or the keywords are less correlated.
Soft enforcing algorithms such as attention and copy mechanisms (Bahdanau et
al., 2014; Gu et al., 2016) can be helpful in preserving keywords, but do not
guarantee that constraints will be included in the output sentence.

Hard-constrained generation (Hokamp and Liu, 2017; Post and Vilar, 2018; Hu et
al., 2019; Miao et al., 2019; Welleck et al., 2019a), on the other hand,
requires that all the lexical constraints be present in the output sentence.
This approach typically involve sophisticated design of network architectures.
Hokamp and Liu (2017) construct a lexical-constrained grid beam search
decoding algorithm to incorporate constraints. However, Hu et al. (2019)
observe that a naive implementation of this algorithm has a high running time
complexity. Miao et al. (2019) introduces a sampling-based conditional
generation method, where the constraints are first placed in a template, then
words in a random position are either inserted, deleted or updated under a
Metropolis-Hastings-like scheme. However, individually sampling each token
result in slow convergence, as the joint distribution of all the tokens in a
sentence is highly correlated. Welleck et al. (2019a) propose a tree-based
text generation scheme, where a token is first generated in an arbitrary
position, and then the model recursively generates words to its left and
right, yielding a binary tree. However, the constructed tree may not reflect
the progressive hierarchy/granularity from high-level concepts to low-level
details. Further, the time complexity of generating a sentence using this
approach is O(n), like standard auto-regressive generation methods.

Zhang et al. (2020c) propose a novel non-autoregressive model for hard-
constrained text generation, called POINTER (PrOgressive INsertion based
TransformER). Given lexical constraints, POINTER first generates high-level
words (e.g., informative nouns, verbs and adjectives) that bridge the keyword
constraints, then these words are used as pivoting points at which to insert
details of finer granularity. This process iterates until a sentence is
finally completed by adding the least informative words (typically pronouns
and prepositions).

Anderson et al. (2017) uses constrained beam search to force the inclusion of
selected tag words in image caption, and fixed, pre-trained word embeddings to
facilitate vocabulary expansion to previously unseen tag words. Constrained
beam search is an approximate search algorithm capable of enforcing any
constraints over resulting output sequences that can be expressed in a finite-
state machine.

####  4.5.4 Image Caption

Figure 13: Several examples of image captioning with unfaithful object
description.

Image caption is an important type of multimodality-to-text generation. Image
captioning models are prone to “hallucinating” objects that are not actually
in a scene. Several examples are shown in Figure 13. The standard evaluation
metrics only measure similarity to ground truth captions and cannot fully
capture image relevance. Rohrbach et al. (2018) propose a image relevance
metric to evaluate image captioning models with veridical visual labels and
assess their rate of object hallucination. Anderson et al. (2017) use
constrained beam search to force the inclusion of selected tag words in the
output, and fixed pre-trained word embeddings to facilitate vocabulary
expansion to previously unseen tag words.

Image-to-Text radiology report generation is a new type of image caption,
which is an important application of natural language generation (NLG). It is
to build assistive systems that take X-ray images of a patient and generate a
textual report describing clinical observations in the images (Jing et al.,
2017; Li et al., 2018c; Liu et al., 2019a; Boag et al., 2020; Chen et al.,
2020c). This is a clinically important task, offering the potential to reduce
radiologists’ repetitive work and generally improve clinical communication
(Kahn Jr et al., 2009). However, the automatic generated reports by neural
models are not always factually complete or consistent. They usually face the
issues of factual incompleteness and inconsistency (Liu et al., 2019a; Boag et
al., 2020).

Miura et al. (2021) show that existing image-to-text radiology report
generation systems can be substantially improved by replacing widely used NLG
metrics with simple alternatives. They propose two new simple rewards that can
encourage the factual completeness and consistency of the generated reports.
First, they propose the Exact Entity Match Reward (factENT) which captures the
completeness of a generated report by measuring its coverage of entities in
the radiology domain, compared with a reference report. The goal of the reward
is to better capture disease and anatomical knowledge that are encoded in the
entities. Second, they propose the Entailing Entity Match Reward (factENTNLI),
which extends factENT with a natural language inference (NLI) model that
further considers how inferentially consistent the generated entities are with
their descriptions in the reference. They add NLI to control the
overestimation of disease when optimizing towards factENT. They directly
optimizes these two rewards with RL, showing that the proposed approach
substantially improves performance on factual consistency and completeness.

##  5 Discussion

The faithfulness problem is the most critical challenge in modern NLG. As
described above, we have discussed evaluation metrics and optimization methods
for different NLG tasks. In the following, we discuss some of the limitations
of current evaluation and optimization methods, and provide several research
directions worth investigating in the general NLG.

###  5.1 Fine-grained and General Evaluation

Most existing faithfulness evaluation metrics measure the faithfulness of the
generated text as a score, such as entailment score or QA matching score. They
cannot distinguish between different types of factual errors and cannot locate
specific error spans, which are detrimental to robust evaluation and repair.
Moreover, most of them mainly focus on specific tasks, rather than be general
to all NLG tasks. More fine-grained and general evaluation methods are needed
to drive further developments in the field of NLG.

##### Intrinsic and Extrinsic

Most of the existing evaluation metrics measure intrinsic and extrinsic
factual errors as a unified metric without distinguishing them. We argue that
intrinsic and extrinsic factual errors should be evaluated separately, as they
have different definition and reference. The reference for the intrinsic error
is the source text, however, the reference for the extrinsic error include the
source text and world knowledge. As the extrinsic hallucinations contains both
factual and non-factual information, it is necessary to distinguish them as
the non-factual extrinsic hallucinations are harmful while the factual
extrinsic hallucinations are usually beneficial for NLG tasks, such as
dialogue generation. The field of fact checking is promising to help detect
non-factual extrinsic hallucinations.

##### Fine-grained Error Types

None of existing evaluation metrics can locate specific error spans and their
fine-grained error types, such as predicate error, entity error, circumstance
error or discourse link error. Fine-grained error types can help post-editing
methods to fix unfaithful content and also provide richer insight to the
researchers.

##### General Evaluation

Existing evaluation metrics are mainly designed for specific tasks, such as
text summarization or table-to-text generation. Although some of them can be
applied to other NLG tasks, task-agnostic general metrics and evaluation
benchmarks are lacking. Although the source and output texts of different
tasks come in various forms, it is worth exploring the relationship between
them and propose a general and fine-grained metric to evaluate faithfulness.
Task-agnostic metrics with cross-domain robustness can help the research
community to establish a unified benchmark, which is important and meaningful
to help collaborate and standardize evaluation metrics for NLG tasks.

###  5.2 Reasoning-based Optimization

Most intrinsic errors are caused by misunderstanding the facts in the source
context. Besides NLU-based pre-training on large scale corpus, to help models
understand the facts correctly requires reasoning over the input context or
world knowledge.

##### Numerical Reasoning

The correctness of the numerals in the generated texts such as date, quantity
and scalar are important for readers to get the correct information. As
existing models usually model numerals in the same way as textual tokens, such
as splitting numerals into sub-words or byte-pairs, they are more vulnerable
to numerical errors. However, most of the existing optimization methods do not
focus on the faithfulness of numerals. Tasks with quantities such as table-to-
text generation, require numerical reasoning. Therefore, adding reasoning
ability to numerical modeling is crucial for faithfulness optimization.

##### Grounded Language Representation

The expressiveness of language generation models are vital to the faithfulness
of the generated results. A model with poor input text representation will
fail to do document-level understanding and inference. Language grounding is