# Survey on Factuality in Large Language Models: Knowledge, Retrieval and
Domain-Specificity

Cunxiang Wang*, Xiaoze Liu*, Yuanhao Yue*, Xiangru Tang, Tianhang Zhang,
Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, Yidong Wang,
Linyi Yang, Jindong Wang, Xing Xie, Zheng Zhang, Yue Zhang The first three
authors contribute equally. Cunxiang Wang (wangcunxiang@westlake.edu.cn),
Wenyang Gao, Yidong Wang, Linyi Yang and Yue Zhang are with Westlake
University, Hangzhou, China. Xiaoze Liu is with Purdue University, IN, USA.
Yuanhao Yue is with Fudan University, Shanghai, China. Xiangru Tang is with
Yale University, CT, USA. Tianhang Zhang is with Shanghai Jiao Tong
University, Shanghai, China. Cheng Jiayang is with the Hong Kong University of
Science and Technology, Hong Kong SAR, China. Cunxiang Wang, Wenyang Gao and
Yunzhi Yao are with Zhejiang University, Hangzhou, China. Xuming Hu and Zehan
Qi are with Tsinghua University, Beijing, China. Jindong Wang and Xing Xie are
with Microsoft Research, Beijing, China. Zheng Zhang is with NYU Shanghai
University, Shanghai, China. Correspondence to: Yue Zhang
(zhangyue@westlake.edu.cn).

###### Abstract

This survey addresses the crucial issue of factuality in Large Language Models
(LLMs). As LLMs find applications across diverse domains, the reliability and
accuracy of their outputs become vital. We define the “factuality issue” as
the probability of LLMs to produce content inconsistent with established
facts. We first delve into the implications of these inaccuracies,
highlighting the potential consequences and challenges posed by factual errors
in LLM outputs. Subsequently, we analyze the mechanisms through which LLMs
store and process facts, seeking the primary causes of factual errors. Our
discussion then transitions to methodologies for evaluating LLM factuality,
emphasizing key metrics, benchmarks, and studies. We further explore
strategies for enhancing LLM factuality, including approaches tailored for
specific domains. We focus two primary LLM configurations—standalone LLMs and
Retrieval-Augmented LLMs that utilizes external data—we detail their unique
challenges and potential enhancements. Our survey offers a structured guide
for researchers aiming to fortify the factual reliability of LLMs. We
consistently maintain and update the related open-source materials at
https://github.com/wangcunxiang/LLM-Factuality-Survey.

###### Index Terms:

Large language Models, Factuality, Knowledge, Retrieval, Domain-Specificity,
Evaluation, Enhancement

##  1 Introduction

forked edges, for tree= grow=east, reversed=true, anchor=base west, parent
anchor=east, child anchor=west, base=center, font=, rectangle, draw=hidden-
draw, rounded corners, align=left, text centered, minimum width=5em,
edge+=darkgray, line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt,
line width=0.8pt, ver/.style=rotate=90, child anchor=north, parent
anchor=south, anchor=center, , where level=1text width=6em,font=,, where
level=2text width=9em,font=,, where level=3text width=9em,font=,, where
level=4text width=7em,font=,, [ Factuality [ The Issue
(Sec 2) [ Definition (Sec 2.2) ] [ Setting (Sec 2.2) [ Standalone LLMs [
ChatGPT (OpenAI, 2022b), GPT-4 (OpenAI, 2023),
BARD (Google, 2023)
, leaf, text width=22em ] ] [ Retrieval Aug-
mented LLMs [ BingChat (Microsoft, 2023), Llama-Index (Liu, 2022) , leaf, text
width=22em ] ] ] [ Impact (Sec 2.3) [ Legal: Curran et al. (2023), Pranshu
Verma (2023), Sands (2023) , leaf, text width=28em ] [ Medicine: Nori et al.
(2023), Shen et al. (2023), Kung et al. (2023) , leaf, text width=28em ] ] ] [
Evaluation
(Sec 3) [ Factuality Evaluation
Metrics 3.1 [ Rule-based evaluation metrics; Neural evaluation metrics;
Human evaluation metrics; LLM-based evaluation metrics, leaf, text width=33em
] ] [ Factuality Evaluation
Benchmark 3.2 [ MMLU (Hendrycks et al., 2021), TruthfulQA (Lin et al., 2022b),
C-Eval (Huang et al., 2023b), AGIEval (Zhong et al., 2023a),
RealTimeQA (Kasai et al., 2022), FreshQA (Vu et al., 2023), leaf, text
width=33em ] ] [ Factuality
Evaluation 3.3 [ SelfCheckGPT (Manakul et al., 2023), FActScore (Min et al.,
2023),
Wang et al. (2023a), Yu et al. (2023), Kadavath et al. (2022), leaf, text
width=33em ] ] [ Domain-Factuality
Evaluation (Sec 3.4) [ FLARE (Finance) (Xie et al., 2023b), CMB (Medicine)
(Wang et al., 2023d),
GeneTuring (Gene) (Jin et al., 2023), EcomInstruct (Ecom) (Li et al., 2023e),
leaf, text width=33em ] ] ] [ Analysis
(Sec 4) [ Factuality Analysis
(Sec 4.1) [ Knowledge Storage (Sec 4.1.1)
Knowledge Completeness and Awareness (Sec 4.1.2)
Contextual Influence and Knowledge Conflict (Sec 4.1.3), leaf, text width=27em
] ] [ Error Causes
(Sec 4.2) [ Model-Level 4.2.1 [ Domain Knowledge Deficit, Outdated
Information,
Immemorization, Forgetting, Reasoning Failure, leaf, text width=23em ] ] [
Retrieval-Level 4.2.2 [ Insufficient Information, Distracting Information,
Misinformation Not Recognized by LLMs,
Misinterpretation of Related Information, leaf, text width=23em ] ] [
Inference-Level 4.2.3 [ Snowballing, Erroneous Decoding, Exposure Bias, leaf,
text width=23em ] ] ] ] [ Enhancement
(Sec 5) [ On Standalone
LLMs (Sec 5.1) [ Pretrain-based 5.1 [ Initial Pretraining: Sadeq et al.
(2023);
Continual Pretraining: TOPICPREFIX (Lee et al., 2022b)
, leaf, text width=23em ] ] [ SFT-based 5.1.2 [ Continual SFT: Moiseev et al.
(2022);
Model Editing: Dong et al. (2022),
CaliNET (Huang et al., 2023c), KN (Dai et al., 2022),
T-Patcher (Dai et al., 2022), SERAC (Mitchell et al., 2022b)
, leaf, text width=23em ] ] [ Multi-Agent 5.1.3 [ Multi-debate: Du et al.
(2023)
Multi-role: Cohen et al. (2023b)
, leaf, text width=18em ] ] [ Decoding 5.1.5 [ Lee et al. (2022b), Chuang et
al. (2023)
, leaf, text width=18em ] ] ] [ On Retrieval-
Augmented
LLMs(Sec 5.2) [ Normal RAG 5.2.1) [ Borgeaud et al. (2022), Liu (2022)
, leaf, text width=18em ] ] [ Interactive
Retrieval 5.2.2 [ CoT-based path: He et al. (2022), IRCoT (Trivedi et al.,
2023)
Agent-based Retrieval: ReAct (Yao et al., 2023a) , leaf, text width=24em ] ] [
Retrieval
Adaptation 5.2.3 [ Prompt-based: LLM-Augmenter (Peng et al., 2023),
SFT-based: ATLAS (Izacard et al., 2022),
REPLUG (Shi et al., 2023), SAIL (Luo et al., 2023a)
RLHF-based: GopherCite (Menick et al., 2022a)
, leaf, text width=24em ] ] [ Retrieval on External
Memory 5.2.4 [ General Knowledge: Li et al. (2022c),
Houlsby et al. (2019), G-MAP (Wan et al., 2022)
Entity Knowledge: knowledge plugin (Zhang et al., 2023g)
KALA (Kang et al., 2022) TOME (de Jong et al., 2022),
, leaf, text width=24em ] ] [ Retrieval on
Structured Data 5.2.5 [ StructGPT (Jiang et al., 2023a) , Zhang et al. (2023d)
, leaf, text width=24em ] ] ] [ Domain Factuality
Enhanced LLMs
(Sec 5.3) [ Medicine: HuatuoGPT (Zhang et al., 2023a), DISC-MedLLM (Ross et
al., 2022)
Legal: LawGPT (Nguyen, 2023), ChatLaw (Cui et al., 2023a)
Finance: EcomGPT (Li et al., 2023e), BloombergGPT (Wu et al., 2023)
Education: GrammarGPT (Fan et al., 2023)
, leaf, text width=30em ] ] ] ]

Figure 1: Taxonomy of research on factuality in Large Language Models that
consists of the issue, evaluation, analysis and enhancement.

The quest for mastery of knowledge has been a foundational aspiration in the
development of artificial intelligence systems. Historically, seminal works by
McCarthy et al. (1955) and Newell and Simon (1976) have underscored the
significance of knowledge representation and reasoning in AI systems. For
instance, the Cyc project embarked on an ambitious journey to codify common-
sense knowledge, aiming to provide AI systems with a comprehensive
understanding of the world (Lenat, 1995). Concurrently, endeavors like the
WordNet project by Miller et al. (1990) sought to create lexical databases
that capture semantic relationships between words, thereby aiding AI systems
in grasping the nuances of human language.

Amidst these pioneering efforts, the emergence of Large Language Models
(LLMs), such as ChatGPT (OpenAI, 2022b), GPT-4 (OpenAI, 2023) and LLaMA
(Touvron et al., 2023a, b), has been seen as a significant leap in both
academics and industries, especially towards AI systems possessing vast
factual knowledge (Petroni et al., 2019a; De Cao et al., 2021; OpenAI, 2023).
The advantages of using LLMs as knowledge bases carriers are manifold.
Firstly, they reduce the overhead and costs associated with building and
maintaining dedicated knowledge bases (Petroni et al., 2019c; AlKhamissi et
al., 2022; Wang et al., 2023b). Additionally, LLMs offer a more flexible
approach to knowledge processing and utilization, allowing for context-aware
reasoning and the ability to adapt to novel information or prompts (Sun et
al., 2023a; Huang and Chang, 2023). Yet, with their unparalleled capabilities,
concerns have arisen about the potential of LLMs to generate non-factual or
misleading content (Bender et al., 2021; OpenAI, 2023; Bubeck et al., 2023).
In light of these advancements and challenges, this survey seeks to delve
deeply into the LLMs, exploring both their potential and the concerns
surrounding their factual accuracy.

Understanding the factuality of Large Language Models is more than just a
technical challenge; it’s essential for the responsible use of these tools in
our daily lives. As LLMs become more integrated into services like search
engines (Microsoft, 2023), chatbots (OpenAI, 2022b; Google, 2023), and content
generators (Cui et al., 2023b), the information they provide directly
influences decisions, beliefs, and actions of millions of people. If an LLM
provides incorrect or misleading information, it can lead to
misunderstandings, spread false beliefs, or even cause harm, especially for
those domains that demand high factual accuracy (Ling et al., 2023b), such as
health (Thirunavukarasu et al., 2023; Tang et al., 2023), law (Huang et al.,
2023a), and finance (Wu et al., 2023). For instance, a physician relying on an
LLM for medical guidance might inadvertently jeopardize patient health, a
corporation leveraging LLM insights might make ill-informed market decisions,
or an attorney misinformed by an LLM might falter in legal proceedings (Curran
et al., 2023). In addition, with the advancement of LLM-based agents, the
factuality of LLMs is becoming even more potent. A driver or an autonomous
driving car might rely on LLM-based agents for planning or driving, where
serious factual mistakes made by LLMs could cause irreversible damage. By
studying the factuality of LLMs, we aim to ensure that these models are both
powerful and trustworthy.

A surge of research has been directed towards evaluating LLMs’ factuality,
which encompasses diverse tasks like factoid question answering and fact
checking. Beyond evaluation, efforts to improve the factual knowledge of LLMs
have been notable. Strategies have ranged from retrieving information from
external knowledge bases to continual pretraining and supervised finetuning.
Yet, despite these burgeoning efforts, a holistic overview that covers the
full spectrum of factuality in LLMs remains elusive. While there are existing
surveys in the field, such as those by Chang et al. (2023) and Wang et al.
(2023g), that delve into the evaluation of LLMs and their factuality, they
only scratch the surface of the broader landscape. There are also a bunch of
recent studies focusing on hallucinations in LLMs (Rawte et al., 2023; Zhang
et al., 2023f; Rawte et al., 2023; Ye et al., 2023). But we differentiate
between the hallucination issue and the factuality issues in Sec 2.2.
Moreover, these surveys often overlook key areas we emphasize, like domain-
specific factuality or the challenge of outdated information. While Ling et
al. (2023a) explores domain specialization in LLMs, our survey takes a more
expansive look at the broader issues of factuality. To the best of our
understanding, our work is the first comprehensive study on the factuality of
large language models.

This survey aims to offer an exhaustive overview of the factuality studies in
LLMs, delving into four key dimensions: Sec 2) The definition and impact of
the factuality issue (Pranshu Verma, 2023; Nori et al., 2023); Sec 3)
Techniques for evaluating factuality and its quantitative assessment (Huang et
al., 2023b; Min et al., 2023); Sec 4) Analyzing the underlying mechanisms of
factuality in LLMs and identifying the root causes of factual errors (Liu et
al., 2023c; Kotha et al., 2023); and Sec 5) Approaches to enhance the
factuality of LLMs (Du et al., 2023; He et al., 2022). Notably, we categorize
the use of LLMs into two primary settings: LLMs without external knowledge,
such as ChatGPT (OpenAI, 2022b) and Retrieval-Augmented LLMs, such as BingChat
(Microsoft, 2023). The complete structure of this survey is illustrated in
Figure 1. Through a detailed examination of existing research, we seek to shed
light on this critical aspect of LLMs, helping researchers, developers, and
users harness the power of these models responsibly and effectively.

##  2 Factuality Issue

In this section, we describe the issue of factuality in large language models,
as well as the impact.

###  2.1 Large Language Models

There is no well-accepted and exact definition of large language models in the
literature (Chang et al., 2023; Zhao et al., 2023b; Huang and Chang, 2022). We
mainly consider the decoder-only generative pre-trained language modes with
emergent abilities, such as ChatGPT (OpenAI, 2022b) and LLaMA (Touvron et al.,
2023a, b). We also include some work that is based on models with encoder-
decoder architectures, such as T5 (Raffel et al., 2020a). We do not talk about
work that is only based on the Encoder-only models, such as BERT (Devlin et
al., 2019) and RoBERTa (Liu et al., 2019), in this survey. To be specific, our
survey includes the following LLMs:

  * •

General Domain LLMs: GPT-2 (Radford et al., 2019), GPT-3 (Brown et al., 2020),
ChatGPT (OpenAI, 2022b), GPT-4 (OpenAI, 2023), GPT-Neo (Black et al., 2021),
OPT (Zhang et al., 2022b), LLaMA (Touvron et al., 2023a), LLaMA-2 (Touvron et
al., 2023b), Incite (Computer, 2023; Together, 2023), Claude (Cla, ), Falcon
(Almazrouei et al., 2023), MPT Team (2023b), Vicuna (Chiang et al., 2023),
FLAN-T5 Chung et al. (2022), BLOOM (Scao et al., 2022), Baichuan & Baichuan2
(Yang et al., 2023a), PaLM (Chowdhery et al., 2022), Gopher (Rae et al.,
2022a), Megatron-LM (Shoeybi et al., 2019), SAIL (Luo et al., 2023a), Codex
(Chen et al., 2021a), Bard (Google, 2023), GLM & ChatGLM (Zeng et al., 2022),
InternLM (Team, 2023a), StableBeluga (Mahan et al., ), Claude (Cla, ), Alpaca
(Taori et al., 2023), New Bing (Microsoft, 2023), Ziya-LLaMA (Zhang et al.,
2022a), BLOOMZ (Muennighoff et al., 2022), Chinese-LLaMA (Cui et al., 2023c),
Phoenix (Chen et al., 2023e), and others.

  * •

Domain-specify LLMs: BloombergGPT (Wu et al., 2023), EcomGPT (Li et al.,
2023e), BioGPT (Luo et al., 2022), LawGPT (Nguyen, 2023), Lawyer LLaMA (Huang
et al., 2023a), ChatLaw (Cui et al., 2023b), BioMedLM (Venigalla et al.,
2022), HuatuoGPT (Zhang et al., 2023a), ChatDoctor (Li et al., 2023f),
MedicalGPT (Xu, 2023), Bentsao (Huatuo as its original name) (Wang et al.,
2023c), Zhongjing (Yang et al., 2023c), LLM-AMT (Wang et al., 2023f), DISC-
MedLLM (Bao et al., 2023), Cohortgpt (Guan et al., 2023), Deid-gpt (Liu et
al., 2023f), Doctorglm (Xiong et al., 2023b), MedChatZH (Tan et al., 2023a),
K2 (Deng et al., 2023), HouYi (Bai et al., 2023), GrammarGPT (Fan et al.,
2023), FoodGPT (Qi et al., 2023), ChatHome (Wen et al., 2023), and others.

###  2.2 Factuality

TABLE I: Examples of different kinds of factual error produced by large
language models. We category the factual error types by the causes of them,
whose details can be found in Sec 4.2

.  Category Cause Example Dialog Notes and references Domain knowledge deficit
Q: CEO of Assicurazioni Generali? BloombergGPT: Philippe Donnet GPT-NeoX:
Antonio De Lorenzo, Simone Gambarini, Enrico Zanetti FLAN-T5-XXL: John M
Forsyth, Christopher K Peters, {empty string} BloombergGPT is a finance
domain-specific language model. Wu et al. (2023) Outdated information Q: When
was Kyiv attacked by Russia? ChatGPT:As of my last knowledge update in
September 2021, Russia had not launched an attack on Kyiv. Kyiv was attacked
by Russia on 25 February 2022. Model- level causes Reasoning error Q: Who is
Tom Cruise’s mother? A: Mary Lee Pfeiffer Q: Who is Mary Lee Pfeiffer’s son?
A: There is no widely known information about… From Berglund et al. (2023). It
is clear that the model knows Tom Cruise’s mother is Lee Pfeiffer, but it
fails to reason that Lee Pfeiffer has a son named Tom Cruise. Distracted by
the retrieval information Retrieved context: _“Alan Melikdjanian - Age, Bio,
Faces and Birthday”_ Melikdjanian was the Director, Writer (with his father),
Editor, and Composer of the 2006 direct-to-DVD Russian-language film Citizen
Mavzik, the story of a Russian immigrant family’s assimilation into American
culture. The film was produced by Vilalan Productions, named after
Melikdjanian and his father Vilen. Q: What is the nationality of the director
of Citizen Mavzik? A: (1) The director of the film Citizen Mavzik is Alan
Melikdjanian. (2) Alan Melikdjanian is from Russia. So, the answer is Russia.
Alan Melikdjanian, known by the alias Captain Disillusion, is a Soviet-born
American independent filmmaker and YouTuber (Melikdjanian, 2018).
Misunderstood the retrieval information Retrieved context: _“Gerard III, Count
of Guelders - Wikipedia”_ Gerard married Margaretha of Brabant, the daughter
of Duke Hendrik I of Brabant and Matilda of Flanders, Duchess of Brabant, in
1206. Gerard and Margaret’s Tomb … Q: Who is Gerard I, Count Of Guelders’s
wife? A: … the name of Gerard I, Count of Guelders’s wife, which is Margaretha
of Brabant. The retrieved context accurately provided the name of Gerard III,
Count of Guelders’ wife, which is Margaretha of Brabant. However, the model
erroneously referred to him as Gerard I, resulting in an incorrect answer.
Retrieval- level causes Fail to address misinformation in retrieved documents
Retrieved context: _[0] “James Vincent - Death Notice and Service Information
- Legacy.com”_ James Vincent died on August 9, 2023 in Long Beach, California
Obituary. James Joseph Vincent was the perfect husband, father, grandfather,…
_[1] “UPDATE: Jury convicts James Vincent of murder after deliberating ”_ A
jury convicted James Vincent of murder in the 2018 shooting death of Luka
Grabb. The jury reached its verdict at 4:30 p.m. Friday. Q: When did James
Vincent die? A: Based on the information provided in the two articles, James
Vincent died on August 9, 2023, in Long Beach, California. In the presence of
contradictory information from retrieved documents, the model should
thoroughly outline the response, while also noting the need for clarity. For
instance, the model could state: “According to the first document, James
Vincent passed away on August 9, 2023, in Long Beach. However, the second
document suggests that James Vincent was murdered in 2018.” Snowballing Q: Was
there ever a US senator who represented the state of New Hampshire and whose
alma mater was the University of Pennsylvania? A: Yes… His name was John P.
Hale John P. Hale was graduated from Bowdoin College. Zhang et al. (2023b)
Inference- level causes Exposure bias Q: Aamari was very stoic. [PRONOUN]
rarely showed any emotion. A: He. The correct answer was Xe according to
Hossain et al. (2023).

By factuality in LLMs we refer to the capability of large language models for
generating contents that follow factual information, which encompasses
commonsense, world knowledge and domain facts. The source of such factual
information can be grounded to dictionaries, Wikipedia or textbooks from
different domains 111We only consider cases where the meaning is clear and the
truthfulness can be determined in this survey. Furthermore, we only take into
account undisputed facts. Minor errors that may exist in reliable sources are
not within the scope of our consideration. . A series of work have discussed
whether LLMs can serve as knowledge bases to store factual knowledge
(AlKhamissi et al., 2022; Yu et al., 2023; Pan et al., 2023b)

TABLE II: Comparison between the factuality issue and the hallucination issue. Factual and Non-Hallucinated | Factually correct outputs.
---|---
Non-Factual and Hallucinated | Entirely fabricated outputs.
Hallucinated but Factual |  | 1\. Outputs that are unfaithful to the prompt but remain factually correct (Cao et al., 2022).
---
2\. Outputs that deviate from the prompt’s specifics but don’t touch on
factuality, e.g.,
a prompt asking for a story about a rabbit and wolf becoming friends, but the
LLM
produces a tale about a rabbit and a dog befriending each other.
3\. Outputs that provide additional factual details not specified in the
prompt, e.g.,
a prompt asking about the capital of France, and the LLM responds with ”Paris,
which
is known for the Eiffel Tower.”
Non-Factual but Non-Hallucinated |  | 1\. Outputs where the LLM states“I don’t know” or avoids a direct answer.
---
2\. Outputs that are partially correct, e.g., for the question, ”Who landed on
the moon
with Apollo 11?” If the LLM responds with just ”Neil Armstrong,” the answer is
incomplete but not hallucinated.
3\. Outputs that provide a generalized or vague response without specific
details, e.g.,
for a question about the causes of World War II, the LLM might respond with
“It was
due to various political and economic factors.”

Existing work focus on measuring factuality in LLMs qualitatively (Lin et al.,
2022b; Chern et al., 2023), discussing the mechanism for storing knowledge
(Meng et al., 2022; Chen et al., 2023d) and tracing the source of knowledge
issues (Gou et al., 2023; Kandpal et al., 2023). The factuality issue for LLMs
receive relatively the most attention. Several instances are shown in Table I.
For instance, an LLM might be deficient in domain-specific factual knowledge,
such as medicine or law domain. Additionally, the LLM might be unaware of
facts that occurred post its last update. There are also instances where the
LLM, despite possessing the relevant facts, fails to reason out the correct
answer. In some cases, it might even forget or be unable to recall facts it
has previously learned. The factuality problem is closely related to several
hot topics in the field of Large Language Models, including Hallucinations (Ji
et al., 2023a; Zhao et al., 2023b), Outdated Information (Nakano et al., 2022;
Qin et al., 2023), and Domain-Specificity (e.g., Health (Xiong et al., 2023a;
Wang et al., 2023c), Law (Cui et al., 2023b), Finance (Wu et al., 2023)). At
their core, these topics address the same issue: the potential for LLMs to
generate contents that contradicts certain facts, whether those contents arise
out of thin air, outdated information, or a lack of domain-specific knowledge.

Therefore, we consider these three topics to fall within the scope of the
factuality problem. However, it is important to note that while these topics
are related, they each have a unique focus. Both hallucinations and factuality
issues in LLMs pertain to the accuracy and reliability of generated content,
they address distinct aspects. Hallucinations primarily revolve around LLMs
generating baseless or unwarranted content. Drawing from definitions by OpenAI
(2023); Ji et al. (2023a), hallucinations can be understood as the model’s
inclination to ”produce content that is nonsensical or untruthful in relation
to certain sources.” This is different from factuality concerns, which
emphasize the model’s ability to learn, acquire, and utilize factual
knowledge. To illustrate the distinction: If an LLM, when prompted to craft ”a
fairy tale about a rabbit and a wolf making friends,” produces a tale about ”a
rabbit and a dog becoming friends,” it’s exhibiting hallucination. However,
this isn’t necessarily a factuality error. If the generated content contains
accurate information but diverges from the prompt’s specifics, it’s a
hallucination but not a factuality issue. For instance, if the LLM’s output
includes more details or different elements than the prompt specifies but
remains factually correct, it’s a case of hallucination. Conversely, if the
LLM avoids giving a direct answer, states ”I don’t know,” or provides a
response that’s accurate but omits some correct details, it’s addressing
factuality, not hallucination. Furthermore, it’s worth noting that
hallucination can sometimes produce content that, while deviating from the
original input, remains factually accurate. For a more structured comparison
between the factuality issue and hallucination, refer to Table II. Outdated
information, on the other hand, focuses on instances where previously accurate
information has been superseded by more recent knowledge. Lastly, domain-
specificity emphasize the generation of content that requires specific,
specialized knowledge. Despite these differences, all three topics contribute
to our understanding of the broader factuality problem in LLMs.

##### Setting

In this survey, our primary focus is on two specific settings: 1\. Standard
LLMs: Directly using LLMs for answering and chatting (OpenAI, 2023, 2022b);
2\. Retrieval-Augmented LLMs: The retrieval-augmented generation (Microsoft,
2023; Liu, 2022). The latter is of particular interest as retrieval mechanisms
are among the most prevalent methods for enhancing the factuality of LLMs.
This involves not just generating accurate responses but also correctly
selecting pertinent knowledge snippets from the myriad of retrieved sources.

While summarization tasks—where the goal is to produce summaries that stay
true to the source input—have seen research on factuality (Maynez et al.,
2020; Tam et al., 2023; Tang et al., 2022), we opted not to focus heavily on
this domain in our survey. There are a few reasons for this decision. Firstly,
the source inputs for summarization often contain content that is not factual.
Secondly, summarization introduces unique challenges like ensuring coherence,
conciseness, and relevance, which deviate from the focus of this survey. It is
also worth noting that Pu et al. (2023) found LLMs to produce fewer factual
errors or hallucinations compared to humans across various summarization
benchmarks. However, we will still discuss some works in this area,
particularly those that overlap with retrieval settings.

###  2.3 Impact

The factuality problem significantly impacts the usability of LLMs. Some of
these issues have even led to losses at the societal or economic level
(Pranshu Verma, 2023; Sands, 2023), drawing the attention of many users,
developers, and researchers (Ji et al., 2023a).

Factuality issues also impacted the legal field, with a lawyer in the United
States facing sanctions for submitting hallucinated case law in court. One
court has mandated that lawyers indicate the portions generated by generative
AI in their submitted materials (Curran et al., 2023). In addition, as part of
a research study, a fellow lawyer asked ChatGPT to generate a list of legal
scholars with a history of sexual harassment. ChatGPT generated a list that
included a law professor. ChatGPT claimed that the professor attempted to
touch a student on a class tripnd referenced an article from The Washington
Post in March 2018. However, the fact is that this article does not exist, nor
does the mentioned class trip (Pranshu Verma, 2023). Besides, a mayor in
Australia, discovered false claims made by ChatGPT stating that he was
personally convicted of bribery, confessed to charges of bribery and
corruption, and received a prison sentence. In response, he plans to initiate
legal action against the company responsible for ChatGPT, accusing them of
defamation for disseminating untrue information about him. This could
potentially be the first defamation case of its kind involving an artificial
intelligence chatbot (Sands, 2023).

A recent study (Nori et al., 2023) provides a comprehensive evaluation of
GPT-4’s performance in medical competency examinations and benchmark datasets.
The evaluation utilizes the text-only version of GPT-4 and investigates its
ability to address medical questions without any training or fine-tuning. The
assessment is conducted using the United States Medical Licensing Examination
(USMLE) Kung et al. (2023) and the MultiMedQA benchmark Singhal et al. (2023),
comparing GPT-4’s performance against earlier models like GPT-3.5 and models
specifically fine-tuned on medical knowledge. The results demonstrate that
GPT-4 significantly outperforms its predecessors, achieving scores on the
USMLE that exceed the passing threshold by more than 20 points and delivering
the best overall performance without specialized prompt crafting or domain-
specific fine-tuning.

While large language models show promise on medical datasets, the introduction
of automation in the healthcare field still requires extreme caution Shen et
al. (2023). Existing metrics and benchmarks are often developed for highly
focused problems. Evaluating LLM outputs in supporting real-world decision-
making poses challenges Singhal et al. (2023), including the stability and
robustness of personalized recommendations and inferences in real-world
contexts. Using large language models carries significant risks, including
inaccurate ranking recommendations Hirosawa et al. (2023) (e.g., differential
diagnosis) and sequencing Zhang et al. (2023c) (e.g., information gathering
and testing), as well as factual errors Shen et al. (2023), particularly
important omissions and erroneous responses.

##  3 Factuality Evaluation

Evaluating the factuality of LLMs is pivotal for ensuring the reliability and
trustworthiness of their generated content (Pezeshkpour, 2023; Lee et al.,
2022b). As LLMs become increasingly integrated into various applications, from
information retrieval to content generation, the accuracy of their outputs
becomes paramount. In this section, we delve into the evaluation metrics and
benchmarks used for assessing the factuality of LLMs, studies that have
undertaken such evaluations, and domain-specific evaluation.

###  3.1 Factuality Evaluation Metrics

TABLE III: Evaluation Metrics for the Factuality of LLMs. Evaluation Type | Metric | Description, Purpose/Usage, and Reference/Findings
---|---|---
Rule-based | Exact Match | Correct matches between generated and input/reference text.
| Common Metrics
---
(Accuracy, Precision, Recall,
F-Measure, Calibration, Brier)
|  Accuracy: Measures overall correctness of model prediction.
---
Precision: Ability of model to classify instances positively.
Recall: Ability of model to identify all positive instances.
F-Measure: Combines both precision and recall into a single score.
Calibration: Agreement between predicted probabilities and observed
frequencies.
Brier: Measures accuracy of probabilistic predictions.
MC1 and MC2 (Lin et al., 2022b) | Used to identify the correct answer from multi-choice questions.
BLEU (Papineni et al., 2002) | Determines frequency of phrase co-occurrence in two sentences.
ROUGE (Lin, 2004) | Calculates similarity between reference and generated text.
METEOR(Lowe et al., 2017) | Comprehensive measure based on harmonic mean of unigram precision and recall.
QUIP-Score (Weller et al., 2023) | Calculates how much of generated text consists of exact spans found in a text corpus.
Neural | ADEM Lowe et al. (2017) |  | Uses a Hierarchical RNN to evaluate the quality of responses in language models.
---
BERTScore (Zhang et al., 2020) |  | Pre-trained embeddings from BERT used to evaluate sentence similarity.
---
BLEURT (Sellam et al., 2020) |  | Trains BERT on larger dataset and rate similarity of sentences .
---
BARTScore (Yuan et al., 2021) |  | Evaluates quality using pre-trained sequence-to-sequence models.
---
Human |  | AIS, Auto-AIS
---
(Rashkin et al., 2023)
(Gao et al., 2023a)
| Evaluates if output is backed by evidence.
---
FActScore (Min et al., 2023) |  | Measures factual accuracy of LLMs breakpointing generated content into atomic facts.
---
LLM-based | GPTScore (Fu et al., 2023) |  | Evaluates quality of AI output. It is efficient and avoids the annotation requirement.
---
GPT-judge (Lin et al., 2022b) |  | Evaluates truthfulness of LLM answers. It is used in evaluating truthfulness
---
and informativeness.
| Truthfulness and
---
Informativeness
(Lin et al., 2022b)
| Truthfulness Measures the honesty of LLM information.
---
It is used to evaluate the factuality of information.
Informativeness Evaluates the relevance and value of LLM responses
LLM-Eval (Lin and Chen, 2023) |  | Evaluates the quality of a conversation. It is adaptable in various scenarios.
---

In this subsection, we delve into metrics established for evaluating the
factuality of LLMs. As the problem formulation is akin to natural language
generation (NLG) (Celikyilmaz et al., 2021; Ji et al., 2023b), we introduce
several automatic evaluation metrics typically used for NLG, as well as
specifically examining the metrics for factuality.

We categorize these metrics into the following groups: (1) Rule-based
evaluation metrics; (2) Neural evaluation metrics; (3) Human evaluation
metrics; and (4) LLM-based evaluation metrics. We list those metrics in Table
III.

####  3.1.1 Rule-based evaluation metrics

Most assessments of factuality in large language models use rule-based
evaluation metrics, due to their consistency, predictability, and ease of
implementation; they allow for reproducible outcomes through a systematic
method. However, they can be rigid and may not account for nuances or
variations in language use, context interpretation, or colloquial expressions.
This means language models rated highly by these metrics may still produce
content that feels unnatural or inauthentic to human readers.

##### Exact Match

An “exact match” refers to a situation where the generated text precisely
matches a specific input or reference text. This means that the LLM produces
output that is identical, word-for-word, to the provided input or reference
text. Exact matches are often used in NLG when you want to replicate or repeat
a piece of text without any variations or alterations.

##### Common metrics

Many factuality evaluation measurements use commonly adapted metrics, such as
Accuracy, Precision, Recall, AUC, F-Measure, Calibration score, Brier score,
and other common metrics used in probabilistic forecasting and machine
learning, specifically in tasks involving probabilistic predictions. The
common definition of these metrics involves the use of correctly predicted
labels and ground-truth labels. As the input and output of LLMs are human-
readable sentences, there is no unified method to convert the sentences into
labels. Most evaluation will define their own way. That is, these scores are
frequently not used in isolation, but rather in combination. For instance,
BERTScore (Zhang* et al., 2020) uses BERT for determining the Precision and
Recall, then uses F-measures to get a final weighted score. In the following,
we will describe the most simple form of these scores.

The _Calibration score_ , used in (Lin et al., 2022a; Kadavath et al., 2022)
measures the agreement between predicted probabilities and observed
frequencies. A perfectly calibrated model should, over a large number of
instances, see the predicted probability of an outcome match the relative
frequency of that outcome.

The _Brier Score_ , used in (Kadavath et al., 2022) is a metric used in
probabilistic forecasting to measure the accuracy of probabilistic
predictions. It calculates the mean squared difference between the predicted
probability assigned to an event and the actual outcome of the event. The
Brier Score ranges from 0 to 1, where 0 indicates a perfect prediction and 1
indicates the worst possible prediction. In other words, the lower the Brier
Score, the better the accuracy of the prediction. It’s worth noting that this
metric is appropriate for binary and categorical outcomes, but not for ordinal
outcomes. For binary outcomes, the Brier Score can be calculated as follows:

$BS=\frac{1}{N}\sum_{i=1}^{N}(forecast_{i}-actual_{i})^{2}$ (1)

where $forecast_{i}$ is the predicted probability, $actual_{i}$ is the actual
outcome (0 or 1), and $N$ is the total number of forecasts made.

##### MC1 (Single-true) and MC2 (Multi-true)

are widely recognized metrics in multi-choice question answering, particularly
in TruthfulQA (Lin et al., 2022b). MC1: For a given question accompanied by
several answer choices, the objective is to identify the sole correct answer.
The model’s selection is determined by the answer choice to which it allocates
the highest log probability of completion, independent of the other choices.
The score is calculated as the straightforward accuracy across all questions.
MC2: Presented with a question and multiple reference answers labeled as true
or false, the score is derived from the normalized total probability assigned
to the collection of true answers.

##### BLEU

(Papineni et al., 2002), also known as Bilingual Evaluation Understudy metric
is commonly employed in the context of factuality evaluation. This metric
calculates the co-occurrence frequency of phrases in two sentences, based on a
weighted average of matched n-gram phrases. This helps in quantitatively
assessing the factual consistency between the generated text and its
reference. The BLEU score is computed as

$BLEU=BP*\exp\left(\frac{1}{N}*\sum_{n=1}^{N}P_{n}\right)$ (2)

where _(1)_ $BP$ (Brevity Penalty) accounts for the length of the candidate
translation. If the candidate translation is shorter than the reference, $BP$
is less than 1, which reduces the BLEU score. _(2)_ $P_{n}$ is the n-gram
precision, which measures the portion of n-grams in the candidate translation
that are also present in the reference translation. _(3)_ $N$ is the maximum
order of n-grams considered (commonly up to 4 in much of the literature) and
$\sum_{n=1}^{N}P_{n}$ is the sum of $\log P_{n}$ for $n$ from 1 to $N$. \-
$\exp$ denotes the exponential function.

##### ROUGE

The Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metric (Lin,
2004) serves as a measure of the similarity between the generated text and the
reference text, with the similarity grounded on recall scores. Primarily used
in the field of text summarization, the ROUGE metric incorporates four
distinct types. These include ROUGE-n, which assesses n-gram co-occurrence
statistics, and ROUGE-l, which measures the longest common subsequence.
ROUGE-w provides evaluation based on the weighted longest common subsequence,
while ROUGE-s measures skip-bigram co-occurrence statistics. These diverse
metrics collectively provide a comprehensive measure of the factual accuracy
of generated text. ROUGE score can be calculated in various ways based on the
length of n-grams (unigram, bigram, etc.) used. The simplest version, ROUGE-n
score can be compactly represented as:

$\text{ROUGE-n}=\frac{\sum_{S\in RS}\sum_{gram_{n}\in
S}Count_{match}(gram_{n})}{\sum_{S\in RS}\sum_{gram_{n}\in S}Count(gram_{n})}$
(3)

where: _(1)_ $RS$ denotes the set of Reference Summaries, _(2)_ $gram_{n}$
represents an n-gram within the reference summary $S$, _(3)_
$Count_{match}(gram_{n})$ signifies the number of times that n-gram $gram_{n}$
appears in both the generated text and the reference summary, _(4)_
$Count(gram_{n})$ is the frequency of occurrence of the n-gram $gram_{n}$
within the reference summary.

##### METEOR

The Metric for Evaluation of Translation with Explicit Ordering (METEOR)
(Banerjee and Lavie, 2005) aims to address several shortcomings presented by
BLEU. These include deficiencies in recall, the absence of higher order
n-grams, an absence of explicit word-matching between the generated and
reference text, and the use of geometric averaging of n-grams. METEOR
overcomes these by introducing a comprehensive measure, calculated based on
the harmonic mean of the unigram precision and recall. This offers potentially
enhanced appraisals of factuality in the generated text.

##### QUIP-Score

(Weller et al., 2023) is an n-gram overlap measure. It quantifies the degree
to which a generated passage consists of exact spans found in a text corpus.
The QUIP-Score serves to evaluate the ’grounding’ ability of LLMs,
specifically assessing whether model-generated answers can be directly located
within the underlying text corpus. It is defined by comparing the precision of
the character n-gram from the generated output to the pre-training corpus.
This is formally illustrated by generation $Y$ and the text corpus $C$:

$\operatorname{QUIP}(Y;C)=\frac{\sum_{\operatorname{gram}_{n}\in
Y}\mathbb{1}_{C}\left(\operatorname{gram}_{n}\right)}{\left|\operatorname{gram}_{n}\in
Y\right|},$ (4)

where $\mathbb{1}($.$)$is an indicator function:

$\mathbb{1}(.)=\begin{cases}1,&\text{if }\text{gram}\in C\\\
0,&\text{otherwise}\end{cases}$ (5)

Therefore, a score of 0.5 implies that 50% of the n-grams derived from the
generated text can be found in the pre-training corpus. The authors calculate
a macro-average of this value over a collection of generations, which results
in a single performance figure representative of a specific test dataset.

####  3.1.2 Neural Evaluation Metrics

These metrics operate by comparing the output of these models with a standard
or reference text by learning evaluator models. This category primarily
comprises three prominent metrics, namely ADEM, BLEURT, and BERTScore. Each
metric approaches the evaluation slightly differently, yet all aim to assess
the semantic and lexical alignment between machine-generated text and its
reference counterpart, thus ensuring the factuality of the generated content.

##### ADEM

The automatic Dialogue Evaluation Model (ADEM) metric is a cogent tool
utilized to gauge the quality of responses generated by language models in a
conversation. Developed by Lowe et al. (2017), this metric trains a
Hierarchical RNN, in a semi-supervised fashion, to predict ratings for the
machine-generated responses. The ADEM’s assessment primarily hinges on a
dialogue context, designated as $\mathbf{c}$, alongside the model’s response,
labeled as $\mathbf{\hat{r}}$, and a reference response, specified as
$\mathbf{r}$. These elements, when encoded via the hierarchical RNN, inform
the ADEM which then predicts a score, reflecting the proximity of the model’s
response to factual accuracy and relevance:

$score=(\mathbf{c}^{\top}M\mathbf{\hat{r}}+\mathbf{r}^{\top}N\mathbf{\hat{r}}-\alpha)/\beta,$
(6)

where $M,N$ are learnable parameters and $\alpha,\beta$, represent scalar
constants that serve as initial values.

##### BERTScore

the BERTScore metric, as introduced by Zhang et al. (2020), utilizes pre-
trained embeddings from BERT to gauge the similarity between two sentences.
This is done by assigning embeddings to a referenced sentence, ”x”, and a
model-generated sentence, ”$\hat{x}$”, denoted as ”$\mathbf{x}$” and
”$\mathbf{\hat{x}}$”, respectively. The recall, precision, and F1-scores are
then calculated to quantify the similarity between ”$\mathbf{x}$” and
”$\mathbf{\hat{x}}$”. This score provides a measure of the generated
sentence’s factuality, and thus, the reliability of the large language model
itself:

$\displaystyle R_{\rm BERT}$ $\displaystyle=\frac{1}{|x|}\sum_{x_{i}\in
x}\max_{\hat{x}_{j}\in\hat{x}}\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j},$ (7)
$\displaystyle P_{\rm BERT}$
$\displaystyle=\frac{1}{|\hat{x}|}\sum_{\hat{x}_{j}\in\hat{x}}\max_{x_{i}\in
x}\mathbf{x}_{i}^{\top}\mathbf{\hat{x}}_{j},$ (8) $\displaystyle F_{\rm BERT}$
$\displaystyle=2\frac{P_{\rm BERT}\cdot R_{\rm BERT}}{P_{\rm BERT}+R_{\rm
BERT}},$ (9)

where the recall and precision elements are calculated based on a token-match
approach. The recall score stems from comparing each token in the reference
sentence ”$x$” to the corresponding token in the generated sentence
”$\hat{x}$”. Conversely, the precision score is derived by comparing each
token in ”$\hat{x}$” to a token in ”$x$”. A greedy matching strategy is
employed to pair tokens that demonstrate the highest degree of similarity.
This method provides a comprehensive analysis of how precise and accurate the
language model’s output aligns with the factual reference sentence.

##### BLEURT

the Bilingual Evaluation Understudy with Representations from Transformers
(BLEURT) metric (Sellam et al., 2020) employs a unique pre-training scheme,
where BERT is initially trained on a significant corpus of synthetic sentence
pairs. This is reinforced by multiple lexical and semantic-level supervision
signals, used concurrently. Following this pre-training stage, BERT is further
fine-tuned on rating data, and its objective is to estimate human rating
scores accurately. The initial stage of pre-training is essential for this
metric because it significantly enhances the model’s robustness, thereby
effectively transforming it into a secure bulwark against quality drifts
inherent in generative systems.

##### BARTScore

(Yuan et al., 2021) is a metric proposed to evaluate the quality of the
generated text, such as in applications like machine translation and
summarization. This metric conceives this evaluation as a problem of text
generation, modeled using pre-trained sequence-to-sequence models. BARTScore
uses BART, an encoder-decoder-based pre-trained model, to translate generated
text to and from a reference point, earning higher scores when the text is
more accurate and fluent. This metric offers several variants that can be
flexibly applied in an unsupervised manner for different perspectives of text
evaluation, such as fluency or factuality. Tests have shown that BARTScore can
outperform existing top-scoring metrics in the majority of test settings
across multiple datasets and perspectives.

####  3.1.3 Human Evaluation Metrics

Human evaluation in factuality assessment is crucial due to its sensitivity to
nuanced elements of language and context that may elude automated systems.
Human evaluators excel at interpreting abstract concepts and emotional
subtleties that can significantly inform the accuracy of evaluation. However,
they are subject to limitations such as subjectivity, inconsistency, and
potential for error. On the other hand, automated evaluations offer consistent
results, and efficient processing of large data sets, and are ideal for tasks
needing quantitative measurements. They also provide an objective benchmark
for model performance comparison. Overall, an ideal evaluation framework might
blend automated evaluation’s scalability and consistency with human
evaluation’s ability to interpret complex linguistic concepts.

##### Attribution

is a metric to verify that the output of LLMs is sharing only verifiable
information about the external world. As proposed by (Rashkin et al., 2023),
Attributable to Identified Sources (AIS) is a human evaluation framework that
adopts a binary concept of attribution. A text passage $y$ is deemed
attributable to a set $A$ of evidence if, and only if, an arbitrary listener
would agree with the statement ”According to $A$, $y$” within the context of
$y$. The AIS framework awards a full score (1.0) if every element of content
in passage $y$ can be linked to the evidence set $A$. Conversely, it gives a
score of zero (0.0) if this condition is not met.

Based on AIS, Gao et al. (2023a) propose a more fine-grained, sentence-level
extension of AIS called Auto-AIS, where annotators assign AIS scores to each
sentence, and an average score across all sentences is reported. This
procedure effectively measures the percentage of sentences that are fully
attributable to the evidence. Context, such as surrounding sentences and the
question the text answers, is provided to annotators for more informed
judgment. A limit is also set for the number of evidence snippets in the
attribution report to maintain conciseness.

During model development, an automated AIS metric is defined to approximate
human AIS evaluations, using a natural language inference model, which
correlates well with AIS scores. Before computing the scores, they improve
accuracy by decontextualizing each sentence in context.

##### FActScore

(Min et al., 2023) is a novel evaluation metric designed to assess the factual
precision of long-form text generated by LLMs. The challenge of evaluating the
factuality of such text arises from two main issues: (1) the generated content
often contains a mix of supported and unsupported information, making binary
judgments insufficient, and (2) human evaluation is both time-consuming and
expensive. To address these challenges, FActScore breaks down a generated text
into a series of atomic facts—short statements that each convey a single piece
of information. Each atomic fact is then evaluated based on its support from a
reliable knowledge source. The overall score represents the percentage of
atomic facts that are supported by the knowledge source. The paper conducted
extensive human evaluations to compute FActScores for biographies generated by
several state-of-the-art commercial LLMs, including InstructGPT, ChatGPT, and
the retrieval-augmented PerplexityAI. The results revealed that these LLMs
often contain factual inaccuracies, with FActScores ranging from 42% to 71%.
Notably, the factual precision of these models tends to decrease as the rarity
of the entities in the biographies increases.

####  3.1.4 LLM-based Metrics

Using LLMs for evaluation offers efficiency, versatility, reduced reliance on
human annotation, and the capability of evaluating multiple dimensions of
conversation quality in a single model call, which improves scalability.
However, potential issues include a lack of established validation, which can
lead to bias or accuracy problems if the LLM used for evaluation is not
thoroughly vetted. The decision process to identify suitable LLMs and decoding
strategies can be complex and pivotal to obtaining accurate evaluations. The
range of evaluation may also be limited, as the focus is often on open-domain
conversations, possibly leaving out assessments in specific or narrow domains.
While reducing human input can be beneficial, it can also miss out on crucial
interaction quality aspects better evaluated by human judges, such as
emotional resonance or nuanced understandings.

##### GPTScore

(Fu et al., 2023) is a new evaluation framework designed to assess the quality
of output from generative AI models. To provide these evaluations, GPTScore
taps into the emergent capabilities of 19 different pre-trained models, such
as zero-shot instruction, and uses them to judge the generated texts. These
models vary in scale from 80M to 175B. Testing across four text generation
tasks, 22 aspects of evaluation, and 37 related datasets, has demonstrated
that GPTScore can effectively evaluate text per instructions in natural
language. This attribute allows it to sidestep challenges traditionally
encountered in text evaluation, like the need for sample annotations and
achieving custom, multi-faceted evaluations.

##### GPT-judge

(Lin et al., 2022b) is a finetuned model based on the GPT-3-6.7B, which is
trained to evaluate the truthfulness of answers to questions in the TruthfulQA
dataset. The training set consists of triples in the form of question-answer-
label combinations, where the label can be either true or false. The model’s
training set includes examples from the benchmark and answers generated by
other models assessed by human evaluation. In its final form, the GPT-judge
uses examples from all models to evaluate the truthfulness of responses. This
training includes all questions from the dataset, with the goal being to
evaluate truth, not generalize new questions.

The study conducted by the authors (Lin et al., 2022b) focuses on the
application of GPT-judge in assessing _Truthfulness_ and _Informativeness_
using the TruthfulQA Dataset. The authors undertook the fine-tuning of two
distinct GPT-3 models to evaluate two essential aspects: Truthfulness, which
pertains to the accuracy and honesty of information provided by the LLM, and
Informativeness, which measures how effectively the LLM conveys relevant and
valuable information in its responses. From these two fundamental concepts,
the authors derived a combined metric denoted as _truth * info_. This metric
represents the product of scalar scores for both truthfulness and
informativeness. It not only quantifies the extent to which questions are
answered truthfully but also incorporates the assessment of informativeness
for each response. This comprehensive approach prevents the model from
generating generic responses like ”I have no comment” and ensures that
responses are not only truthful but also valuable. These metrics have found
widespread deployment in evaluating the factuality of information generated by
LLMs (Chuang et al., 2023; Li et al., 2023d).

##### LLM-Eval

(Lin and Chen, 2023) is a novel evaluation methodology for open-domain
dialogues with LLMs. Unlike conventional evaluation methods which rely on
human annotations, ground-truth responses, or multiple LLM prompts, LLM-Eval
uses a unique prompt-based evaluation process employing a unified schema to
assess various elements of a conversation’s quality during a single model
function. Extensive evaluations of LLM-Eval’s performance using multiple
benchmark datasets indicate it is effective, efficient, and adaptable compared
to traditional evaluation practices. Further, the authors stress the necessity
of selecting appropriate LLMs and decoding strategies for precise evaluation
outcomes, underscoring LLM-Eval’s versatility and dependability in assessing
open-domain conversation systems across a variety of circumstances.

TABLE IV: Benchmarks for Evaluating Factuality in LLMs. This table details each benchmark’s task type, associated sub-datasets, evaluation metrics, and the performance of select representative LLMs. Reference | Task Type | Dataset | Metrics |  Performance of Representative LLMs
---|---|---|---|---
MMLU (Hendrycks et al., 2021) |  | Multi-Choice QA
---
| Humanities,
---
Social,
Sciences,
STEM…
ACC |  | (ACC, 5-shot)
---
GPT-4: 86.4 GPT-3.5: 70
LLaMA2-70B: 68.9
TruthfulQA (Lin et al., 2022b) |  | QA
---
| Health, Law,
---
Conspiracies,
Fiction…
| Human Score,
---
GPT-judge,
ROUGE, BLEU,
MC1,MC2…
| (zero-shot)
---
GPT-4: $\sim$29 (MC1)
GPT-3.5: $\sim$28 (MC1), 79.92(%true)
LLaMA2-70B: 53.37 (%true)
C-Eval (Huang et al., 2023b) | Multi-Choice QA |  | STEM,
---
Social Science,
Humanities…
| ACC
---
| (zero-shot, average ACC)
---
GPT-4: 68.7 GPT-3.5: 54.4
LLaMA2-70B: 50.13
AGIEval (Huang et al., 2023b) | Multi-Choice QA |  | Gaokao,
---
(geometry, Bio,
history…),
SAT, Law…
| ACC
---
| (zero-shot, average ACC)
---
GPT-4: 56.4 GPT-3.5: 42.9
LLaMA2-70B: 40.02
HaluEval (Li et al., 2023c) |  | Hallucination
---
Evaluation
HaluEval | ACC |  | (general ACC)
---
GPT-3.5: 86.22
BigBench (Srivastava et al., 2023) |  | Multi-tasks (QA, NLI,
---
Fact Checking,
Reasoning…)
BigBench |  | Metric to each
---
type of task
| (Big-Bench Hard)
---
GPT-3.5: 49.6
LLaMA-65B: 42.6
ALCE (Gao et al., 2023b) |  | Citation
---
Generation
| ASQA, ELI5,
---
QAMPARI
| MAUVE,
---
Exact Match,
ROUGE-L…
| (ASQA, 3-psg, citation prec)
---
GPT-3.5: 73.9
LLaMA-33B: 23.0
QUIP (Weller et al., 2023) | Generative QA |  | TriviaQA,
---
NQ, ELI5,
HotpotQA
| QUIP-Score
---
Exact match
| (ELI5, QUIP, null prompt)
---
GPT-4: 21.0
GPT-3.5: 27.7
PopQA (Mallen et al., 2023) | Multi-Choice QA |  | PopQA,
---
EntityQuestions
ACC |  | (overall ACC)
---
GPT-3.5: $\sim$37.0
UniLC (Zhang et al., 2023e) | Fact Checking |  | Climate,
---
Health, MGFN
| ACC
---
F1
| (zero-shot, fact tasks, average F1)
---
GPT-3.5: 51.62
Pinocchio (Hu et al., 2023) |  | Fact Checking
---
QA, Reasoning
| Pinocchio
---
| ACC
---
F1
| GPT-3.5: (Zero-shot ACC: 46.8, F1:44.4)
---
GPT-3.5: (Few-shot ACC: 47.1, F1:45.7)
SelfAware (Yin et al., 2023) | Self-evaluation |  | SelfAware
---
| ACC
---
| (instruction input, F1)
---
GPT-4: 75.47 GPT-3.5: 51.43
LLaMA-65B: 46.89
RealTimeQA (Kasai et al., 2022) |  | Multi-Choice QA,
---
Generative QA
RealTimeQA | ACC, F1 |  | (original setting, GCS retrieval)
---
GPT-3: 69.3 (ACC for MC)
GPT-3: 39.4 (F1 for generation)
FreshQA (Vu et al., 2023) | Generative QA | FRESHQA | ACC (Human) |  | (strict ACC, null prompt)
---
GPT-4: 28.6 GPT-3.5: 26.0

###  3.2 Benchmarks for Factuality Evaluation

In this section, we delve into the benchmarks that are prominently employed to
assess the factuality of LLMs. Specific benchmarks tailored for evaluating
factuality in LLMs are tabulated in Table IV.

TABLE V: Factuality Evaluation Studies of LLMs. This table shows studies focused on factuality evaluation, and those providing valuable insights into such evaluations. The column ‘Human Eval’ denotes if the evaluation incorporated human evaluation. The ‘Granularity’ column specifies the level of evaluation, with token-level (T) and sentence-level (S) distinctions. Reference | Task | Dataset | Metrics |  Human Eval | Evaluated LLMs | Granularity
---|---|---|---|---|---|---
| FActScore
---
(Min et al., 2023)
| Biography
---
Generation
| 183 people
---
entities
F1 | ✓ |  | GPT-3.5
---
ChatGPT…
T
| SelfCheckGPT
---
(Manakul et al., 2023)
Bio Generation | WikiBio |  | AUC-PR,
---
Human Score
✓ |  | GPT-3,
---
LLaMA,
OPT,
GPT-J…
S
(Wang et al., 2023a) | Open QA |  | NQ, TQ
---
| ACC,
---
EM
✓ |  | GPT-3.5
---
ChatGPT
GPT-4
Bing Chat
S
(Pezeshkpour, 2023) |  | Knowledge
---
Probing
| T-REx,
---
LAMA
ACC |  | GPT3.5 | T
(De Cao et al., 2021) |  | QA,
---
Fact Checking
| KILT,
---
FEVER,
zsRE
ACC |  |  | GPT-3
---
FLAN-T5
| S/ T
---
(Varshney et al., 2023) |  | Article
---
Generation
| Unnamed
---
Dataset
| ACC,
---
AUC
|  | GPT3.5
---
Vicuna
S
| FactTool
---
(Chern et al., 2023)
KB-based QA |  | RoSE
---
| ACC
---
F1…
|  | GPT-4
---
ChatGPT
FLAN-T5
S
(Kadavath et al., 2022) | Self-evaluation |  | BIG Bench,
---
MMLU, LogiQA,
TruthfulQA,
QuALITY
TriviaQA
Lambada
| ACC,
---
Brier Score,
RMS
Calibration
Error…
| Claude | T
(Chen et al., 2023c) | QA |  | NQ,
---
WoW
| Factuality,
---
Relevance,
Coherence,
Informativeness,
Helpfulness,
and Validity
| LLaMA, FLAN-T5, … | T

MMLU (Hendrycks et al., 2021) and TruthfulQA (Lin et al., 2022b) stand as two
pivotal benchmarks in the realm of evaluating the factuality of LLMs (OpenAI,
2023; Touvron et al., 2023b; Park, 2023) . The MMLU benchmark is proposed to
measure a text model’s multitask accuracy across a diverse set of 57 tasks.
These tasks span a wide range of subjects, from elementary mathematics to US
history, computer science, law, and more. The benchmark is designed to test
both the world knowledge and problem-solving ability of models. The findings
from the paper suggest that while most recent models perform at near random-
chance accuracy, the largest GPT-3 model showcased a significant improvement.
However, even the best models still have a long way to go before achieving
expert-level accuracy across all tasks (OpenAI, 2023). TruthfulQA is a
benchmark designed to assess the truthfulness of a language model’s generated
answers. The benchmark consists of 817 questions spanning 38 categories,
including health, law, finance, and politics. These questions were crafted in
such a way that some humans might answer them falsely due to misconceptions or
false beliefs. The goal for models is to avoid generating these false answers
that they might have learned from imitating human texts. The TruthfulQA
benchmark serves as a tool to highlight the potential pitfalls of relying
solely on LLMs for accurate information and emphasizes the need for continued
research in this area.

HaluEval (Li et al., 2023c) is a benchmark designed to understand and evaluate
the propensity of LLMs like ChatGPT to generate hallucinations. A
hallucination, in this context, refers to content that either conflicts with
the source or cannot be verified based on factual knowledge. The HaluEval
benchmark offers a vast collection of generated and human-annotated
hallucinated samples, aiming to evaluate the performance of LLMs in
recognizing such hallucinations. The benchmark utilizes a two-step framework,
termed ”sampling-then-filtering”, based on ChatGPT to generate these samples.
Additionally, human labelers were employed to annotate hallucinations in
ChatGPT responses. The HaluEval benchmark is a comprehensive tool that not
only evaluates the hallucination tendencies of LLMs but also provides insights
into the types of content and the extent to which these models are prone to
hallucinate.

BigBench (Srivastava et al., 2023) focuses on the capabilities and limitations
of LLMs. It comprises 204 tasks from diverse domains such as linguistics,
childhood development, math, common-sense reasoning, biology, physics, social
bias, software development, and more. The benchmark is designed to evaluate
tasks believed to be beyond the capabilities of current language models. The
study evaluates the performance of various models, including OpenAI’s GPT
models, on BIG-bench and compares them to human expert raters. Key findings
suggest that model performance and calibration improve with scale but are
still suboptimal when compared to human performance. Tasks that involve a
significant knowledge or memorization component show predictable improvement,
while tasks that exhibit ”breakthrough” behavior at a certain scale often
involve multiple steps or components.

Huang et al. (2023b) propose C-Eval, the first comprehensive Chinese
evaluation suite. It can be used to evaluate the advanced knowledge and
reasoning abilities of foundational models within a Chinese context. The
evaluation suite comprises multiple-choice questions spanning 52 diverse
disciplines, with four levels of difficulty: middle school, high school,
college, and professional. Additionally, C-Eval Hard is introduced for very
challenging subjects within the C-Eval suite, which demands advanced reasoning
abilities to solve. Their evaluations of state-of-the-art LLMs, including both
English and Chinese-oriented models, have shows that there is significant room
for improvement as only GPT-4 managed to achieve an average accuracy of over
60%. The authors focus on assessing LLMs’ advanced abilities within a Chinese
context. The researchers assert that LLMs intended for a Chinese environment
should be evaluated based on their knowledge of Chinese users’ primary
interests, such as Chinese culture, history, and laws. With C-Eval, the
authors aim to guide developers in understanding the abilities of their models
from multiple dimensions to facilitate the development and growth of
foundational models for Chinese users. Simultaneously, C-Eval has not just
introduced a whole suite but subsets that can serve as individual benchmarks,
thereby assessing certain model abilities and analyzing key strengths and
limitations of foundational models. Experiment results have shown that
although GPT-4, ChatGPT, and Claude were not exclusively tailored for Chinese
data, they emerged as the top performers on C-Eval.

SelfAware (Yin et al., 2023) aims to investigate whether models recognize what
they don’t know. This dataset encompasses two types of questions: unanswerable
and answerable. The dataset comprises 2,858 unanswerable questions gathered
from various websites and 2,337 answerable questions extracted from sources
such as SQuAD, HotpotQA, and TriviaQA. Each unanswerable question is confirmed
as such by three human evaluators. In the conducted experiments, GPT-4
achieves the highest F1 score of 75.5, compared to a human score of 85.0.
Larger models tend to perform better and in-context learning can enhance
performance.

The Pinocchio benchmark (Hu et al., 2023) serves as an extensive evaluation
platform, emphasizing factuality and reasoning for LLMs. This benchmark
encompasses 20,000 varied factual queries from diverse sources, timeframes,
fields, regions, and languages. It tests an LLM’s capability to discern
combined facts, process both organized and scattered evidence, recognize the
temporal evolution of facts, pinpoint minute factual disparities, and
withstand adversarial inputs. Each reasoning challenge within the benchmark is
calibrated for difficulty to allow for detailed analysis.

Kasai et al. (2022) introduce a dynamic QA platform named REALTIMEQA. This
platform is unique in that it announces questions and evaluates systems on a
regular basis, specifically weekly. The questions posed by REALTIMEQA pertain
to current events or novel information, challenging the static nature of
traditional open domain QA datasets. The platform aims to address
instantaneous information needs, pushing QA systems to provide answers about
recent events or developments. Their preliminary findings indicate that while
GPT-3 can often update its generation results based on newly-retrieved
documents, it sometimes returns outdated answers when the retrieved documents
lack sufficient information.

Vu et al. (2023) present a dynamic benchmark called FreshQA designed to
evaluate up-to-date world knowledge of LLMs. Its questions range from those
that are never-changing to those that are fast-changing, as well as questions
based on false premises. The aim is to challenge the static nature of LLMs and
test their adaptability to ever-changing knowledge through human evaluations.
They develop a reliable evaluation protocol that uses a two-mode system:
RELAXED and STRICT for a comprehensive understanding of model performance,
ensuring that answers are confident, definitive, and accurate. They also
provide a strong baseline named FRESHPROMPT, which seeks to enhance LLM
performance by integrating real-time data from search engines. Initial
experiments reveal that, outdated training data weakens the performance of
LLMs and the FRESHPROMPT method can significantly enhance it. The research
underscores the need for LLMs to be refreshed with current information to
ensure their relevance and accuracy in a constantly evolving world.

FreshQA (Vu et al., 2023) is a dynamic benchmark designed to evaluate up-to-
date world knowledge of LLMs. Its questions range from those that are never-
changing to those that are fast-changing, as well as questions based on false
premises. The aim is to challenge the static nature of LLMs and test their
adaptability to ever-changing knowledge through human evaluations. They
develop a reliable evaluation protocol that uses a two-mode system: RELAXED
and STRICT for a comprehensive understanding of model performance, ensuring
that answers are confident, definitive, and accurate. They also provide a
strong baseline named FRESHPROMPT, which seeks to enhance LLM performance by
integrating real-time data from search engines. Initial experiments reveal
that, outdated training data weakens the performance of LLMs and the
FRESHPROMPT method can significantly enhance it. The research underscores the
need for LLMs to be refreshed with current information to ensure their
relevance and accuracy in a constantly evolving world.

Several benchmarks, such as BigBench (Srivastava et al., 2023) and C-Eval
(Huang et al., 2023b), encompass subsets that extend beyond the realm of
factual knowledge or factuality. In this work, we specifically emphasize and
focus on those subsets related to factuality.

There are benchmarks primarily designed for Pre-trained Language Models (PLMs)
that can also be adapted for LLMs. Some studies use them for evaluating LLM’s
factuality, but they are not so widely-used, so we have chosen to exclude them
from the table for clarity. These benchmarks predominantly encompass
knowledge-intensive tasks, as highlighted by (Petroni et al., 2021). Those
benchmarks include NaturalQuestions (NQ) (Kwiatkowski et al., 2019), TriviaQA
(TQ) (Joshi et al., 2017), OTT-QA (Chen et al., 2021b), AmbigQA Min et al.
(2020) and WebQuestion (WQ) (Berant et al., 2013) of Open-domain Question
Answering (QA) task, the HotpotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et
al., 2020), IIRC (Ferguson et al., 2020), MuSiQue (Trivedi et al., 2022) of
multi-step QA, the ELI5 (Fan et al., 2019) of the Long-form QA task, the FEVER
(Thorne et al., 2018), FM2 (Eisenschlos et al., 2021), HOVER (Jiang et al.,
2020), FEVEROUS (Aly et al., 2021) of the Fact Checking task, the T-REx
(Elsahar et al., 2018), zsRE (Levy et al., 2017) and LAMA (Petroni et al.,
2019a) to examine factual knowledge contained in pretrained language models,
the WikiBio (Lebret et al., 2016) of biography generation task, the RoSE (Liu
et al., 2023d), WikiAsp (Hayashi et al., 2021) of summarization task, the KILT
(Petroni et al., 2021) of comprehensive knowledge intensive tasks, the
MassiveText (Rae et al., 2022b), Curation Corpus (Curation, 2020), Wikitext103
(Merity et al., 2016), Lambada (Paperno et al., 2016), C4 (Raffel et al.,
2020b), Pile (Gao et al., 2020) of langauge modeling task, the WoW (Dinan et
al., 2019), DSTC7 track2 (Galley et al., 2019a), DSTC11 track5 (Zhao et al.,
2023a) of dialogue task, the RealToxicityPrompts Gehman et al. (2020) of
toxicity reduction, the CommaQA (Khot et al., 2022), StrategyQA (Geva et al.,
2021a), TempQuestions (Jia et al., 2018), IN-FOTABS (Gupta et al., 2020a) of
diverse reasoning tasks.

Some studies (Min et al., 2023; Manakul et al., 2023) also provide a small
dataset, but they mainly concentrate on the evaluation metrics or methods for
factuality, we choose to discuss them in the next subsection.

###  3.3 Factuality Evaluation Studies

TABLE VI: Factuality Evaluation of LLMs. This table lists the evaluation metrics used by studies aiming to improve the factuality of LLMs. The column ‘Human Eval’ denotes if the evaluation incorporated human evaluation. The ‘Granularity’ column specifies the level of evaluation, with token-level (T) and sentence-level (S) distinctions. Reference | Task | Dataset | Metrics |  Human Eval | Evaluated LLMs | Granularity
---|---|---|---|---|---|---
| Retro
---
(Borgeaud et al., 2022)
| QA,
---
Language
Modeling
| MassiveText,
---
Curation Corpus,
Wikitext103,
Lambada,
C4,Pile, NQ
| PPL,
---
ACC,
Exact Match
✓ |  | Retro
---
T
| GenRead
---
(Yu et al., 2023)
| QA,
---
Dialogue,
Fact Checking
| NQ, TQ, WebQ,
---
FEVER,
FM2, WoW
| EM, ACC,
---
F1, Rouge-L
|  | GPT3.5, Codex
---
GPT-3, Gopher
FLAN, GLaM
PaLM
S
| GopherCite
---
(Menick et al., 2022b)
| Self-supported QA
---
| NQ, ELI5,
---
TruthfulQA
(Health
Law, Fiction
Conspiracies)
| Human Score
---
✓ |  | GopherCite
---
S
(Trivedi et al., 2023) |  | QA
---
| HotpotQA, IIRC
---
2WikiMultihopQA,
MuSiQue(music)
| Retrieval recall,
---
Answer F1
|  | GPT-3
---
FLAN-T5
| S/ T
---
(Peng et al., 2023) |  | QA,
---
Dialogue
| DSTC7 track2,
---
DSTC11 track5,
OTT-QA
| ROUGE, chrF,
---
BERTScore,
Usefulness,
Humanness…
✓ | ChatGPT |  | S/ T
---
| CRITIC
---
(Gou et al., 2023)
| QA
---
Toxicity Reduction
| AmbigNQ,
---
TriviaQA,
HotpotQA,
RealToxicityPrompts
| Exact Match,
---
maximum toxicity,
perplexity,
n-gram diversity,
AUROC…,
|  | GPT-3.5
---
ChatGPT
T
(Khot et al., 2023) |  | QA,
---
long-context
QA
| CommaQA-E,
---
2WikiMultihopQA,
MuSiQue,
HotpotQA
| Exact Match,
---
Answer F1
|  | GPT-3
---
FLAN-T5
T
| ReAct
---
(Yao et al., 2023a)
| QA
---
Fact Verification
| HotpotQA,
---
FEVER
| Exact Match,
---
ACC
|  | PaLM
---
GPT-3
| S/T
---
(Jiang et al., 2023b) |  | QA,
---
Commonsense
Reasoning,
long-form QA…
| 2WikiMultihopQA,
---
StrategyQA,
ASQA, WikiAsp
| Exact Match,
---
Disambig-F1,
ROUGE,
entity F1…
| GPT-3.5 | T
(Lee et al., 2022b) |  | Open-ended
---
Generation
FEVER |  | Entity score,
---
Entailment
Ratio, ppl…
| Megatron-LM | T
| SAIL
---
(Luo et al., 2023a)
| QA
---
Fact Checking
UniLC |  | ACC
---
F1
|  | LLaMA Vicuna
---
SAIL
T
(He et al., 2022) |  | Commonsense
---
Reasoning,
Temporal
Reasoning,
Tabular
Reasoning
| StrategyQA,
---
TempQuestions,
IN-FOTABS
ACC |  | GPT-3 | T
(Pan et al., 2023a) | Fact Checking |  | HOVER
---
FEVEROUS-S
Macro-F1 |  |  | Codex
---
FLAN-T5
S
(Du et al., 2023) |  | Biography
---
MMLU
| Unnamed
---
Biography
Dataset,
MMLU
| ChatGPT
---
Evaluator,
ACC
|  | Bard
---
ChatGPT
S

In this section, we delve into studies that evaluate factuality in LLMs
without introducing a specific benchmark, focusing primarily on those whose
main contribution lies in the evaluation methodology. We spotlight works that
have pioneered evaluation techniques, metrics, or have offered distinctive
insights into the factuality evaluation of LLMs.

Manakul et al. (2023) use an evaluation process that encompasses several key
steps. Initially, synthetic Wikipedia articles are generated using GPT-3,
focusing on individuals from the Wikibio dataset. Subsequently, manual
annotation is performed at the sentence level, classifying sentences as “Major
Inaccurate” , “Minor Inaccurate”, or “Accurate”, with “Major Inaccurate”
denoting sentences unrelated to the topic. Passage-level scores are derived by
averaging sentence-level labels, and identifying cases of total inaccuracies
through score distribution analysis. Inter-annotator agreement is assessed
using Cohen’s $\kappa$ Cohen (1960). Evaluation metrics primarily employ
precision-recall curves (PR-Curves), distinguishing between “Non-Factual
Sentences,” “Non-Factual* Sentences” (a specific subset), and “Factual
Sentences.” These PR-Curves elucidate the trade-off between precision and
recall for different detection methods.

Wang et al. (2023a) ask several LLMs, including ChatGPT, GPT-4 (OpenAI, 2023),
BingChat (Microsoft, 2023) to answer open questions from NaturalQuestions
(Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017). They manually
estimate the accuracy of those LLMs on open question answering, and find that
though LLMs can achieve nice performance but still far away from perfect.

Pezeshkpour (2023) propose a new metric for measuring whether a certain type
of knowledge is present in LLM. This metric is based on information theory and
measures knowledge by analyzing the probability distribution of predictions
made by LLM before and after injecting target knowledge. The accuracy of
GPT-3.5 in the Knowledge Probing task is tested on the T-REx (Elsahar et al.,
2018) and LAMA (Petroni et al., 2019a) datasets.

Varshney et al. (2023) point out a common real-world occurrence where users
often ask questions based on false premises. These questions are challenging
for state-of-the-art models. This necessitated the creation of a new
evaluation dataset. To this end, the authors have conducted a case study and
compiled a set of 50 such adversarial questions, all of which the GPT-3.5
model answered incorrectly. The aim is to create a challenging experimental
setup to assess the performance of models faced with such questions. In order
to enhance the evaluation, corresponding true premise questions were created
for each of the false premise questions. This allowe for a holistic evaluation
of the model’s performances, taking into consideration both correct and
incorrect premises. The authors make sure to evaluate the complete answers
given by the model for correct and incorrect questions - in this context, it’s
not enough for an answer to be partially correct, the entire answer needs to
be accurate to be marked as correct. For example: For the false premise
question “Why does Helium have an atomic number of 1?”, the corresponding true
premise question is “Why does Hydrogen have an atomic number of 1?”.

FACTOOL (Chern et al., 2023) is a tool designed to function as a factuality
detector, with the primary purpose of auditing generative chatbots and
assessing the reliability of their outputs. This tool is employed to evaluate
several contemporary chatbots, including GPT-4, ChatGPT, Claude, Bard (Google,
2023), and Vicuna (Chiang et al., 2023). Notably, FACTOOL itself leverages the
capabilities of GPT-4. For the evaluation process, the researchers have
curated a diverse set of prompts: 30 from knowledge-based question answering
(KB-QA), 10 each from code, math, and scientific domains. The KB-QA prompts
were sourced from a prior study, code prompts were taken from HumanEval, math
prompts from another distinct study, while the scientific prompts were crafted
by the authors themselves. The evaluation metrics included both claim-level
and response-level accuracies for each chatbot. To offer a more comprehensive
and equitable evaluation, a weighted claim-level accuracy is used. The
weighting is determined based on the proportion of prompts from each category.
The findings are illuminating. GPT-4 emerge as the top performer in terms of
both weighted claim-level factual accuracy and response-level accuracy among
all the chatbots assessed. Another intriguing observation is that chatbots
that underwent supervised fine-tuning, such as Vicuna-13B, exhibited
commendable performance in standard scenarios like KB-QA. However, their
performance dip in more intricate scenarios, including those involving math,
code, and scientific queries.

Kadavath et al. (2022) investigate whether language models can evaluate the
accuracy of their own assertions and predict which questions they can answer
correctly. It is found that larger models are well-calibrated on diverse
multiple-choice and true/false questions if given in the appropriate format.
The approach to self-evaluation on open-ended tasks is to ask the models to
initially suggest answers, and then evaluate the probability (P[True]) that
their answers are correct. This resulted in compelling performance,
calibration, and scaling on a diverse range of tasks. Furthermore, self-
evaluation performance improved when the models are allowed to consider many
of their own suggestions before predicting the validity of a specific one.

Yu et al. (2023) explore whether the internal knowledge of LLMs can replace
the retrieved documents on knowledge intensive tasks. They ask LLMs, such as
InstructGPT (OpenAI, 2022a), to directly generate contexts given a question
rather than retrieving from database. They find the generated documents
contain the golden answers more often than the top retrieved documents. Then
they feed the generated docs and retrieved docs to the Fusion-in-Decoder model
(Izacard and Grave, 2021b) for knowledge-intensive tasks such as Open-domain
QA (Kwiatkowski et al., 2019) and find the generated docs are more effective
than the retrieved docs, suggesting that the LLMs contain enough knowledge for
knowledge-intensive tasks.

Menick et al. (2022b) propose a task named Self-supported QA to evaluate LLMs’
ability in also producing citations when generating answers. Authors ask
humans to evaluate whether the responses of their proposed model GopherCite
are plausible and whether they are supported by the accompanying quote
evidence on datasets such as NQ, ELI5, TruthfulQA.

Chen et al. (2023c) propose CONNER, a framework that evaluates LLMs as
generators of knowledge. It focuses on six areas: Factuality, Relevance,
Coherence, Informativeness, Helpfulness, and Validity. It evaluates whether
the generated information can be backed by external proof (Factuality), is
relevant to the user’s query (Relevance), and is logically consistent
(Coherence). It also checks if the knowledge provided is novel or surprising
(Informativeness). The Extrinsic evaluation measures whether the knowledge
enhances downstream tasks (Helpfulness) and its results are factually accurate
(Validity).

In the realm of factuality evaluation, the Model Editing task holds a unique
position, focusing on refining the internal knowledge of models. This task
comes with its own set of specialized evaluation metrics. Predominantly, the
Zero-Shot Relation Extraction (zsRE) Levy et al. (2017) and CounterFact Meng
et al. (2022) serve as the primary benchmarks for assessing Model Editing
techniques. When evaluating these methods, several key criteria emerge:
Reliability: Post-editing, the model should consistently generate the intended
output. Generalization: The model should be adept at producing the target
output even when presented with paraphrased inputs. Locality: Edits should be
localized, ensuring that facts not related to the specific edit remain intact.
However, given the intricate web of interconnected facts, recent studies Zhong
et al. (2023b); Yao et al. (2023b); Cohen et al. (2023a) have advocated for a
more holistic evaluation approach. They introduce broader criteria for fact
updates, encompassing aspects like portability, logical generalization, among
others.

Some work’s main contributions lie at the methods to improve the factuality of
LLMs, and their evaluation part may also be informative and important when
people reach the related study. So we choose to list evaluation parts in the
Table VI but not to discuss their evaluation in detail.

###  3.4 Evaluating Domain-specific Factuality

TABLE VII: Benchmarks for domain-specific factuality evaluation. The table presents the domain, tasks, datasets, and the LLMs evaluated in the respective study. Reference | Domain | Task | Datasets | Metrics | Evaluated LLMs
---|---|---|---|---|---
Xie et al. (2023b) | Finance |  | Sentiment analysis,
---
News headline classification
Named entity recognition
Question answering
Stock movement prediction
| FLARE
---
| F1, Acc,
---
Avg F1,
Entity F1,
EM, MCC
| GPT-4 ,
---
BloombergGPT,
FinMA-(7B, 30B, 7B-full),
Vicuna-7B
Li et al. (2023e) | Finance | 134 E-com tasks |  | EcomInstruct
---
| Micro-F1,
---
Macro-F1,
ROUGE
| BLOOM, BLOOMZ,
---
ChatGPT, EcomGPT
Wang et al. (2023d) | Medicine | Multi-Choice QA | CMB | Acc |  | GPT-4, ChatGLM2-6B,
---
ChatGPT, DoctorGLM,
Baichuan-13B-chat,
HuatuoGPT, MedicalGPT,
ChatMed-Consult,
ChatGLM-Med ,
Bentsao, BianQue-2
Li et al. (2023b) | Medicine | Generative-QA | Huatuo-26M |  | BLEU,
---
ROUGE,
GLEU
T5, GPT2
Jin et al. (2023) | Medicine |  | Nomenclature,
---
Genomic location,
Functional analysis,
Sequence alignment
GeneTuring | Acc |  | GPT-2, BioGPT,
---
BioMedLM, GPT-3,
ChatGPT, New Bing
Guha et al. (2023) | Law |  | Issue-spotting,
---
Rule-recall,
Rule-application,
Rule-conclusion,
Interpretation,
Rhetorical-understanding
LegalBench |  | Acc, EM
---
| GPT-4, GPT-3.5,
---
Claude-1, Incite, OPT
Falcon, LLaMA-2, FLAN-T5…
Fei et al. (2023) | Law |  | Legal QA, NER,
---
Sentiment Analysis,
Reading Comprehension
LawBench |  | F1, Acc,
---
ROUGE-L,
Normalized
log-distance …
| GPT-4,
---
ChatGPT,
InternLM-Chat,
StableBeluga2…

To assess the performance and factuality of these specialized LLMs, a plethora
of datasets and benchmarks have been proposed across various domains. These
resources not only serve as critical tools for evaluating the capabilities of
LLMs but also facilitate advancements in specialized applications. We
summarize them in Table VII. The distinction between this subsection and the
previous two lies in its focus. This subsection delves deeper into factuality
evaluation tailored to specific domains, while Sec 3.2 and 3.3 primarily
concentrate on general factuality evaluation, with only a portion of their
content dedicated to datasets evaluating factuality within specific domains.

##### Finance

Xie et al. (2023b) designed a financial natural language understanding and
prediction evaluation benchmark dubbed FLARE, based on their collected
financial instruction tuning dataset FIT. This benchmark is used to evaluate
their FinMA model. It randomly selects validation sets from FIT to choose the
best model checkpoint and utilizes distinct test sets for evaluation. FLARE is
a broader variant compared to the existing FLUE benchmark Shah et al. (2022)
as it also encapsulates financial prediction tasks like stock movement
prediction in addition to standard NLP tasks. The FLARE dataset includes
several subtasks, such as sentiment analysis (FPB, FiQA-SA), news headline
classification (Headline), named entity recognition (NER), question answering
(FinQA, ConvFinQA), and stock movement prediction (BigData22, ACL18, CIKM18).
Performance is gauged via a variety of metrics for each task, such as the
accuracy and weighted F1 Score for sentiment analysis, entity-level F1 score
for named entity recognition, and accuracy and Matthews correlation
coefficient for stock movement prediction. Several methods, including their
own FIT-fine-tuned FinMA and other LLMs (BloombergGPT, GPT-4, ChatGPT, BLOOM,
GPT-NeoX, OPT-66B, Vicuna-13B) are dedicated to their comparison.
BloombergGPT’s performance is assessed in various shot scenarios, while the
zero-shot performance is reported for the remaining results. Some of the
baselines depend on human evaluations given that LLMs without fine-tuning fail
to generate instruction-defined answers. Conversely, FinMA’s results are
conducted on a zero-shot basis and can be evaluated automatically. To enable
direct comparison between the performance of FinMA and BloombergGPT, despite
the former not releasing their test datasets, test datasets were constructed
with the same data distribution.

Li et al. (2023e) propose EcomInstruct benchmark, which experiment
investigates the performance of their EcomGPT language model in comparison to
baseline models such as BLOOM and BLOOMZ. Categories of these baseline models
include pre-trained large models with decoder-only architecture like BLOOM,
and instruction-following language models like BLOOMZ and ChatGPT. The
evaluation metric of the EcomInstruct involves converting all tasks to
generative paradigms and using text generation evaluation metrics like
ROUGE-L. Classification tasks were evaluated with precision, recall, and F1
scores. The EcomInstruct dataset, comprising 12 tasks across four major
categories, is divided into training and testing sections. The EcomGPT is
trained on 85,746 instances of E-commerce data. The performance of the model
is assessed based on its capacity to generalize unseen tasks or datasets, with
emphasis on cross-language and cross-task paradigm settings.

##### Medicine

Wang et al. (2023d) propose a localized medical benchmark called CMB, or the
Comprehensive Medical Benchmark in Chinese. CMB is rooted entirely in the
native Chinese linguistic and cultural framework. While traditional Chinese
medicine is a significant part of this evaluation, it does not make up the
entire benchmark. Both prominent LLMs, such as ChatGPT and GPT-4, and
localized Chinese LLMs, including those specializing in the health domain, are
evaluated using CMB. However, the benchmark is not designed as a leaderboard
competition, but rather as a tool for self-assessment and understanding the
progression of models in this field. CMB embodies a comprehensive, multi-
layered medical benchmark in Chinese, comprised of hundreds of thousands of
multiple-choice questions and complex case consultation questions. This wide
range of queries covers all clinical medical specialties and various
professional levels, seeking to evaluate a model’s medical knowledge and
clinical consultation capabilities comprehensively.

Li et al. (2023b) introduce Huatuo-26M dataset, the largest Chinese medical
Question and Answer (QA) dataset to date, including over 26 million high-
quality medical QA pairs. It covers a wide range of topics, such as diseases,
symptoms, treatments, and drug information. The dataset is a valuable resource
for anyone looking to improve AI applications in the medical field, such as
chatbots and intelligent diagnostic systems. The Huatuo-26M dataset is
gathered and integrated from various sources, including online medical
encyclopedias, online medical knowledge bases, and online medical consultation
records. Each QA pair in the dataset contains a problem description and a
corresponding answer from a doctor or expert. Although a significant
proportion of the Huatuo-26M dataset is constituted by the online medical
consultation records, the text format data from these records is not publicly
available, for unspecified reasons. This dataset is expected to be
instrumental for multiple types of research and AI applications in the medical
field. These areas of application extend to Natural Language Processing tasks
like developing QA systems, text classification, sentiment analysis, and
Machine Learning model training tasks like disease prediction and personalized
treatment recommendation. Consequently, the dataset is favorable for
developing AI applications in the medical field, from intelligent diagnosis
systems to medical consultation chatbots.

Jin et al. (2023) use nine tasks related to NCBI resources to evaluate the
proposed GeneGPT model. The dataset used is GeneTuring benchmark Hou and Ji
(2023), which contains 12 tasks with 50 question-answer pairs each. The tasks
are divided into four modules - Nomenclature (Gene alias, Gene name
conversion), Genomic location(Gene SNP association, Gene location, SNP
location), Functional analysis(Gene disease association, Protein-coding
genes), Sequence alignment (DNA to human genome, DNA to multiple species). Two
settings of GeneGPT are assessed: a full setting where all prompt components
are used, and a slim setting using only two components. The performance of
GeneGPT is compared against various baselines including general-domain GPT-
based LLMs like GPT-2, GPT-3, and ChatGPT. Additionally, GPT-2-sized
biomedical domain-specific LLMs such as BioGPT and BioMedLM are evaluated. The
new Bing, a retrieval-augmented LLM with access to relevant web pages, is also
assessed. The result evaluation of the compared methods is based on the
results reported in the original benchmark and are manually evaluated.
However, the evaluation of the proposed GeneGPT method is determined through
automatic evaluations.

##### Law

LegalBench (Guha et al., 2023) is a benchmark for legal reasoning introduced
due to the increasing use of LLMs in the legal field. It consists of 162 tasks
covering six types of legal reasoning and was collaboratively constructed
through an interdisciplinary process with significant contributions from legal
professionals. The tasks designed aim to either demonstrate practical legal
reasoning capabilities or measure reasoning skills that are of interest to
lawyers. To facilitate discussions between different fields relating to LLMs
in law, LegalBench tasks correspond to popular legal frameworks for describing
legal reasoning, thus creating a shared language between lawyers and LLM
developers. The paper not only describes LegalBench, but also presents an
evaluation of 20 different open-source and commercial LLMs and highlights the
types of research opportunities that LegalBench can provide.

LawBench (Fei et al., 2023) is an evaluation framework dedicated to assessing
the capabilities of LLMs in relation to legal tasks. The context-specificity
and high-stakes nature of the law field make it crucial to have a clear grasp
of LLMs’ legal knowledge and their ability to execute legal tasks. LawBench
dives deep into three cognitive evaluations of LLMs: the capability to
memorize crucial legal details, understanding legal texts, and applying legal
knowledge to resolve complicated legal problems. A total of 20 diverse tasks
have been put together, covering five main task categories: single-label
classification, multi-label classification, regression, extraction, and
generation. Throughout the evaluation process, 51 LLMs were extensively tested
under LawBench, compiling a spectrum of language models including 20
multilingual LLMs, 22 Chinese-oriented LLMs, and 9 law-specific LLMs. The
results concluded that GPT-4 ranked as the superior LLM in the law domain,
significantly surpassing its competitors. Despite the noted improvements when
LLMs were fine-tuned on law-specific texts, the study acknowledged that there
is still a long road ahead in achieving highly reliable LLMs for legal tasks.

##  4 Analysis of Factuality

In the previous Section 3, we provide quantitative statistics related to
evaluating factuality. In this section, we delve deeper, exploring the
underlying mechanisms that influence factuality in large language models.

###  4.1 Analysis of Factuality

This subsection delves into intriguing analyses concerning the factuality of
LLMs, focusing on aspects that aren’t directly tied to evaluation, or
enhancement. Specifically, we explore the mechanisms through which LLMs
process, interpret, and produce factual content. The subsequent sections offer
an in-depth examination of different dimensions of factuality in LLMs, ranging
from their knowledge storage, and awareness to their approach to managing
conflicting data. 222 While some research (Wang et al., 2021; Neeman et al.,
2023) applies some similar analyses to Pretrained Language Models. This review
excludes them as they are not typically considered Large Language Models.

####  4.1.1 Knowledge Storage

The language model serves as a repository of knowledge, storing a multitude of
information about the world within its parameters Petroni et al. (2019b).
However, the organization of this knowledge within LLMs remains largely
mysterious. Meng et al. (2022) introduced a methodology called causal tracing
to measure the indirect impact of hidden states or activations. This technique
was employed to illustrate that factual knowledge is primarily stored in the
early-layer feed-forward networks (FFNs) of such models. Similarly, Geva et
al. (2021c) also suggests that a substantial portion of factual information is
encoded within the FFN layers. They conceptualize the input of the FFN as a
query, the first layer as keys, and the second layer as values. Consequently,
the intermediate hidden dimension of the FFN can be interpreted as the number
of memories within the layer, and the intermediate hidden state represents a
vector comprising activation values for each memory. As a result, the final
output of the FFN can be understood as the weighted sum of activated values.
The authors further demonstrate that the value vectors often encapsulate
human-interpretable concepts and knowledge Geva et al. (2022, 2023). In
addition, Chen et al. (2023d) made an intriguing finding that the language
model contains language-independent neurons that express multilingual
knowledge and degenerate neurons that convey redundant information by applying
the integrated gradients method Lundstrom et al. (2022). Nevertheless, it is
important to note that the aforementioned studies primarily focus on the
representation of individual facts, and the comprehensive understanding of how
factual knowledge is precisely organized and interconnected within these
models remains an ongoing challenge.

####  4.1.2 Knowledge Completeness and Awareness

This subsubsection delves into the intriguing realm of LLMs’ self-awareness,
their capacity to discern their knowledge gaps, and the balance between their
internally generated knowledge and externally retrieved information. We delve
into the dichotomy between parametric knowledge and retrieved knowledge,
exploring the promises and challenges these models bring to the forefront of
knowledge-intensive tasks.

##### Knowledge Awareness

Several studies have investigated the knowledge awareness of Large Language
Models, specifically assessing whether LLMs can accurately estimate the
correctness of their own responses. Most of these studies treat LLMs as ”black
boxes,” prompting the models to report their confidence levels or calculating
the perplexity of the model’s output as an indicator of response likelihood.
Gou et al. (2023) explore the model’s ability to validate and iteratively
refine its outputs, akin to how humans interact with tools. The authors find
that solely relying on self-correction without external feedback can lead to
marginal improvements or even diminished performance. Ren et al. (2023)
experiment with settings either augmented or not with external document
retrieval to determine whether models recognize their own knowledge
boundaries. Their findings indicate that LLMs possess an inaccurate perception
of their factual knowledge boundaries and tend to be overly confident about
their responses. LLMs often fail to fully harness the knowledge they possess;
however, retrieval enhancement can somewhat compensate for this shortcoming.
Yin et al. (2023) introduce a dataset named “SelfAware” to test if models
recognize what they don’t know, encompassing both answerable and unanswerable
questions. The experiment suggests that models do possess some capacity to
discern their own knowledge gaps, but they are still far from human levels.
GPT-4 outperforms other models, instructions and In-Context-Learning Dong et
al. (2023) can enhance a model’s discriminatory ability. Kadavath et al.
(2022) focus on LLM self-assessment based on Language Model calibration using
multiple-choice questions. Their findings revealed that the ”none of the
above” option decreased accuracy, larger models showed better calibration, and
RLHF hindered model calibration levels. However, simply adjusting the
temperature parameter can rectify this issue. Azaria and Mitchell (2023)
assess the truthfulness of statements generated by LLMs, by using the model’s
internal state and hidden layer activations. The authors, employing a
feedforward neural network, can classify if the model is misleading by
utilizing the hidden output states.

##### Parametric Knowledge vs Retrieved Knowledge

Yu et al. (2023) explore whether the internal knowledge of LLMs can replace
the retrieved documents on knowledge-intensive tasks. They ask LLMs, such as
InstructGPT, to directly generate contexts given a question rather than
retrieving them from the database. They find the generated documents contain
the golden answers more often than the top retrieved documents. Then they feed
the generated docs and retrieved docs to the Fusion-in-Decoder model (Izacard
and Grave, 2021b) for knowledge-intensive tasks such as Open-domain QA
(Kwiatkowski et al., 2019) and find the generated docs are more effective than
the retrieved docs, suggesting that the LLMs contain enough knowledge for
knowledge-intensive tasks.

On the contrary, these observations have been contested in subsequent
investigations. Kandpal et al. (2023) underscore the dependency of LLMs on the
number of associated documents seen during pre-training. They argue that the
success in answering fact-based questions is highly linked to the number of
documents containing the topic of the question that were encountered in pre-
training. The study further posited the necessity of scaling models
extensively to achieve competitive performance for questions with minimum
representation in the training data. Adding to these concerns, Sun et al.
(2023b) critically evaluate the factual knowledge base of LLMs, using a
specifically designed Head-to-Tail benchmark comprised of 18K question-answer
pairs. The results show that the understanding of factual knowledge,
particularly related to torso-to-tail entities, by currently available LLMs is
suboptimal.

In summary, while LLMs show promise in handling knowledge-intensive tasks,
their dependency on pre-training information and limitations in factual
accuracy remain significant hurdles. It underscores the need for further
advancements in the field and the importance of incorporating complementary
methods, such as retrieval augmentation, to enhance the learning of long-tail
knowledge in LLMs.

####  4.1.3 Contextual Influence and Knowledge Conflict

This sub-subsection examines the interplay between an LLM’s inherent
parametric knowledge and the provided contextual knowledge, exploring both the
model’s capacity to utilize context and its behavior when confronted with
conflicting information.

##### Contextual Influence on Generation

Some works explore the model’s capacity to utilize context, for example, Li et
al. (2023a) observe that larger models tend to rely on their parametric
knowledge, even when faced with counterfactual contexts. This suggests that as
models increase in size, they might grow more confident in their internal
knowledge, potentially sidelining external context. However, the introduction
of irrelevant contexts can still influence their outputs. The balance between
controllability (relying on relevant context) and robustness (resisting
irrelevant context) emerges as a challenge in LLM training. The study
indicates that reducing context noise improves controllability, but the effect
on robustness remains to be seen. In contrast, Zhou et al. (2023b) propose
prompt templates to guide LLMs towards more faithful text generation. Among
these, opinion-based prompts prove most effective, indicating that when LLMs
are queried about opinions, they adhere more closely to the context.
Interestingly, the study finds that using counterfactual context enhances the
model’s faithfulness, while original context, sourced from platforms like
Wikipedia, might induce a simplicity bias, leading LLMs to answer questions
without heavily relying on the context. Chen et al. (2023b) conduct a
comprehensive evaluation of LLMs’ ability to effectively utilize retrieved
information. The study reveals that while retrieved documents can boost LLM
performance, the presence of noise in these documents can hinder it. Yue et
al. (2023) investigate the nature of LLM-generated content in relation to
provided references. They categorize the generated content as attributable,
contradictory, or extrapolatory to the reference. Both fine-tuned models and
instruction-based LLMs struggle to accurately evaluate the alignment between
generated content and references, underscoring the challenge of ensuring that
LLMs produce content consistent with the provided context.

##### Handling Knowledge Conflicts

A series of studies are interested in LLMs’ behavior when confronted with
conflicting information. Longpre et al. (2021) introduce the concept of
knowledge conflicts, where the provided context contradicts the model’s
learned information. Their findings suggest that such conflicts lead to
increased prediction uncertainty, especially for in-domain examples.
Observations across models, ranging from T5-60M to 11B, indicate that larger
models tend to default to their parametric knowledge. Moreover, there’s an
inverse relationship between retrieval quality and the tendency to rely on
internal knowledge: the more irrelevant the evidence, the more the model
defaults to its parametric knowledge. Chen et al. (2022) conduct experiments
on typical ODQA models, including FiD and RAG. Their results show that FiD
models rarely resort to memorization (less than 3.6% for NQ) compared to RAG
models. Instead, FiD primarily grounds its answers in the provided evidence.
Interestingly, when confronted with conflicting retrieved passages, models
tend to fall back on their parametric knowledge. Xie et al. (2023a) explore
the behavior of recent LLMs, including ChatGPT and GPT-4. Contrary to findings
on smaller LMs, they discover that LLMs can be highly receptive to external
evidence, even if it contradicts their parametric memory, provided the
external evidence is coherent and convincing. Additionally, LLMs exhibit a
strong confirmation bias, especially when presented with evidence that aligns
with their parametric memory. This bias becomes even more pronounced for
widely accepted knowledge. In scenarios where no relevant evidence is
provided, LLMs tend to express uncertainty. However, when presented with both
relevant and irrelevant evidence, they demonstrate an ability to filter out
irrelevant information.

In conclusion, while studies like Li et al. (2023a) and Zhou et al. (2023b)
emphasize the challenges and potential solutions in making LLMs more context-
aware, others like Yue et al. (2023) and Xie et al. (2023a) highlight the
inherent biases and limitations of LLMs. The overarching theme is the need for
a balanced approach, where LLMs effectively leverage both their internal
knowledge and external context to produce accurate and coherent outputs.

###  4.2 Causes of Factual Errors

Understanding the root causes of these factual inaccuracies is crucial for
refining these models and ensuring their reliable application in real-world
scenarios. In this subsection, we delve into the multifaceted origins of these
errors, categorizing them based on the stages of model operation: Model Level,
Retrieval Level, Generation Level, and other miscellaneous causes. Table I
shows examples of factuality errors caused by different factors.

####  4.2.1 Model-level Causes

This subsection delves into the intrinsic factors within large language models
that contribute to factual errors, originating from their inherent knowledge
and capabilities.

##### Domain Knowledge Deficit

The model may lack comprehensive expertise in specific domains, leading to
inaccuracies. Every LLM has its limitations based on the data it was trained
on. If an LLM hasn’t been exposed to comprehensive data in a specific domain
during its training, it’s likely to produce inaccurate or generalized outputs
when queried about that domain. For instance, while an LLM might be adept at
answering general science questions, it might falter when asked about niche
scientific subfields (Lu et al., 2022).

##### Outdated Information

The model’s dependence on older datasets can make it unaware of recent
developments or changes. LLMs are trained on datasets that, at some point,
become outdated. This means that any events, discoveries, or changes post-
dating the last training update won’t be known to the model. For example,
ChatGPT and GPT-4 are both trained on data up to 2021.09 might not be aware of
events or advancements after then.

##### Immemorization

The model does not always retain knowledge from its training corpus. While
it’s a misconception that LLMs “memorize” data, they do form representations
of knowledge based on their training. However, they might not always recall
specific, less-emphasized details from their training parameters, especially
if such details were rare or not reinforced through multiple examples. For
example, ChatGPT has pretrained with Wikipedia, but it still fail in answering
some questions from NaturalQuestions (Kwiatkowski et al., 2019) and TriviaQA
(Joshi et al., 2017), which are constructed from Wikipedia (Wang et al.,
2023a).

##### Forgetting

The model might not retain knowledge from its training phase or could forget
prior knowledge as it undergoes further training. As models are further fine-
tuned or trained on new data, there’s a risk of “catastrophic forgetting”
(Goodfellow et al., 2015; Wang et al., 2022; Chen et al., 2020; Zhai et al.,
2023) where they might lose certain knowledge they were previously know. This
is a well-known challenge in neural network training, where networks forget
previously learned information when exposed to new data, which also happens in
large language models (Luo et al., 2023b).

##### Reasoning Failure

While the model might possess relevant knowledge, it can sometimes fail to
reason with it effectively to answer queries. Even if an LLM has the requisite
knowledge to answer a question, it might fail to connect the dots or reason
logically. For instance, ambiguity in the input (Liu et al., 2023a) can
potentially lead to a failure in understanding by LLMs, consequently resulting
in reasoning errors. In addition, Berglund et al. (2023) find the LLM traps in
the reversal curse, for example, it knows that A is B’s mother but fail to
answer who is B’s son. This is especially evident in complex multi-step
reasoning tasks or when the model needs to infer based on a combination of
facts (Tan et al., 2023b; Kotha et al., 2023).

####  4.2.2 Retrieval-level Causes

The retrieval process plays a pivotal role in determining the accuracy of
LLMs’ responses, especially in retrieval-augmented settings. Several factors
at this level can lead to factual errors:

##### Insufficient Information

If the retrieved data doesn’t provide enough context or details, the LLM might
struggle to generate a factual response. This can result in generic or even
incorrect outputs due to the lack of comprehensive evidence.

##### Misinformation Not Recognized by LLMs

LLMs can sometimes accept and propagate misinformation present in the
retrieved data. This is especially concerning when the model encounters
knowledge conflicts, where the retrieved information contradicts its pre-
trained knowledge, or multiple retrieved documents contradict each other Li et
al. (2015). For instance, Longpre et al. (2021) observed that the more
irrelevant the evidence, the more likely the model is to rely on its intrinsic
knowledge. Recent studies, such as Pan et al. (2023c), have also shown that
LLMs are susceptible to misinformation attacks within the retrieval process.

##### Distracting Information

LLMs can be misled by irrelevant or distracting information in the retrieved
data. For example, if the evidence mentions a ”Russian movie” and ”Directors”,
the LLM might incorrectly infer that ”The director is Russian”. Luo et al.
(2023a) highlighted this vulnerability, noting that LLMs can be significantly
impacted by distracting retrieval results. They further proposed instruction
tuning as a potential solution to enhance the model’s ability to sift through
and leverage retrieval results more effectively.

Additionally, when dealing with long retrieval inputs, the models tend to show
their best performance when processing information given at the beginning or
end of the input context, according to Liu et al. (2023c). In contrast, the
models are likely to experience a significant decrease in performance when
they are required to extract relevant data from the middle of these extensive
contexts.

##### Misinterpretation of Related Information

Even when the retrieved information is closely related to the query, LLMs can
sometimes misunderstand or misinterpret it. While this might be less frequent
when the retrieval process is optimized, it remains a potential source of
error. For instance, in the ReAct study (Yao et al., 2023a), the rate of
errors dropped significantly when the retrieval process was improved.

####  4.2.3 Inference-level Causes

##### Snowballing

During the generation process, a minor error or deviation at the beginning can
compound as the model continues to generate content. For instance, if an LLM
misinterprets a prompt or starts with an inaccurate premise, the subsequent
content can veer further from the truth (Zhang et al., 2023b; Varshney et al.,
2023).

##### Erroneous Decoding

The decoding phase is crucial for translating the model’s internal
representations into human-readable content (Chuang et al., 2023; Massarelli
et al., 2020). Mistakes during this phase, whether due to issues like beam
search errors or suboptimal sampling strategies, can lead to outputs that
misrepresent the model’s actual ”knowledge” or intention. This can manifest as
inaccuracies, contradictions, or even nonsensical statements.

##### Exposure Bias

LLMs are a product of their training data. If they’ve been exposed more
frequently to certain types of content or phrasing, they might have a bias
toward generating similar content, even when it’s not the most factual or
relevant. This bias can be especially pronounced if the training data has
imbalances or if certain factual scenarios are underrepresented (Felkner et
al., 2023; Gallegos et al., 2023). The model’s outputs, in such cases, reflect
its training exposure rather than objective factuality. For example, research
by Hossain et al. (2023) suggests that LLMs can correctly identify the gender
of individuals who conform to the binary gender system, but they perform
poorly when determining non-binary or neutral genders.

##  5 Enhancement

Figure 2: An overview of methods to enhance factuality in large language
models. This encompasses three primary areas: enhancement techniques for pure
LLMs, strategies for retrieval-augmented LLMs, and methods employed by domain-
specific LLMs to boost their factual accuracy within their respective domains.

This section discusses methods to enhance factuality in LLMs across different
phases, including LLM generation, retrieval-augmented generation, inference-
phase enhancements, and domain-specific factuality improvements, outlined in
Figure 2.

TABLE VIII: Performance of Select Factuality Enhancement Methods. The table displays performance metrics on various datasets for both baseline models and their enhanced counterparts, denoted with a $\rightarrow$. Due to space constraints, only a subset of datasets, metrics, and models from each work is presented. Reference | Dataset | Metrics | Baselines $\rightarrow$ Theirs |  |  | Dataset | Metrics | Baselines $\rightarrow$ Theirs
---|---|---|---|---|---|---|---|---
Li et al. (2022c) | NQ | EM | 34.5 $\rightarrow$ 44.35 (T5 11B) |  |  | GSM8K | ACC | 77.0 $\rightarrow$ 85.0 (ChatGPT)
Yu et al. (2023) | NQ | EM | 20.9 $\rightarrow$ 28.0 (InstructGPT) |  |  | TriviaQA | EM | 57.5 $\rightarrow$ 59.0 (InstructGPT)
|  |  |  |  |  | WebQA | EM | 18.6 $\rightarrow$ 24.6 (InstructGPT)
Chuang et al. (2023) | FACTOR News | ACC | 58.3 $\rightarrow$ 62.0 (LLaMa-7B) |  |  | FACTOR News | ACC | 61.1 $\rightarrow$ 62.5 (LLaMa-13B)
| FACTOR News | ACC | 63.8 $\rightarrow$ 65.4 (LLaMa-33B) |  |  | FACTOR News | ACC | 63.6 $\rightarrow$ 66.2 (LLaMa-65B)
| FACTOR Wiki | ACC | 58.6 $\rightarrow$ 62.2 (LLaMa-7B) |  |  | FACTOR Wiki | ACC | 62.6 $\rightarrow$ 66.2 (LLaMa-13B)
| FACTOR Wiki | ACC | 69.5 $\rightarrow$ 70.3 (LLaMa-33B) |  |  | FACTOR Wiki | ACC | 72.2 $\rightarrow$ 72.4 (LLaMa-65B)
| TruthfulQA | %Truth * Info | 32.4 $\rightarrow$ 44.6 (LLaMa-13B) |  |  | TruthfulQA | %Truth * Info | 34.8 $\rightarrow$ 49.2 (LLaMa-65B)
Li et al. (2022b) | TruthfulQA | %Truth * Info | 32.4 $\rightarrow$ 44.4 (LLaMa-13B) |  |  | TruthfulQA | %Truth * Info | 31.7 $\rightarrow$ 36.7 (LLaMa-33B)
| TruthfulQA | %Truth * Info | 34.8 $\rightarrow$ 43.4 (LLaMa-65B) |  |  |  |  |
Li et al. (2023d) | NQ | ACC | 46.6 $\rightarrow$ 51.3 (LLaMA-7B) |  |  | TriviaQA | ACC | 89.6 $\rightarrow$ 91.1 (LLaMA-7B)
| MMLU | ACC | 35.7 $\rightarrow$ 40.1 (LLaMA-7B) |  |  | TruthfulQA | %Truth * Info | 32.5 $\rightarrow$ 65.1 (Alpaca)
| TruthfulQA | %Truth * Info | 26.9 $\rightarrow$ 43.5 (LLaMa-7B) |  |  | TruthfulQA | %Truth * Info | 51.5 $\rightarrow$ 74.0 (Vicuna)
Cohen et al. (2023b) | LAMA | F1 | 50.7 $\rightarrow$ 80.8 (ChatGPT) |  |  | TriviaQA | F1 | 56.2 $\rightarrow$ 82.6 (ChatGPT)
| NQ | F1 | 60.6 $\rightarrow$ 79.1 (ChatGPT) |  |  | PopQA | F1 | 65.2 $\rightarrow$ 85.4 (ChatGPT)
| LAMA | F1 | 42.5 $\rightarrow$ 79.3 (GPT-3) |  |  | TriviaQA | F1 | 46.7 $\rightarrow$ 77.2 (GPT-3)
| NQ | F1 | 52.0 $\rightarrow$ 78.0 (GPT-3) |  |  | PopQA | F1 | 43.7 $\rightarrow$ 77.4 (GPT-3)
Weller et al. (2023) | TriviaQA | QUIP | 31.6 $\rightarrow$ 33.6 (ChatGPT) |  |  | NQ | QUIP | 32.8 $\rightarrow$ 34.3 (ChatGPT)
| HotpotQA | QUIP | 28.3 $\rightarrow$ 29.2 (ChatGPT) |  |  | ELI5 | QUIP | 24.1 $\rightarrow$ 26.5 (ChatGPT)
| TriviaQA | EM | 77.8 $\rightarrow$ 78.8 (ChatGPT) |  |  | NQ | EM | 32.9 $\rightarrow$ 34.8 (ChatGPT)
| HotpotQA | F1 | 35.7 $\rightarrow$ 36.6 (ChatGPT) |  |  | ELI5 | R-L | 22.7 $\rightarrow$ 21.7 (ChatGPT)
Dhuliawala et al. (2023) | MultiSpanQA | F1 | 39.0 $\rightarrow$ 48.0 (LLaMA 65B) |  |  | - | FactScore | 55.9 $\rightarrow$ 71.4 (LLaMA 65B)
|  |  |  |  |  | - | Avg. # facts | 16.6$\rightarrow$ 12.3 (LLaMA 65B)
Yao et al. (2023a) | HotpotQA | EM | 28.7 $\rightarrow$ 35.1 (PaLM-540B) |  |  | FEVER | ACC | 57.1 $\rightarrow$ 62.0 (PaLM-540B)
Jiang et al. (2023b) | 2WikiMultihopQA | EM | 28.2 $\rightarrow$ 51 (ChatGPT) |  |  | StrategyQA | EM | 72.9 $\rightarrow$ 77.3 (ChatGPT)
| WikiAsp | UniEval | 47.1 $\rightarrow$ 53.4 (ChatGPT) |  |  | ASQA | EM | 33.8 $\rightarrow$ 41.3 (ChatGPT)
|  |  |  |  |  | ASQA-hint | EM | 40.1 $\rightarrow$ 46.2 (ChatGPT)
Izacard et al. (2022) | MMLU | ACC | 42.4 $\rightarrow$ 56.3 (T5-770M) |  |  | MMLU | ACC | 50.4 $\rightarrow$ 59.9 (T5-3B)
|  |  |  |  |  | MMLU | ACC | 54 $\rightarrow$ 65.8 (T5-11B)
Shi et al. (2023) | MMLU | ACC | 68.3 $\rightarrow$ 73.2 (Codex) |  |  | NQ | EM | 40.6 $\rightarrow$ 45.5(Codex)
|  |  |  |  |  | TriviaQA | EM | 73.6 $\rightarrow$ 77.3 (Codex)

Table VIII provides a summary of enhancement methods and their respective
improvements over baseline LLMs. It’s essential to recognize that various
research papers may employ distinct experimental settings, such as zero-shot,
few-shot, or full settings. Consequently, when examining this table, it’s
important to note that performance metrics for different methods, even when
evaluating the same metric on the same dataset, may not be directly
comparable.

###  5.1 On Standalone LLM Generation

When focusing on standalone LLM generation, enhancement strategies can be
broadly grouped into three main categories:

_(1) Improving Factual Knowledge from Unsupervised Corpora_ (Sec 5.1.1): This
involves refining the training data during pretraining, such as through
deduplication and emphasizing informative words (Lee et al., 2022a).
Techniques like TOPICPREFIX (Lee et al., 2022b) and sentence completion loss
are also explored to enhance this approach.

_(2) Enhancing Factual Knowledge from Supervised Data_ (Sec 5.1.2): Examples
in this category include supervised fine-tuning strategies (Chung et al.,
2022; Zhou et al., 2023a) focus on finetuning with labelled data or
integrating structured knowledge from such as knowledge graphs (KGs) or making
precise adjustments to model parameters (Li et al., 2023d).

_(3) Optimally Eliciting Factual Knowledge from the Model_ (Sec 5.1.3, 5.1.4,
5.1.5): This category encompasses methods like Multi-agent collaboration (Du
et al., 2023) and innovative prompts (Yu et al., 2023). Additionally, novel
decoding methodologies, such as factual-nucleus sampling, are introduced to
further improve factuality (Lee et al., 2022b; Chuang et al., 2023).

Some work (Liu et al., 2023b; Sun et al., 2023d) aim for improve the
factuality of large multi-modal models, we choose to not emphasize them.

####  5.1.1 Pretraining-based

Pretraining plays a pivotal role in equipping the model with the factual
knowledge derived from the corpus. By emphasizing strategies during this
phase, the model’s inherent factuality can be significantly enhanced. This
approach is particularly crucial for addressing challenges like immemorization
and forgetting.

##### Initial Pretraining

Methods employed during the foundational pretraining of the model.

Lee et al. (2022a) develop two distinct tools for deduplicating training data,
addressing the issue of redundant texts and long repeated substrings present
in the training sets of current LLMs. These tools effectively reduce the
recall of memorized texts in model outputs. Remarkably, they achieve similar
or even superior accuracy with fewer training steps.

Sadeq et al. (2023) introduce a modification to the Masked Language Model
(MLM) training objective used in the pretraining of LLMs. They discover that
high-frequency words do not consistently contribute to the model’s ability to
learn factual knowledge. To address this, they devise a strategy that
encourages the language model to prioritize informative words during
unsupervised training. This is achieved by masking tokens more frequently
based on their informative relevance. To quantify this relevance, they utilize
Pointwise Mutual Information (PMI) (Fano and Hawkins, 1961), positing that
words with elevated PMI values, in relation to their adjacent words, are
likely to be more informatively pertinent. Experimental results indicate that
this innovative approach significantly bolsters the efficacy of pretrained
language models across various tasks, including factual recall, question
answering, sentiment analysis, and natural language inference, in a closed-
book setting.

##### Continual Pretraining

Iterative pretraining processes that allow the model to progressively refine
and update its knowledge base.

Lee et al. (2022b) introduce TOPICPREFIX as a pre-processing method and the
sentence completion loss as training objective. Some factual sentences may be
unclear when the LM training corpus is chunked, especially when these
sentences contain pronouns (e.g., she, he, it). So they prepend TOPICPREFIX
(e.g., Wikipedia document name) to sentences in the factual documents to
transform each sentence into an independent factual statement. They also
introduce the sentence completion loss, with the aim of enabling the model to
capture facts from entire sentences rather than just focusing on the
associations between subwords. For implementation, they establish a pivot $t$
for each sentence and require zero-masking for all token prediction losses
before $t$. This pivot is only necessary during the training phase.
Experiments show that such methods can further reduce the factual errors than
standard factual-domain adaptive training.

####  5.1.2 Supervised Finetuning

Supervised fine-tuning leverages labeled datasets to refine the model’s
performance. This approach serves a dual purpose: it imparts specific task or
knowledge base-oriented information to the model and addresses challenges like
immemorization and forgetting. Several studies, such as Chung et al. (2022);
Zhou et al. (2023a), emphasize the pivotal role of supervised fine-tuning in
eliciting the inherent knowledge of the base model, subsequently enhancing its
reasoning capabilities.

##### Continual SFT

A cyclic fine-tuning approach, where the model undergoes consistent refinement
using sequential sets of labeled data.

Moiseev et al. (2022) investigate how to inject structured knowledge from a
knowledge graph into LLMs. The approach involves directly training T5 using
triplets containing relationship knowledge. Previous methods often describe
the triplets using prompts and then train LLMs with masked language model
task, but some triplets are not easy to describe. They compare three
knowledge-enhancement fine-tuning methods, which are MLM training on the C4
corpus (Raffel et al., 2020b), masking the subject or object in KG triplets,
and masking the subject or object in the KELM corpus (Agarwal et al., 2021).
Experiments show that effectiveness of the latter two methods achieved better
exact match scores in closed-book QA tasks (Jiang et al., 2019; Welbl et al.,
2018; Joshi et al., 2017; Kwiatkowski et al., 2019). This demonstrates that
training directly based on KG triplets is an effective way to inject knowledge
into the model

Sun et al. (2023c) introduce negative samples and contrastive learning to
supervised finetuning process with MLE loss. The sources of these negative
samples are either from the Knowledge Graph or are generated by large language
models. While traditional contrastive learning can only function at the token
or sentence level, the intrinsic value of span information cannot be
overlooked, they employ Named Entity Recognition to extract this critical span
data. The training process incorporates a blend of MLE loss, standard
contrastive learning loss, and span-based contrastive learning loss, with the
parameters finely tuned to optimize results. Experiments indicate that the
method delivers performance comparable with SOTA KB-based methods but offers
significant benefits in efficiency and scalability.

Yang et al. (2023b) presents a development framework for Knowledge Graph-
enhanced Large Language Models (KGLLMs), drawing from established
technologies. They detail various enhancement strategies, including a before-
training enhancement that refines input quality by integrating factual data; a
during-training enhancement that synergizes textual and structural knowledge,
utilizing tools like graph neural networks and attention mechanisms; multi-
task learning which focuses on knowledge-guided pre-training tasks to bolster
the factual knowledge acquisition of LLMs; and a post-training enhancement
that fine-tunes LLMs for domain-specific tasks using knowledge-rich data.
Additionally, the significance of prompt learning in LLMs is highlighted,
emphasizing the importance of choosing appropriate prompt templates. They also
suggest knowledge graphs as a valuable resource for crafting these templates
to harness domain-specific knowledge.

##### Model Editing

Instead of directly finetuning the model, model edit is a more precise
approach to enhance the model’s factuality. By editing specific areas that are
related to the fact, the model can correctly express that fact without
compromising other unrelated knowledge. Current editing methods (Yao et al.,
2023b) can be categorized into weight-preserved and weight-modified paradigms.
When modifying the model’s weight, KN (Dai et al., 2022) and ROME (Meng et
al., 2022) first analyze representations to locate those underlying factual
errors and then directly update the relevant weights. Meanwhile, KE (De Cao et
al., 2021) and MEND (Mitchell et al., 2022a) employ a hypernetwork to learn
the necessary weight changes. While effective, the robustness and
generalization of directly updating weights remains an open question.

Li et al. (2023d) present a technique known as Inference-Time Intervention
(ITI) designed to boost the factual accuracy of large language models. ITI
adjusts model activations during inference to enhance the truthfulness of
responses. This method, categorized under activation editing, is both
adjustable and less intrusive, setting it apart from weight editing
techniques. Drawing inspiration from prior research, they utilize steering
vectors, which are proven effective for style transfer, to guide model
activations. They detail a multi-step model selection process involving the
calibration of intervention strength hyperparameters, pinpointing truth-
telling related heads, and determining their truth-telling directions. The
TruthfulQA dataset serves as the foundation for training and validation, with
a rigorous 2-fold cross-validation employed to prevent data leakage.
Experiments on the TruthfulQA benchmark demonstrate that ITI improves Alpaca’s
(Taori et al., 2023) truthfulness from 32.5% to 65.1%.

####  5.1.3 Multi-Agent

Engaging multiple models in a collaborative or competitive manner, enhancing
factuality through their collective prowess, helps in the immemorization and
reasoning failure problems.

Du et al. (2023) propose an approach to enhance the performance of language
models by treating different LLMs as intelligent agents engaged in multi-agent