#  RefChecker: Reference-based Fine-grained Hallucination Checker and
Benchmark for Large Language Models

Xiangkun Hu1, Dongyu Ru1, Lin Qiu1, Qipeng Guo2, Tianhang Zhang1
Yang Xu3, Yun Luo4, Pengfei Liu3, Yue Zhang4 and Zheng Zhang1
1Amazon AWS AI 2Shanghai AI Lab
3Shanghai Jiaotong University 4Westlake University
{xiangkhu, rudongyu, quln, zzthang<EMAIL_ADDRESS><EMAIL_ADDRESS>
{xuyang0112<EMAIL_ADDRESS>{luoyun<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have shown impressive capabilities but also a
concerning tendency to hallucinate. This paper presents RefChecker, a
framework that introduces _claim-triplets_ to represent claims in LLM
responses, aiming to detect fine-grained hallucinations. In RefChecker, an
extractor generates claim-triplets from a response, which are then evaluated
by a checker against a reference. We delineate three task settings: Zero,
Noisy and Accurate Context, to reflect various real-world use cases. We
curated a benchmark spanning various NLP tasks and annotated 11k claim-
triplets from 2.1k responses by seven LLMs. RefChecker supports both
proprietary and open-source models as the extractor and checker. Experiments
demonstrate that claim-triplets enable superior hallucination detection,
compared to other granularities such as response, sentence and sub-sentence
level claims. RefChecker outperforms prior methods by 6.8 to 26.1 points on
our benchmark and the checking results of RefChecker are strongly aligned with
human judgments111This work is open sourced at https://github.com/amazon-
science/RefChecker.

##  1 Introduction

Large Language Models (LLMs) have sparked a revolution in Natural Language
Processing (NLP), covering diverse tasks with a unified architecture Zhao et
al. (2023). However, LLMs exhibit a tendency to generate hallucinated contents
that can be difficult to discern, posing a potential risk of misleading users.
Huang et al. (2023). Hallucination detection is therefore an important task
Manakul et al. (2023); Min et al. (2023); Chern et al. (2023).

Figure 1: An example response split into sentence, sub-sentence Min et al.
(2023), triplets, and the hallucination _1983_. Triplets define the boundary
of claims more clearly, are fine-grained and covers non-overlapping facts
(unlike sub-sentences). Figure 2: The RefChecker framework comprises two main
components: an extractor denoted as $E$ and a checker denoted as $C$. Given a
text to be checked, typically a response generated by an LLM, the extractor
takes it as input and generates a set of knowledge triplets, referred to as
claim-triplets. Subsequently, the checker assesses each claim-triplet by
comparing it against a reference, assigning a hallucination label based on the
evaluation.

Detecting hallucination is essentially a job of comparing a response against a
reference. However, several challenges remain: determining the appropriate
unit of analysis for comparison, building a comprehensive benchmark reflecting
real-world LLM applications, developing a unified, automated framework that
scales detection across diverse tasks. In this work, we introduce RefChecker
to tackle these challenges.

In terms of checking granularity, response level checking Lin et al. (2022);
Li et al. (2023) suffices if the query and response is about a simple fact.
However, when responses are complex and long, response-level checking is not
only uninformative but can also cause false-negative when hallucination is
local. This is common in real-world use cases, for example, the response from
Llama 2 Touvron et al. (2023) in our benchmark (described later) contains 150
tokens on average. For fine-grained detection, Manakul et al. (2023) takes the
sentences in a response as the checking units. Min et al. (2023) and Chern et
al. (2023) further extracts short phrases (we term them as sub-sentences) as
the claims, as one sentence may contain multiple hallucinations, or one
hallucination may span across sentence boundaries. However, sub-sentences are
structurally difficult to define, making it challenging to form high-quality
demonstrations to be used by LLMs with in-context learning. To this end, we
propose to extract knowledge triplets as checking units. We show an example
with different granularity in Figure 1. Triplets exhibit fine-grained and
clearly separated semantics. These triplets are called _claim-triplets_.
Experiments show that checking with claim-triplets gains 4 to 9 points of
improvement over other granularity on our benchmark (cf. Sec. 5.1).

For better evaluation, we curate a comprehensive dataset on which we can
benchmark hallucination under different context quality and availability.
Using this benchmark, we conducted human evaluation on 2,100 responses from 7
LLMs. We annotated 11k claim-triplets with 95% Inter-Annotator Agreement on
23% of the annotations. Compared with recent proposed benchmarks Manakul et
al. (2023); Min et al. (2023); Chern et al. (2023), it covers a more diverse
range of domains and tasks, with more LLMs and responses evaluated (see Table
1). As expected, we found by human evaluation that hallucination is the most
pronounced (cf. Appendix A.4) when LLMs are asked to generate responses solely
from its memory (Zero Context), followed by responding to noisy references in
RAG (retrieval augmented generation) setting Shuster et al. (2021) (Noisy
Context) and finally when references are more or less noisy free (Accurate
Context).

RefChecker (Figure 2) is a fully automated framework that scales hallucination
detection across different tasks. The extractor generates claim-triplets from
the response and the checker evaluates each of the claim-triplets by comparing
them with the reference. In contrast to recent work that only differentiates
factual and non-factual claims, the checker in RefChecker also considers
unverifiable claims when the reference is insufficient for checking. Both the
extractor and checker supports proprietary (e.g. GPT-4 OpenAI et al. (2023) or
Claude 2 Anthropic (2023)) and open-source models (e.g. Mistral Jiang et al.
(2023) and RoBERTa Liu et al. (2019) based models). We made careful study to
select and recommend configurations that give results consistent with human
annotation, and show 6.8 to 26.1 points of improvement over the best
alternative (Sec. 5.2).

Our key contributions include:

  * •

Claim-triplet formulation: Our novel “claim-triplet” analysis outperforms
existing methods by up to 9 points, pinpointing factual inconsistencies within
responses.

  * •

Comprehensive benchmark: We developed a robust benchmark covering three
classes of real-world LLM tasks with 11,000 manually annotated claim-triplets
across 7 LLMs.

  * •

Automatic checking framework: Our RefChecker framework extracts and verifies
claim-triplets, boosting consistency by 19-26 points over prior methods and
works with both proprietary and open-source models.

##  2 Related Work

Method | Context Setting | Claim Extraction | Checking | Benchmark
---|---|---|---|---
Claim | Extractor | Label | Checker | Domain | Task | Evaluated Responses
SelfCheckGPT | Zero Context | Sentence | - | 3-way | GPT | Wikipedia | Bio Generation | 238 from GPT-3
FActScore | Zero Context | Sub-sentence | GPT | Binary | GPT | Wikipedia | Bio Generation | 505 from 3 LLMs
FacTool | Zero Context |  Sub-sentence Snippet Statement Tuple | GPT | Binary | GPT |  Wikipedia Python Math Sci. Text |  QA Code Generation Math Problems Sci. Review | 514 from ChatGPT
RefChecker |  Zero Context Noisy Context Accurate Context | Triplet |  GPT Claude Mixtral* Mistral* | 3-way |  GPT Claude AlignScore* NLI* RepC* |  Wikipedia Web |  QA RAG Summarization IE | 2,100 from 7 LLMs

Table 1: A comparison of RefChecker with previous approaches for hallucination
detection. The “*” symbols alongside the extractors and checkers indicate that
these models are open-sourced. RefChecker uses triplets as the claims instead
of sentences or sub-sentences. The RefChecker benchmark covers more context
settings and more diverse tasks. The human evaluation covers more LLMs and
responses. RefChecker pipeline supports both proprietary and open-source
models, facilitating broader adoption across various applications.

We undertake a review of prior work relevant to our study and compare them
with RefChecker. The comparative analysis with three representative methods is
encapsulated in Table 1.

#### Hallucinations in LLMs

Hallucinations frequently occur in NLP tasks like summarization Maynez et al.
(2020); Cao et al. (2022), machine translation Guerreiro et al. (2023a, b),
dialog systems Honovich et al. (2021); Dziri et al. (2022) and RAG Shuster et
al. (2021). According to a recent comprehensive survey Huang et al. (2023),
hallucinations can be categorized to factuality hallucinations and
faithfulness hallucinations. Factuality hallucinations involve claims
contradicted by real-world facts, while faithfulness hallucinations are
inconsistent with the input content. Recent research on hallucination
detection primarily concentrates on factuality hallucinations, such as
SelfCheckGPT Manakul et al. (2023), FActScore Min et al. (2023) and FacTool
Chern et al. (2023). We address both factuality and faithfulness
hallucinations and further categorizing them into three contextual settings to
align with real-world use cases.

#### Granularity of Claims

Claims are pivotal for evaluating responses generated by LLMs. Response level
checking Lin et al. (2022); Li et al. (2023) is too coarse-grained for long-
form responses. For fine-grained detection, sentence level Manakul et al.
(2023) and sub-sentence level checking Min et al. (2023); Chern et al. (2023)
have been proposed. However, these approaches still face limitations, as
discussed in Sec. 1. In this paper, we employ knowledge triplets extracted
from responses as claims which provide a structured framework for defining
claim granularity.

#### Hallucination Checking

One line of work for hallucination checking focus on resource-free checking
with no requirements of reference. They mainly depend on self-contradiction
Mündler et al. (2023) or uncertainty Zhang et al. (2023b) of the LLMs, or
self-consistency between randomly sampled responses Manakul et al. (2023);
Chen and Mueller (2023); Zhang et al. (2023a). The effectiveness of these
methods depends on the LLM-based checker’s capability including knowledge
coverage, instruction following ability and calibration. Furthermore, the
self-consistency based methods needs to sample several responses for cross
checking, which is costly. RefChecker is aligned with another line of work
which requires references to check with Min et al. (2023); Chern et al.
(2023). In addition, RefChecker adopts a 3-way classification framework to
cover unverifiable claims as opposed to the binary classification used in
previous work, which can only distinguish factual and non-factual claims.
Moreover, we have introduced open-source solutions for both claim extraction
and verification, diverging from the predominant reliance on proprietary LLMs
in prior research.

#### Hallucination Detection Benchmarks

The existing benchmarks for hallucination detection primarily focus on
response-level detection Lin et al. (2022); Yang et al. (2023), or limited to
specific domains and tasks Manakul et al. (2023); Min et al. (2023), or solely
address factuality hallucinations Chen et al. (2023); Chern et al. (2023);
Wang et al. (2023). In contrast, our proposed benchmark offers a broader
scope, encompassing a diverse range of tasks and domains. Moreover, our human
evaluation process involves a more extensive examination of various LLMs with
more responses.

##  3 RefChecker: Definition and Benchmark

Figure 3: Illustration of three settings of context, tasks and references.
Zero Context is about seeking factual knowledge from the internal memory of
the LLMs. Noisy Context has context information retrieved from a knowledge
source, which is a RAG use case. Accurate Context has context provided in the
input prompt. For Noisy and Accurate Context, we take the input context as the
reference.

Hallucinations are claims made by LLMs not supported by factual knowledge,
which we refer to as references; detecting hallucinations involves comparing
the claims against the references. This process depends on context settings,
granularity of checking and how we categorize the hallucinations. We will
discuss them in turn.

###  3.1 Context Settings and Benchmarks

We differentiate three context settings covering various tasks and employ
different benchmarks for each setting.

#### Zero Context (ZC)

Tasks in this setting require the LLM to respond solely based on its internal
knowledge. Therefore, in principle, references should be in the training
corpus. In practise, we use NaturalQuestions (NQ) Kwiatkowski et al. (2019) as
proxy dataset to represent such stable knowledge. The knowledge seeking
questions from NQ serve as prompts to perform the closed-book question
answering task, and the annotated long answer as the reference for a fair
evaluation of LLMs and comparison between different approaches. These
references are paragraphs from Wikipedia articles, which are widely used for
pre-training LLMs Zhao et al. (2023), making them a suitable proxy for the
internal knowledge of these LLMs.

#### Noisy Context (NC)

In this setup, the LLM receives additional context retrieved from some
external knowledge source, which may contain noisy or irrelevant information.
NC is also known as RAG, a crucial use case frequently encountered in real-
world applications. We utilize questions sourced from MS MARCO Nguyen et al.
(2016). Each question in this dataset is accompanied by a list of documents
retrieved from the internet, serving as the input context.

#### Accurate Context (AC)

This setting is similar to NC but the reference is typically noise-free.
Examples include text summarization, closed-QA and information extraction
tasks. We employ a subset from the databricks-dolly-15k Conover et al. (2023)
instruction tuning dataset which covers the 3 tasks mentioned before.

Figure 4: Definition of fine-grained hallucinations in an LLM-generated
response compared with references. The intersections of the response and the
references are the claims either supported (Entailment) or refuted
(Contradiction) by the references. The remaining parts in the response are
claims not verifiable by the references (Neutral). The other parts of the
references are the content not mentioned in the response.

For both Noisy and Accurate Context settings, the prompt is the reference
followed by the query, whereas for Zero Context, the prompt is just the query
(Figure 3). The benchmark contains 300 examples in total, 100 for each
setting, and we summarize it in Table 6. The details of the benchmark curation
process are described in Appendix A.

###  3.2 Claim-Triplets and Definition of Hallucination

Informally, claims are the units for the checking. This work explores the
approach of representing claims with knowledge triplets. This concept is
inspired by the field of knowledge graph studies, where triplets are employed
to encapsulate factual knowledge units. Knowledge triplets adopt a
(head_entity, relation, tail_entity) structure to capture fine-grained
information within the response. We call the triplet-format claims as _claim-
triplets_ , examples of which are shown in Figure 2.

Subsequently, the claim-triplets are compared with a reference to determine
the type of hallucinations, as illustrated in Figure 4. If a claim-triplet can
be directly inferred from the reference, we classify it as Entailment.
Conversely, if it contradicts the information in the reference, it is labeled
as Contradiction. However, in cases where the reference is insufficient to
verify the claim-triplets, we classify it as Neutral. In this study, we focus
on verifying hallucinations in the response and do not consider unmentioned
aspects in the reference, which may also be important for certain tasks.

###  3.3 Human Evaluation

We performed a human evaluation of responses generated by seven LLMs on this
benchmark dataset, including GPT-4, GPT-3.5-Turbo OpenAI (2022), InstructGPT
(text-davinci-001) Ouyang et al. (2022), Claude 2, Llama 2 70B Chat, Falcon
40B Instruct Almazrouei et al. (2023) and Alpaca 7B Taori et al. (2023). The
process involves three steps: gathering responses, extracting claim-triplets
with an extractor, and asking human annotators to evaluate these claim-
triplets. We annotated a total of 11k claim-triplets for 2.1k responses. 23%
of the claim-triplets were double annotated, with 95.0% Inter-Annotator
Agreement. See Appendix A.3 for the details of the annotation process.

A significant finding from the human evaluation is the crucial role of
contextual information for factual responses. On average, the rate of
Contradiction decreased from 25% in the absence of contextual cues (ZC) to 13%
with NC, and further reduced to 6% with AC, which reflects the necessity of
distinguishing the three context settings for separate study (cf. Appendix
A.4).

##  4 RefChecker Framework

As illustrated in Figure 2, the RefChecker framework is designed as a 2-stage
pipeline: an Extractor $E$ decomposes the LLM response into a set of triplets,
with each of them verified by the Checker $C$. The categorization of the
triplets can be optionally aggregated according to specified rules. We explain
them in the subsequent subsections.

###  4.1 Extractor

Our checking framework hinges on a key assumption: the decomposition of the
original text into triplets facilitates finer-grained detection and more
accurate evaluation. The extraction of these triplets plays a pivotal role in
achieving this objective. We apply LLMs to extract knowledge triplets from the
given text. We began with GPT-4 and Claude 2 and, for both cost and efficiency
concern, Mixtral 8x7B and Mistral. More specifically, we performed knowledge
distillation to train a 7B Mistral-based extractor with Mixtral 8x7B as the
teacher. We conducted supervised fine-tuning on 10k responses. Evaluation in
Sec. 6.1 shows competitive extraction quality of the open-source extractor.
Please refer to Appendix B.1 for prompts used for extraction and details on
extractor training.

###  4.2 Checker

We experimented with two families of checkers, the first is off-the-shelf LLMs
including GPT-4 and Claude 2 (see Appendix B.2 for prompts), and the second is
smaller NLI models including AlignScore Zha et al. (2023) and RoBERTa-
NLI.222https://huggingface.co/ynie/roberta-large-
snli_mnli_fever_anli_R1_R2_R3-nli Long references in AC/NC setting are split
to fit into small context windows of these small models (e.g. 200 tokens), and
the results are aggregated later.

Mistral 7B Jiang et al. (2023), unlike GPT-4/Claude 2, performs poorly as a
zero-shot checker, its performance improves with demonstrations but is still
not satisfactory. However, the fact that the model weight is open gives us the
opportunity to improve it by fine-tuning with NLI data. There are many options
we have experimented: 1) fine-tune by adding small amount of new parameters
using LoRA (LoRA-sft) Hu et al. (2021), 2) attach a shallow classifier, eg.
SVM, 2-layer MLP, KNN after NCA projection Goldberger et al. (2004), on top of
the internal states of the model. We call such checker RepC (for
Representation-based Classifier). Such states can be selected from one layer
(layer selection, LS) or an ensemble of all layers (layer ensemble, LE). As we
will report in Sec. 6.2, RepC checkers are competitive in general.

###  4.3 Aggregation

Triplet results can be aggregated to obtain the ratio of each category,
therefore gives an overall measure of hallucination distribution in a
response. To derive the performance of a particular LLM, we take a macro
average on Entailment/Neutral/Contradiction ratios of all responses. If a
scalar is preferred, we can assign certain numeric values to the catogories,
for instance $-1,0,1$ for contradictory, neutral and entail, respectively.

The aggregation can be customized and this is one of the benefits of the fine-
grained hallucination design in RefChecker. For instance, to compare against
other response-level approaches (cf. Sec. 5.1), we adopt a rule where the
response is flagged as contradictory if any one of the claim triplet is
contradictory.

| Zero Context | Noisy Context | Accurate Context
---|---|---|---
| Pearson | Spearman | Pearson | Spearman | Pearson | Spearman
SelfCheckGPT | 35.40 | 43.15 | 36.31 | 32.15 | 40.23 | 32.55
FActScore | 42.58 | 45.60 | 33.36 | 29.91 | 27.80 | 27.05
FacTool | 59.78 | 62.57 | 46.35 | 38.69 | 31.41 | 32.82
RefChecker |  |  |  |  |  |
Claud 2 + GPT-4 | 83.69 | 82.99 | 53.14 | 47.89 | 60.99 | 58.96
Mistral-SFT + AlignScore | 75.81 | 74.16 | 53.88 | 45.09 | 46.34 | 43.22
Table 2: Automated checking results of RefChecker and previous approaches. The
results of RefChecker are from the best performing combinations (extractor +
checker) of purely proprietary (blue) and purely open-source models (orange).

##  5 Experiments

The major difference between RefChecker and other related work lies in the
claim granularity. So we conduct experiments to differentiate granularity
first, then evaluate the whole framework.

###  5.1 Comparing with Other Granularity

Figure 5: Performance statistics of 7 checkers under different claim
granularities on 2.1k manual annotated responses. The detailed checker
performance can be found in Table 7 of Appendix B.2.

Previous works use different granularity for hallucination detection,
including response, sentence and sub-sentence levels (cf. Sec. 2 and 3.2). We
compare with them to verify the effectiveness of checking on facts with the
triplet format.

To make the results with different granularities comparable to each other, we
first breakdown the 2.1k annotated responses into different granularities,
then collect corresponding checker predictions respectively, and finally
aggregate finer-level results all into the response-level. We utilize a strict
aggregation rule with zero-tolerance on hallucinations, which means we apply
max-pooling (Entailment $<$ Neutral $<$ Contradiction) over claim predictions
within a response. We compare the results of 7 checkers, including 4 baseline
checkers (RoBERTa-NLI, AlignScore, GPT-4 and Claude 2) and 3 RepC-LE checkers
with KNN, SVM and 2-layer MLP classifiers respectively. The evaluation metric
is macro-f1 on three categories.

As shown in Figure 5, checking at triplet-level claims is superior over other
granularities, with a significant lead against response-level (10 pts macro-f1
score on average). Checking at sentence-level improves over response-level by
5 pts. However, we see a 3.5 pts drop moving to sub-sentence, one of the
reasons being sub-sentence claims can overlap. Apparently, the flexibility of
sub-sentences leads to poor quality of claim extraction, which subsequently
affects checking.

###  5.2 Comparing with Other Approaches

To contrast with other hallucination detection methodologies, we compare
RefChecker with three popular approaches on our benchmark, SelfCheckGPT
Manakul et al. (2023), FActScore Min et al. (2023) and FacTool Chern et al.
(2023). We convert metrics in these approaches to “hallucination rates” as
follows:

  * •

SelfCheckGPT: the average score of the sentences within a response. The score
is 0, 0.5 and 1 for an accurate, minor_inaccurate, and major_inaccurate,
respectively.

  * •

FactScore/FacTool: the proportion of claims not supported (FactScore) or non-
factual (FacTool) by the reference.

  * •

RefChecker: the proportion of Neutral and Contradiction claims.

Table 2 presents the Pearson and Spearman correlations between the
hallucinations rates as evaluated by human and the model results. It reveals
that RefChecker significantly outperforms previous methods across all three
context settings with both proprietary and open-source models. Specifically,
the combination of Claude 2 + GPT-4 outperforms the best alternative, FacTool,
by 6.8 to 26.1 points. We additionally apply RefChecker to SelfCheckGPT’s
dataset, and observe that in 11 out of the 15 combinations (73%) of RefChecker
outperform SelfCheckGPT (cf. Table 13 in Appendix C).

Figure 6: Spearman’s rank correlation coefficients between RefChecker and
human evaluation. Results are grouped regarding extractors and checkers used.
Results for each extractor (column) and checker (row) are arranged into a sub-
matrix in the figure, with values for 4 context settings (one additional for
average) and 3 ranking metrics.

###  5.3 Evaluation on RefChecker Framework

To validate the robustness and reliability of RefChecker, we checked 7 LLMs’
responses on our benchmark and compared rankings (ratios macro-averaged on
responses of each LLM) by RefChecker and humans with Spearman’s rank
correlation coefficient. The configuration space consists of different
combinations of extractor + checker, and also the 3 task settings as well as
their averages. The results are reported in Figure 6.

We observe that the combination of Claude 2 + GPT-4 is the most competitive
option with very strong correlations across all settings, benefiting from more
powerful LLMs. Replacing the extractor with our Mistral alternative yields a
marginal decline of performance, yet still maintains very strong correlations
in most settings. The best non-proprietary combination is Mistral +
NLI/AlignScore checker (356M), which achieve consistently strong correlations
in all settings. The Mistral-RepC checker is robust against different
extractors, owing to its stronger reasoning capability than small NLI-based
checker. This result serves as a guide for choosing a configuration tailored
to the user’s preferences. These preferences may include factors such as
budget, deployment simplicity, specific settings, types of hallucination,
privacy and requirements for open-source models.

##  6 Analysis and Discussion

We evaluate each component in RefChecker separately and discuss its
limitations and corresponding future work in this part.

###  6.1 Evaluation on Extractors

Extractor Model | GPT-4 Turbo Evaluation | Human Evaluation
---|---|---
Precision | Recall | F1 | Precision | Recall | F1
GPT-4 | 97.2 | 92.5 | 94.2 | 98.2 | 92.2 | 94.8
Claude 2 | 95.5 | 91.8 | 93.0 | 97.4 | 94.5 | 95.7
Mixtral | 87.7 | 85.2 | 85.5 | 87.6 | 85.5 | 85.4

Table 3: Claim extraction evaluation by GPT-4 Turbo and human on 30 samples.

Extractor Model | Precision | Recall | F1 |  Speed (sec/iter)
---|---|---|---|---
Mistral | 82.2 | 68.2 | 71.3 | 1.7
Mistral-SFT | 90.5 | 84.8 | 86.4 | 1.7
Mixtral | 86.7 | 80.2 | 81.6 | 5.7
Claude 2 | 89.8 | 86.6 | 87.0 | 6.9
GPT-4 | 92.4 | 88.6 | 89.3 | 8.7

Table 4: Automatic evaluation results of extractors. Mistral-SFT refers to our
Mistral-based extractor after supervised fine-tuning. The other extractors
directly prompt corresponding LLMs with two in-context examples. The best and
the second best results are bolded and underlined, respectively.

To ensure precise hallucination detection, it requires precise claims that
faithfully represent the facts in the original response. Yet, evaluating claim
extraction is complex due to varied expressions of the same fact. To address
this, we employ an automatic evaluation pipeline utilizing GPT-4 Turbo
(gpt-4-1106-preview) to lessen the need for post-hoc human evaluation for each
extractor.

Models |  Average of three settings |  Zero-context (NQ) |  Noisy-context (MS MARCO) |  Accurate-context (databricks-dolly-15k)
---|---|---|---|---
Accuracy | Macro-F1 | Accuracy | Macro-F1 | Accuracy | Macro-F1 | Accuracy | Macro-F1
| Baseline Checkers
RoBERTa-NLI | 76.56 | 55.88 | 74.06 | 69.90 | 78.36 | 46.67 | 77.27 | 51.06
AlignScore | 78.85 | 59.45 | 73.40 | 70.28 | 78.86 | 50.42 | 84.30 | 57.66
GPT-4 | 74.77 | 59.80 | 67.46 | 66.10 | 76.67 | 55.49 | 80.17 | 57.80
Claude 2 | 51.98 | 36.55 | 43.42 | 42.90 | 40.35 | 25.89 | 72.18 | 40.87
| Mistral-based Checkers
zero-shot | 69.43 | 46.64 | 70.83 | 61.10 | 71.75 | 43.01 | 65.72 | 35.81
1-shot | 76.68 | 50.66 | 65.44 | 63.25 | 81.23 | 42.18 | 83.38 | 46.56
LoRA-sft-n4000 | 77.84 | 57.98 | 77.43 | 73.64 | 79.21 | 50.29 | 76.89 | 50.00
RepC-LE-svm-n1000-e1000 | 79.03 | 60.05 | 77.98 | 73.53 | 79.56 | 51.29 | 79.54 | 55.34
RepC-LE-nn-n2000-e2000 | 81.27 | 60.80 | 75.23 | 71.98 | 82.08 | 47.56 | 86.50 | 62.86

Table 5: Checker evaluation results on 11k human annotated claim triplets. In
Mistral-based checkers, the model names start with the variant types, eg.
LoRA-sft indicates the LoRA fine-tuned variant and RepC-LE-nn indicates the
representation based classification variant using layer ensemble with 2-layer
MLP as the classifier. Here “nxxx” and “exxx” indicates the number of training
samples and ensemble learning samples. Due to the space limitation, we do not
include all variant results here, please refer to Table 8 of Appendix B.2 for
full results.

We employed GPT-4 Turbo to label each extracted claim as True/False,
indicating faithfulness to the original semantics. Additionally, we tasked it
with completing missing claims, enabling automatic calculation of precision,
recall, and F1 score. To validate results, we conducted a human evaluation on
30 random samples, ensuring agreement between human annotators and the model.
The comparison in Table 3 demonstrates strong alignment between human and
automatic evaluations, achieving 93.7% agreement on precision and 91.9% on
recall.

Leveraging the reliability of our automatic evaluation pipeline, we evaluated
the performance of four extractors (see Table 4). Our open-source Mistral
extractor achieves performance comparable to Claude 2 extractor with faster
inference speed and no need for API tokens.

###  6.2 Evaluation on Checkers

As described in Sec. 4.2, the baseline checkers we include in the evaluation
are RoBERTa-NLI, AlignScore, GPT-4 and Claude 2. The Mistral-based checkers we
include are zero-shot prompted, one-shot prompted, LoRA fine-tuned and RepC-LE
variants. The training and development data of these variants are 4k samples
from the ANLI dataset Nie et al. (2020). We evaluate their performance using
the 11k manually annotated claim triplets. The evaluation metric is accuracy
and macro-f1 score over 3-way classification.

Table 5 shows the evaluation results. Among the baseline checkers, AlignScore
is a strong competitor to GPT-4, and Claude 2 has a significant gap. We found
Claude 2’s neutral F1 score is very low (less than 20%) (cf. Table 9 in
Appendix B.2), with a tendency to flag neutral claims as contradiction, as a
result of biasing towards its own internal knowledge when asked to perform
checking (cf. Appendix D).

Besides, the Mistral-based checkers can often gives the best performance,
though there does not yet exist a single winner across the board. The weakness
of Mistral-based checkers lies in the NC setting. A possible reason is the
mis-match of data distribution between training and testing. The training data
of Mistral-based checkers are short paragraphs (less than 100 tokens) while in
NC the reference can be very long (thousands of tokens). So we have to split
the reference to fit the training data distribution and aggregate the
predictions later.

However, there are clear gaps between the performance of the checkers on NC
and AC in contrast to their performance on ZC, which suggests ample room of
improvement for the checkers.

###  6.3 Future Work

There remain several limitations of RefChecker that require addressing: a) The
triplet format of claims, while effectively breaking down LLM responses into
finer granularity, can be overly restrictive and may not have the flexibility
to cover important semantics (consider (Trump, president of, US), which is
factual in 2018, but non-factual in 2022). b) While extraction is relatively
simple with the outstanding comprehension capability of LLMs, there exists a
large improvement space for more powerful checkers. c) RefChecker has a
rudimentary support for source attribution (cf. Appendix B.3). Better source
attribution is critical to lend explainable and provide training signal to
mitigate hallucination. d) We found that model-based checkers may exhibit bias
towards internal knowledge, declaring a neutral claim to be entailment or
contradiction (cf. Table 9 and Appendix D). This requires we inject some forms
of “source control” in LLMs. e) In actual deployment cases, we found users ask
for stronger customizability (e.g. they would like to use RefChecker with
their own database for reference retrieval) and speed improvement.

##  7 Conclusion

We introduce RefChecker, a unified framework for detecting hallucination in
LLM responses . RefChecker operates at the level of knowledge triplets, termed
claim-triplets, extracted from LLM responses, allowing for fine-grained
detection. These claim-triplets are then evaluated against references to
determine their hallucination categories. An automated pipeline pairs an
extractor and a checker to identify potential hallucinations, calibrated to
match human annotations, yielding superior performance compared to prior
methods.

## Limitations

a) We have anecdote evidence that sometimes hallucination is due to reasoning
and limited context-window. These advanced form of hallucination is difficult
to be dealt with using triplet which bias towards local contexts. b) At its
current form, RefChecker primarily focuses on plain text in general domains.
Exploring extensions to include various data formats (table, code, math, etc.)
and specific domains (business, medical, legal, etc.) is worthy of
consideration.

## Ethics Statement

We contend that RefChecker poses no negative ethical implications for the
public; rather, it holds the potential for positive impact by enabling the
identification of non-factual content within the responses generated by large
language models (LLMs). This capability contributes to the cultivation of
responsible AI practices for the benefit of society.

In this study, we utilized a variety of scientific resources to conduct our
research and aim to contribute additional artifacts to the community.
Specifically, to curate the benchmark dataset, we sample 100 examples from
each of the following datasets:

  * •

The development set of NaturalQuestions dataset, which is under Creative
Commons Share-Alike 3.0 License.

  * •

The validation set of MS MARCO dataset, which is under Creative Commons
Attribution 4.0 International License.

  * •

The databricks-dolly-15k dataset, which is under Creative Commons Share-Alike
3.0 License.

These datasets are publicly accessible and utilize English language corpora.
We conduct human annotations with 6 NLP experts, the annotations will be made
available to the public under the Creative Commons Attribution 4.0
International License.

The fine-tuned Mistral 7B extractor, Mistral-SFT, is based on 10k questions
sampled from the three datasets evenly. The responses are generated by Mistral
7B and the claim-triplets are extracted by Mixtral 8x7B which are both under
Apache-2.0 License. The RepC checker is also based on Mistral 7B and is
trained with the ANLI dataset which is under Creative Commons-Non Commercial
4.0 License. The fine-tuned models will be released to the public under
Apache-2.0 License.

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.
  * Anthropic (2023) Anthropic. 2023.  Introducing claude.
  * Cao et al. (2022) Meng Cao, Yue Dong, and Jackie Cheung. 2022.  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3340–3354, Dublin, Ireland. Association for Computational Linguistics.
  * Chen and Mueller (2023) Jiuhai Chen and Jonas Mueller. 2023.  Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment.  _arXiv preprint arXiv:2308.16175_.
  * Chen et al. (2023) Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. 2023.  Felm: Benchmarking factuality evaluation of large language models.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Chern et al. (2023) I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, et al. 2023.  Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios.  _arXiv preprint arXiv:2307.13528_.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  * Dziri et al. (2022) Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M Ponti, and Siva Reddy. 2022.  FaithDial: A Faithful Benchmark for Information-Seeking Dialogue.  _Transactions of the Association for Computational Linguistics_ , 10:1473–1490.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Goldberger et al. (2004) Jacob Goldberger, Geoffrey E Hinton, Sam Roweis, and Russ R Salakhutdinov. 2004\.  Neighbourhood components analysis.  _Advances in neural information processing systems_ , 17.
  * Guerreiro et al. (2023a) Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023a.  Hallucinations in Large Multilingual Translation Models.  _Transactions of the Association for Computational Linguistics_ , 11:1500–1517.
  * Guerreiro et al. (2023b) Nuno M. Guerreiro, Elena Voita, and André Martins. 2023b.  Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1059–1075, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Honovich et al. (2021) Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2021.  $q^{2}$: Evaluating factual consistency in knowledge-grounded dialogues via question generation and question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7856–7870, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Li et al. (2023) Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023.  HaluEval: A large-scale hallucination evaluation benchmark for large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6449–6464, Singapore. Association for Computational Linguistics.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark Gales. 2023.  SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9004–9017, Singapore. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Min et al. (2023) Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023.  FActScore: Fine-grained atomic evaluation of factual precision in long form text generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 12076–12100, Singapore. Association for Computational Linguistics.
  * Mündler et al. (2023) Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2023.  Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016_ , volume 1773 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial nli: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901.
  * OpenAI et al. (2023) OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023.  Gpt-4 technical report.
  * OpenAI (2022) OpenAI. 2022.  Introducing chatgpt.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval augmentation reduces hallucination in conversation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wang et al. (2023) Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. 2023.  Factcheck-gpt: End-to-end fine-grained document-level fact-checking and correction of llm output.  _ArXiv_ , abs/2311.09000.
  * Yang et al. (2023) Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023.  A new benchmark and reverse validation method for passage-level hallucination detection.
  * Zha et al. (2023) Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023.  Alignscore: Evaluating factual consistency with a unified alignment function.  _arXiv preprint arXiv:2305.16739_.
  * Zhang et al. (2023a) Jiaxin Zhang, Zhuohang Li, Kamalika Das, Bradley Malin, and Sricharan Kumar. 2023a.  SAC3: Reliable hallucination detection in black-box language models via semantic-aware cross-check consistency.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15445–15458, Singapore. Association for Computational Linguistics.
  * Zhang et al. (2023b) Tianhang Zhang, Lin Qiu, Qipeng Guo, Cheng Deng, Yue Zhang, Zheng Zhang, Chenghu Zhou, Xinbing Wang, and Luoyi Fu. 2023b.  Enhancing uncertainty-based hallucination detection with stronger focus.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 915–932, Singapore. Association for Computational Linguistics.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.

##  Appendix for _RefChecker : Reference-based Fine-grained Hallucination
Checker and Benchmark for Large Language Models
_

##  Appendix A Details of the Benchmark Data Curation Process

Setting | Data Source | Task | Reference
---|---|---|---
Zero Context |  NaturalQuestions (development set) | Closed Book QA | Annotated Long Answer
Noisy Context |  MS MARCO (validation set) |  Retrieval-Augmented Generation (RAG) | Retrieved Passages
Accurate Context | databricks-dolly-15k |  Summarization Closed QA Information Extraction | Input Context
Table 6: A summary of the RefChecker hallucination detection benchmark. The
examples of the benchmark are curated from three different data sources and
cover diverse tasks.

The basic information of the benchmark dataset are summarized in Table 6. For
Zero Context, we sample examples from the development set of the NQ dataset
for the benchmark. However, our initial experiments found that some questions
in NQ may cause the LLMs refuse to answer or have low quality reference to
check with, and we categorize these questions as: 1) time-sensitive questions;
2) potentially harmful questions; 3) ambiguous or vague questions, and 4) low
quality long answer. We will talk about the data filtering later.

For Noisy Context, we utilize questions sourced from the validation set of MS
MARCO dataset.333 https://huggingface.co/datasets/ms_marco/viewer/v2.1Nguyen
et al. (2016) To prevent LLMs from declining to provide answers, we choose
examples where a golden passage containing the answer to the question has been
annotated.

For Accurate Context, we employ the databricks-
dolly-15k444https://huggingface.co/datasets/databricks/databricks-dolly-15k
instruction tuning dataset for the benchmark. Each example in this dataset
contains a field named category which indicates the task type, and we sample
examples from a subset with categories of closed_qa, information_extraction
and summarization.

During the response collection for benchmarking, we use fixed prompt templates
in each task setting for collecting responses from LLMs for fair comparisons.
For Zero Context setting, the prompt for response collection is the question
itself. For Noisy and Accurate Context settings, we use prompt templates shown
in Figure 7.

Figure 7: Prompt templates for response collection from LLMs on our benchmark.
For Zero Context setting, we just use the question itself as the prompt.

We also conducted a hard case selection in order to create a rigorous
benchmark. We talk about the details in the following part of this section.

###  A.1 Data Filtering for the NQ Dataset

Figure 8: Prompts for filtering out inappropriate examples from the
development set of NQ.

We employ ChatGPT (GPT-3.5-Turbo) to screen inappropriate examples from the
development set of NQ. The specific prompts utilized are illustrated in Figure
8.

Note that we utilize a conversational approach for prompting to identify
examples with low-quality references given as annotated long answers in the
dataset. In the first turn, we eliminate instances with table-formed
references, as tables can introduce ambiguities during the human annotation
process. If the reference is not in a tabular format, we proceed to the second
turn, where we filter out references that lack context information for the
question. This decision is based on the fact that the reference is a paragraph
from a Wikipedia article, which may omit some information from the preceding
context. Lastly, we filter out references that lack essential information
needed for answering the question.

###  A.2 Details of Hard Case Selection

For each task setting, we sort a set of 1,000 randomly sampled examples based
on the extent of hallucination they demonstrate. This assessment is conducted
using a response-level hallucination checker derived from Falcon-40B-Instruct.
Specifically, responses were gathered from four different LLMs ,
GPT-3.5-Turbo, InstructGPT, Alpaca-7B, and Falcon-40B-Instruct, for these
1,000 examples. Falcon-40B-Instruct is subsequently employed to evaluate
whether these responses contain hallucinations according to the prompt
template depicted in Figure 9. In this prompt, the “claim” refers to the
response generated by an LLM.

Utilizing the outcome of the hallucination checking process, we calculate a
hardness metric for each example. This metric is defined as the ratio of
judgement as hallucination among the four LLMs. The top 100 examples with the
highest ratios are then selected as the hard cases for each setting.

Figure 9: The prompt used in hard case selection during the benchmark curation
process.

###  A.3 Human Annotation

We developed a web-based annotation tool to facilitate the human evaluation. A
screenshot of the annotation tool is presented in Figure 10. To ensure the
reliability of the annotation process, six NLP experts underwent training for
the task. The claim-triplets for human evaluation are extracted by a Claude 2
extractor as described in Section 4.1.

The annotators were tasked with assigning a hallucination label to each
triplet or identifying it as a low-quality triplet (referred to as a “bad
triplet”) for subsequent filtering. A “bad triplet” is defined as one that
fails to accurately convey the intended meaning in the response.

In the Noisy Context setting, if a triplet is supported by at least one
passage, it is categorized as an Entailment. Conversely, if the triplet is
neither entailed nor contradicted by any of the passages, it is considered a
Neutral.

Figure 10: The screenshot of the annotation tool for human evaluation.

###  A.4 Observations from Human Evaluation

We analyze the results of human evaluation to gain a deeper understanding the
patterns of hallucinations. We establish our evaluation metric as follows.
Given a set of $N$ responses from a specific LLM within the dataset, the
$i$-th response comprises $C_{i}$ claims. Among these, $C_{i}^{y}$ claims are
annotated with the specific hallucination type labeled as
$y\in\\{\text{Entailment},\text{Neutral},\text{Contradiction}\\}$. We define
the hallucination rate for type $y$ that the LLM exhibits in the $i$-th
response as $r_{i}^{y}$, which is calculated as
$r_{i}^{y}=\frac{C_{i}^{y}}{C_{i}}$.

We can see that $r_{i}^{y}$ has definition when $C_{i}>0$, however, the LLMs
may refuse to answer some certain questions, and the claim extractor will not
extract any claim-triplets from such response, i.e., $C_{i}=0$. To cover these
cases in the metric, we define a new metric of Abstain Rate
$r^{\text{abstain}}$ as did in FActScore, and the rate of abstain is the ratio
of abstained responses, which is
$r^{\text{abstain}}=\frac{\sum_{i=1}^{N}\mathds{1}(C_{i}=0)}{N}$ where
$\mathds{1}(x)$ is an indicator function which is $1$ if $x$ holds and $0$
otherwise. Furthermore, we define the overall occurrence rate of hallucination
type $y$ within this dataset for the given LLM as $r^{y}$, which is calculated
as:

$r^{y}=\frac{\sum_{i=1}^{N}r_{i}^{y}\cdot\mathds{1}(C_{i}>0)}{\sum_{i=1}^{N}\mathds{1}(C_{i}>0)}$
(1)

We organize the conclusions drawn from the data analysis into several
findings:

#### Context Information is Critical

Figure 11: Results of different task settings by averaging the results of the
seven LLMs.

Figure 11 displays hallucination label distributions and abstain rates across
the three context settings, averaged from the seven LLMs. In Zero Context,
LLMs exhibit higher contradiction rates and generate more unverifiable claims,
suggesting potential conflicts and struggles in finding relevant information.
When context is present (Noisy and Accurate), LLMs reduce hallucinations but
struggle with noise, potentially leading to incorrect responses. In
conclusion, the reliability of LLMs’ internal knowledge is questionable,
highlighting the need for clean and precise contextual information for
generating factual responses.

#### GPT Family Steadily Improved Factuality

Figure 12: Detailed results of the human evaluation with the rates of
Contradiction, Neutral, Entailment and also Abstain across the three settings,
and ranked by the contradiction rates. The last ranking is the results
averaged over the three settings.

We present the detailed results and rankings of the seven evaluated LLMs in
Figure 12. We rank the LLMs based on the contradiction rates, where a lower
contradiction rate is considered better. Across all three settings within our
benchmark, a consistent enhancement in factuality is evident, progressing from
InstructGPT to ChatGPT and to GPT-4.

#### Open Source LLM is Catching Up

The analysis presented in Figure 12 also reveals noteworthy trends regarding
the performance of open-source LLMs. Notably, proprietary LLMs, particularly
GPT-4 and Claude 2, exhibit superior performance compared to open-source LLMs,
on average. It is worth highlighting, however, that the recently open-sourced
Llama 2 model demonstrates exceptional performance. Notably, in all evaluated
settings, Llama 2 70B Chat outperforms ChatGPT and even surpasses Claude 2 in
the Noisy Context setting. These findings suggest a potential for open-source
LLMs to narrow the performance gap with their proprietary counterparts.

#### Copy from Context is Safer

Figure 13: An example of factual claim-triplet whose content are mostly copied
from the context.

Replicating content in the context enhances the factuality, as illustrated in
Figure 13. In order to quantitatively assess the relationship between copying
and hallucination in both Noisy and Accurate Context settings, we introduce
the concept of Copy Rate. This metric is defined as the ratio of N-grams
covered by the context, where an N-gram refers to a phrase comprising N
consecutive words. Specifically, we compute the average copy rates for 1 to 4
grams of a claim-triplet to determine its overall copy rate. The findings
presented in Figure 14 reveal a discernible trend: a higher copy rate
corresponds to an increased likelihood of entailment.

Figure 14: Copy rate of the claim-triplet v.s. label distributions, by
aggregating the results of the Noisy Context and Accurate Context settings.

##  Appendix B Details of RefChecker

###  B.1 Extractor

The prompts used for few-shot claim extraction are shown in Figure 15. They
are used for claim extraction by GPT-4, Claude 2, Mixtral, and the Mistral
baseline. For Mistral-SFT, we removed the in-context examples in the prompt
because we find it doesn’t affect the extraction quality after supervised
fine-tuning but saves context length.

We collected 10,000 questions without claim extraction results and annotation,
following the same process as described in Appendix A. The collected questions
cover the three context settings evenly. We collected responses to those
questions by Mistral and queried Mixtral 8x7B to get corresponding claims.
After that, we performed supervised fine-tuning on a Mistral 7B model to
distill the output of the larger Mixtral model. We trained the model for 1
epoch with a initial learning rate 1e-5.

Figure 15: The prompt used for the GPT-4 and Claude 2 extractors. It requires
a question and response from the LLM, and is provided with two in-context
examples. Figure 16: Prompt for the GPT-4 Checker and Claude 2 Checker.

###  B.2 Checker

| Response | Sentence | Sub-sentence | Triplet
---|---|---|---|---
RoBERTa-NLI | 44.92 | 51.97 | 50.18 | 55.19
AlignScore | 46.05 | 53.19 | 50.71 | 57.60
GPT4 | 55.86 | 56.66 | 47.50 | 58.78
Claude2 | 46.49 | 46.67 | 34.57 | 41.00
RepC-LE-knn | 45.36 | 50.79 | 46.29 | 55.33
RepC-LE-svm | 48.91 | 54.26 | 52.29 | 59.81
RepC-LE-nn | 44.03 | 53.26 | 50.83 | 57.54

Table 7: Detailed performance of 7 checkers under different claim
granularities on 2.1k manual annotated responses. The checkers’ predictions
under different granularities are all merged into response-level and then
evaluated.

Models |  Average of three settings |  Zero-context (NQ) |  Noisy-context (MS MARCO) |  Accurate-context (databricks-dolly-15k)
---|---|---|---|---
Accuracy | Macro-F1 | Accuracy | Macro-F1 | Accuracy | Macro-F1 | Accuracy | Macro-F1
| Baseline Checkers
RoBERTa-NLI | 76.56 | 55.88 | 74.06 | 69.90 | 78.36 | 46.67 | 77.27 | 51.06
AlignScore | 78.85 | 59.45 | 73.40 | 70.28 | 78.86 | 50.42 | 84.30 | 57.66
GPT-4 | 74.77 | 59.80 | 67.46 | 66.10 | 76.67 | 55.49 | 80.17 | 57.80
Claude 2 | 51.98 | 36.55 | 43.42 | 42.90 | 40.35 | 25.89 | 72.18 | 40.87
| Mistral-based Checkers
zero-shot | 69.43 | 46.64 | 70.83 | 61.10 | 71.75 | 43.01 | 65.72 | 35.81
1-shot | 76.68 | 50.66 | 65.44 | 63.25 | 81.23 | 42.18 | 83.38 | 46.56
3-shot | 74.24 | 45.89 | 56.67 | 56.20 | 81.55 | 37.41 | 84.50 | 44.07
LoRA-sft-n2000 | 72.06 | 52.62 | 74.09 | 68.22 | 75.20 | 48.65 | 66.90 | 40.99
LoRA-sft-n4000 | 77.84 | 57.98 | 77.43 | 73.64 | 79.21 | 50.29 | 76.89 | 50.00
RepC-LS-knn-n100 | 74.36 | 51.98 | 72.67 | 68.58 | 77.54 | 45.19 | 72.86 | 42.17
RepC-LE-knn-n100-e100 | 69.72 | 51.64 | 70.26 | 66.05 | 71.14 | 46.33 | 67.75 | 42.55
RepC-LS-svm-n1000 | 79.15 | 59.36 | 78.34 | 74.04 | 79.82 | 47.62 | 79.29 | 56.43
RepC-LE-svm-n1000-e1000 | 79.03 | 60.05 | 77.98 | 73.53 | 79.56 | 51.29 | 79.54 | 55.34
RepC-LS-nn-n2000 | 80.17 | 57.31 | 75.50 | 71.95 | 81.78 | 46.90 | 83.22 | 53.07
RepC-LE-nn-n2000-e2000 | 81.27 | 60.80 | 75.23 | 71.98 | 82.08 | 47.56 | 86.50 | 62.86

Table 8: Checker evaluation results on 11k human annotated claim triplets. In
Mistral-based checkers, the model names start with the variant types, eg.
LoRA-sft indicates the LoRA fine-tuned variant and RepC-LE-nn indicates the
representation based classification variant using layer ensemble with 2-layer
MLP as the classifier. Here “nxxx” and “exxx” indicates the number of training
samples and ensemble learning samples.

The prompts used for the GPT-4 and Claude 2 based checkers are shown in Figure
16.

As a supplement of Figure 5, Table 7 shows the detailed checker performance
under different claim granularities. As a supplement of Table 5, Table 8 shows
the full results of checker evaluation.

In the analysis of Table 5, we claim that Claude 2 checker tends to flag a
neutral claim as contradiction. We can observe such tendency in Table 9,
especially for MS MARCO and Dolly datasets.

| NQ | MS MARCO | Dolly | Total
---|---|---|---|---
# triplets | 3319 | 3420 | 3994 | 10733
# neutral labels | 1818 | 436 | 368 | 2622
# pred as E | 313 | 88 | 165 | 566
# pred as N | 201 | 9 | 16 | 226
# pred as C | 1304 | 339 | 187 | 1830

Table 9: Detailed statistics of the Claude 2 checker’s neutral F1 score. We
show its prediction results of all neutral labels. Figure 17: Performance
tendency of different layers in RepC-LS checkers. The corresponding RepC-LE
checkers are included as dashed lines.

We also study the performance tendency of RepC-LS and RepC-LE in Figure 17.
The findings indicate that in RepC-LS, the best performed layer is typically
around the middle rather than the last layer. Despite RepC-LS trailing behind
RepC-LE, it maintains its advantages in model size and data efficiency.

Besides, in Figure 18, we evaluate the RepC checker performance with respect
to different number of training data. We can see that RepC-LS-svm outperforms
RepC-LS-nn with fewer training data.

Figure 18: Performance of different RepC checkers with respect to numbers of
training samples.

###  B.3 Source Attribution

In many cases, users of hallucination detection systems care not only the
verdicts of the checker, but also where the hallucination happens in the
original response, as well as which evidence in the reference supports such
verdicts. We provided a rudimentory support of such demand. Specifically, we
apply a sentence embedding model (SimCSE Gao et al. (2021)) to encode spans in
responses and references, compare them to the encoding of elements in claim-
triplets, then use a threshold to filter matched spans as source attribution
results. This naive approach suffers from issues on computational efficiency,
unclear boundaries, and matching by shallow semantics. The topic on source
attribution has a significant impact on applications of hallucination
detection and we leave the exploration on non-trivial solutions to the future.

##  Appendix C Comparisons with Other Hallucination Detection Frameworks

###  C.1 Comparison on the RefChecker Benchmark

We compare RefChecker with recently proposed hallucination detection
frameworks, SelfCheckGPT, FActScore and FacTool, on our benchmark. The four
frameworks use different representations of claims and hallucination labels as
described in Table 1, we aggregate the claim-level results into two types of
response-level results:

  * •

Response-level binary classification. We aggregate the claim-level labels into
response-level binary labels as Factual and Non-Factual. Thus, we use
Accuracy, Factual F1 (Fact. F1 for short) and Non-Factual F1 (Non-Fact. F1) as
the evaluation metrics. We use a strict configuration that a response is non-
factual if at least one claim contains hallucination. For SelfCheckGPT, we
consider minor_inaccurate and major_inaccurate labels as hallucination. For
RefChecker, we consider both Contradiction and Neutral as hallucination.

  * •

Correlations of response-level hallucination rate. Following SelfCheckGPT, we
also compare the hallucination rate of a response with human evaluation by
Pearson and Spearman correlations. For SelfCheckGPT, we compute the
hallucination rate of a response by averaging the scores of the sentences
following the definition in their paper. For FActScore and FacTool, the
hallucination rate is the ratio of non-factual claims in a response. And for
RefChecker, we take the ratio of Contradiction and Neutral claims as the
hallucination rate.

The results are shown in Table 10 for Zero Context setting, Table 11 for Noisy
Context setting and Table 12 for Accurate Context setting. Following their
configurations in their papers, we apply InstructGPT(text-davinci-003) and
GPT-4 as the extractors for FActScore and FacTool, respectively, apply
ChatGPT(gpt-3.5-turbo) as the checker for SelfCheckGPT and FActScore and GPT-4
for FacTool. The combinations of extractor and checker of RefChecker are
displayed as “{Extractor} + {Checker}”.

We conclude these results with the following observations:

  * •

RefChecker is effective. Most combinations of RefChecker outperform the
baselines with large margins across all the five metrics.

  * •

RefChecker is more effective with a GPT-4 checker. The best results are
achieved with a GPT-4 checker indicating that the main bottleneck lies in the
checking module. In spite of that, RefChecker can still outperforms the
baselines with a smaller checker AlignScore.

  * •

Purely open-sourced combinations can also outperform the baselines which are
using proprietary LLMs for both extractor and checker.

Zero Context Setting
---
| Accuracy | Fact. F1 | Non-Fact. F1 | Pearson | Spearman
SelfCheckGPT | 77.99 | 54.03 | 85.53 | 35.40 | 43.15
FActScore | 66.41 | 49.42 | 74.86 | 42.58 | 45.60
FacTool | 84.94 | 73.29 | 89.52 | 59.78 | 62.57
RefChecker |  |  |  |  |
GPT-4 + GPT-4 | 93.82 | 86.89 | 95.96 | 83.95 | 82.35
GPT-4 + Claude 2 | 91.31 | 82.76 | 94.19 | 80.11 | 79.53
GPT-4 + NLI | 83.98 | 71.28 | 88.89 | 60.81 | 62.32
GPT-4 + AlignScore | 90.54 | 78.97 | 93.90 | 71.95 | 70.37
GPT-4 + RepC | 89.96 | 81.16 | 93.16 | 77.42 | 77.26
Claude 2 + GPT-4 | 92.66 | 84.30 | 95.21 | 83.69 | 82.99
Claude 2 + Claude 2 | 92.08 | 83.40 | 94.8 | 80.04 | 79.39
Claude 2 + NLI | 83.20 | 70.51 | 88.26 | 59.25 | 60.39
Claude 2 + AlignScore | 90.54 | 78.97 | 93.90 | 75.07 | 73.83
Claude 2 + RepC | 89.58 | 80.71 | 92.86 | 76.34 | 76.24
Mistral-SFT + GPT-4 | 92.47 | 83.40 | 95.13 | 80.88 | 78.88
Mistral-SFT + Claude 2 | 90.93 | 80.66 | 94.07 | 77.98 | 77.04
Mistral-SFT + NLI | 89.96 | 78.86 | 93.42 | 72.89 | 72.07
Mistral-SFT + AlignScore | 90.54 | 78.79 | 93.91 | 75.81 | 74.16
Mistral-SFT + RepC | 89.38 | 80.43 | 92.72 | 77.14 | 76.74
Table 10: A comparison of RefChecker with previous works on our benchmark
under Zero Context setting. We highlight the best results using proprietary
LLMs with blue colors and best results results using pure open-source models
with orange colors. Noisy Context Setting
---
| Accuracy | Fact. F1 | Non-Fact. F1 | Pearson | Spearman
SelfCheckGPT | 58.55 | 51.63 | 63.74 | 36.31 | 32.15
FActScore | 63.57 | 69.94 | 53.77 | 33.36 | 29.91
FacTool | 68.40 | 72.84 | 62.22 | 46.35 | 38.69
RefChecker |  |  |  |  |
GPT-4 + GPT-4 | 74.54 | 76.42 | 72.32 | 64.56 | 57.30
GPT-4 + Claude 2 | 66.73 | 67.16 | 66.29 | 52.40 | 42.90
GPT-4 + NLI | 66.73 | 74.54 | 52.01 | 39.69 | 32.98
GPT-4 + AlignScore | 67.66 | 73.48 | 58.57 | 44.31 | 37.58
GPT-4 + RepC | 65.99 | 74.04 | 50.67 | 28.19 | 28.94
Claude 2 + GPT-4 | 71.38 | 73.72 | 68.57 | 53.14 | 47.89
Claude 2 + Claude 2 | 63.38 | 63.59 | 63.18 | 39.96 | 34.08
Claude 2 + NLI | 64.13 | 71.58 | 51.39 | 29.04 | 25.50
Claude 2 + AlignScore | 69.70 | 74.33 | 63.04 | 46.39 | 41.26
Claude 2 + RepC | 65.80 | 74.01 | 50.00 | 32.30 | 29.03
Mistral-SFT + GPT-4 | 75.28 | 77.57 | 72.46 | 67.29 | 59.94
Mistral-SFT + Claude 2 | 64.50 | 61.10 | 67.35 | 56.14 | 47.02
Mistral-SFT + NLI | 70.82 | 75.12 | 64.72 | 52.21 | 45.61
Mistral-SFT + AlignScore | 69.70 | 74.73 | 62.18 | 53.88 | 45.09
Mistral-SFT + RepC | 65.99 | 73.97 | 50.94 | 38.11 | 31.01
Table 11: A comparison of RefChecker with previous works on our benchmark
under Noisy Context setting. We highlight the best results using proprietary
LLMs with blue colors and best results results using pure open-source models
with orange colors. Accurate Context Setting
---
| Accuracy | Fact. F1 | Non-Fact. F1 | Pearson | Spearman
SelfCheckGPT | 62.15 | 68.70 | 52.12 | 40.23 | 32.55
FActScore | 69.37 | 78.57 | 46.30 | 27.80 | 27.05
FacTool | 72.53 | 80.98 | 50.63 | 31.41 | 32.82
RefChecker |  |  |  |  |
GPT-4 + GPT-4 | 80.81 | 86.85 | 64.50 | 58.61 | 55.50
GPT-4 + Claude 2 | 77.46 | 84.04 | 61.68 | 48.62 | 48.66
GPT-4 + NLI | 73.06 | 82.23 | 44.36 | 28.59 | 30.14
GPT-4 + AlignScore | 76.23 | 83.44 | 57.94 | 49.97 | 46.89
GPT-4 + RepC | 76.94 | 84.86 | 51.66 | 45.58 | 41.01
Claude 2 + GPT-4 | 82.22 | 87.64 | 68.34 | 60.99 | 58.96
Claude 2 + Claude 2 | 74.47 | 81.62 | 58.21 | 49.81 | 44.72
Claude 2 + NLI | 72.36 | 81.68 | 43.73 | 27.39 | 29.12
Claude 2 + AlignScore | 74.30 | 81.57 | 57.56 | 50.17 | 44.59
Claude 2 + RepC | 77.46 | 85.25 | 52.24 | 52.11 | 42.78
Mistral-SFT + GPT-4 | 79.75 | 85.96 | 63.72 | 56.09 | 53.72
Mistral-SFT + Claude 2 | 70.95 | 77.85 | 57.80 | 38.82 | 40.75
Mistral-SFT + NLI | 73.59 | 81.44 | 54.27 | 44.34 | 40.81
Mistral-SFT + AlignScore | 74.12 | 81.60 | 56.38 | 46.34 | 43.22
Mistral-SFT + RepC | 73.94 | 82.83 | 45.99 | 39.59 | 33.86
Table 12: A comparison of RefChecker with previous works on our benchmark
under Accurate Context setting. We highlight the best results using
proprietary LLMs with blue colors and best results results using pure open-
source models with orange colors.

###  C.2 Comparison on the SelfCheckGPT Dataset

| Pearson | Spearman
---|---|---
SelfCheckGPT | 78.32 | 78.30
RefChecker |  |
GPT-4 + GPT4 | 80.86 | 83.44
GPT-4 + Claude 2 | 87.67 | 89.23
GPT-4 + NLI | 79.96 | 80.16
GPT-4 + AlignScore | 76.20 | 77.33
GPT-4 + RepC | 79.63 | 79.23
Claude 2 + GPT4 | 79.18 | 82.89
Claude 2 + Claude 2 | 85.70 | 87.15
Claude 2 + NLI | 76.40 | 76.29
Claude 2 + AlignScore | 73.47 | 74.91
Claude 2 + RepC | 76.01 | 76.48
Mistral-SFT + GPT4 | 80.98 | 83.92
Mistral-SFT + Claude 2 | 85.46 | 86.49
Mistral-SFT + NLI | 78.54 | 79.66
Mistral-SFT + AlignScore | 75.10 | 76.08
Mistral-SFT + RepC | 76.59 | 76.70
Table 13: RefChecker results on the SelfCheckGPT dataset. The results of
SelfCheckGPT are from their paper. We highlight the best results using
proprietary LLMs with blue colors and best results results using pure open-
source models with orange colors.

We also ran RefChecker on the SelfCheckGPT dataset which contains 237 examples
on WikiBio domain. The results are shown in Table 13. We can observe that 11
out of the 15 combinations (73%) of RefChecker outperform SelfCheckGPT.

##  Appendix D Analysis of Internal Knowledge Bias

In this section, we further analyze the emergence of the hallucination from
the perspective of the LLMs’ bias to the internal knowledge. We analyze
whether the evaluated model and the checker generate response based on their
internal knowledge in Section D.1 and D.2, respectively. In general, we
observe that LLMs/Checkers may incorporate internal knowledge even when
provided with contextual information, contributing to the occurrence of
hallucination.

###  D.1 Internal Knowledge Bias of Evaluated Model

In order to analyze whether the evaluated LLMs generate responses based on
their own knowledge or the provided context in Noisy and Accurate Context
settings, we convert each claim-triplet extracted from the response into a
simple interrogative query for knowledge checking. For simplicity, we design a
prompt template and ask GPT-4-Turbo to generate these queries (Figure 19).
Then we feed the query into the evaluated LLM to check whether it has such
knowledge. The answer from the evaluated LLMs could be one of the following:

  1. 1.

Yes, means the evaluated LLM has this knowledge in its internal memory.

  2. 2.

No, means the evaluated LLM contains knowledge that is contradicted with the
triplet.

  3. 3.

Unsure, means the evaluated LLM does not have this knowledge or it has
confusion on the knowledge.

The label pairs (Yes, Contradiction) and (Yes, Neutral) indicate that the
model is utilizing internal information to generate this claim-triplet. On the
other hand, (No, Entailment) and (Unsure, Entailment) signify that the model
is relying on contextual information for generation. Pairs like (No,
Contradiction) suggest that the evaluated model may be less proficient in
processing context information, leading to the production of less reliable
claim-triplets.

The outcomes of the Accurate Context are illustrated in Figure 21. Upon
examination, it is evident that GPT-4-Turbo demonstrates the most notable
performance, primarily generating responses aligned with the reference
context. Conversely, GPT-3.5-Turbo tends to generate responses by relying on
its internal knowledge to some extent, leading to contradictions or neutrality
to the reference context. Claude 2 sometimes generate unsure information but
neutral to the reference context. In the case of InstructGPT, the model
further generates unsure information, which also contradicts the reference
context. This behavior may stem from contradictions within the model’s
internal knowledge or difficulties in comprehending the amalgamated content of
internal and reference information. Regarding LLaMA-2-70B, and
Falcon-40B-Instruct, our observations indicate that these models exhibit
inferior performance. They generate information that contradicts internal
knowledge and is irrelevant to the reference context. Alpaca 7B performs
similarly to GPT-3.5-Turbo, but seldom generates information contradicting to
its internal knowledge, Different from the accurate context setting, all the
models tend to generate more Neutral labels in the noisy context setting
(Figure 22).

Figure 19: Designed prompt for converting triplets to simple interrogative sentences. Figure 20: Designed prompt for masking triplet information in the reference context. Figure 21: The results of knowledge checking for evaluated models in the accurate-context setting. The labels Yes, No and Unsure are the responses to the interrogative sentences generated from knowledge triplets. Each value refers to the percentage of each checking pairs in the total number of triplets. Model | Checker | Entail | Contr | Neut
---|---|---|---|---
GPT-3.5 | GPT-4 | 37.36 | 6.28 | 56.36
Claude 2 | 61.49 | 25.95 | 12.56
RoBERTa-NLI | 21.82 | 21.76 | 62.64
GPT-4 | GPT-4 | 35.88 | 10.88 | 53.24
Claude 2 | 62.50 | 26.39 | 11.11
RoBERTa-NLI | 23.38 | 22.92 | 53.70
Table 14: Results for the information masking scenario in accurate-context
setting.

###  D.2 Internal Knowledge Bias of Checker

We also conduct an analysis to determine whether the checker provides
predictions based on its internal knowledge. In this analysis, a triplet
extracted from the response is taken, and we mask the subjective or objective
information in the context with ‘####’. The modified context, along with the
triplet, is then inputted into the checker to obtain the label. In theory, the
prediction label should be neutral because the relevant information in the
context is masked. If the label is not neutral, it implies that the model is
making inferences based on its internal knowledge. For the implementation of
this analysis, we query GPT-4-Turbo with a specifically designed prompt to
mask the triplet information, as illustrated in Figure 20. Specifically, in
the noisy-context setting, we implement the query for each reference document
and keep the document unchanged if there is no relevant information to the
extracted triplet.

The results of the accurate context setting are shown in Table 14. As we
observe, RoBERTa-NLI achieves the most significant Neutral labels, 62.64% and
53.70% for evaluated model GPT-3.5-Turbo and GPT-4-Turbo, respectively. The
checker GPT-4-Turbo achieves the second performance. But Claude 2 predicts a
large number of Entailment and Contradiction, which implies that the checker
Claude 2 highly relies on the internal knowledge for checking. The results of
the zero context setting are in a similar pattern with those of accurate-
context setting (Table 15). But in the noisy context setting (Table 16),
RoBERTa-NLI outperforms GPT-4-Turbo and Claude 2 with a large margin in the
ratio of Neutral labels. The results may results from the strong bias to
internal knowledge of GPT-4-Turbo and Claude 2 when the context is extremely
long, or the RoBERTa-NLI model has less associative ability to the memorized
knowledge.

Model | Checker | Entail | Contr | Neut
---|---|---|---|---
GPT-3.5 | GPT-4 | 37.91 | 22.55 | 39.54
Claude 2 | 56.54 | 33.99 | 9.48
RoBERTa-NLI | 35.62 | 30.72 | 33.66
GPT-4 | GPT-4 | 43.33 | 13.67 | 43.00
Claude 2 | 61.00 | 28.00 | 11.00
RoBERTa-NLI | 34.67 | 23.00 | 42.33
Table 15: Results for the information masking scenario in zero-context setting.  Model | Checker | Entail | Contr | Neut
---|---|---|---|---
GPT-3.5 | GPT-4 | 58.52 | 6.67 | 34.82
Claude 2 | 60.25 | 20.00 | 19.75
RoBERTa-NLI | 9.38 | 10.62 | 80.00
GPT-4 | GPT-4 | 65.71 | 6.29 | 28.00
Claude 2 | 58.57 | 32.29 | 9.14
RoBERTa-NLI | 8.57 | 11.14 | 80.28
Table 16: Results for the information masking scenario in noisy-context
setting.  Figure 22: The results of knowledge checking for evaluated models in
the noisy-context setting. The label Yes, No and Unsure are the respones to
the interrogative sentences generated from knowledge triplets. Each value
refers to the percentage of each checking pairs in the total number of
triplets.