printacmref=false Robotics Research Lab, RPTU Kaiserslautern Germany Robotics
Research Lab, RPTU Kaiserslautern Germany Robotics Research Lab, RPTU
Kaiserslautern Germany Robotics Research Lab, RPTU Kaiserslautern Germany

# Evaluating the Robustness of Off-Road Autonomous Driving Segmentation
against Adversarial Attacks: A Dataset-Centric analysis

Pankaj Deoli<EMAIL_ADDRESS>,  Rohit Kumar<EMAIL_ADDRESS>,
Axel Vierling<EMAIL_ADDRESS>and  Karsten Berns<EMAIL_ADDRESS>
###### Abstract.

This study investigates the vulnerability of semantic segmentation models to
adversarial input perturbations, in the domain of off-road autonomous driving.
Despite good performance in generic conditions, the state-of-the-art
classifiers are often susceptible to (even) small perturbations, ultimately
resulting in inaccurate predictions with high confidence. Prior research has
directed their focus on making models more robust by modifying the
architecture and training with noisy input images, but has not explored the
influence of datasets in adversarial attacks. Our study aims to address this
gap by examining the impact of non-robust features in off-road datasets and
comparing the effects of adversarial attacks on different segmentation network
architectures. To enable this, a robust dataset is created consisting of only
robust features and training the networks on this robustified dataset. We
present both qualitative and quantitative analysis of our findings, which have
important implications on improving the robustness of machine learning models
in off-road autonomous driving applications. Additionally, this work
contributes to the safe navigation of autonomous robot Unimog U5023 in rough
off-road unstructured environments by evaluating the robustness of
segmentation outputs. The code is publicly available at
https://github.com/rohtkumar/adversarial_attacks_on_segmentation

###### Key words and phrases:

Adversarial examples, Off-road autonomous driving, Semantic segmentation,
Robotics

##  1\. Introduction

Technology has come a long way since the development of the first automobile
by Karl Benz and Gottlieb Daimler. With the advent of high-stakes
applications, such as autonomous driving, which heavily relies on Deep Neural
Networks (DNNs) for perception, people are increasingly fascinated by the
capabilities of machines. Today, DNNs form the backbone of all computer vision
solutions, thereby providing the industry with human-like capabilities in the
field of computer vision. However, safety concerns have arisen in relation to
autonomous driving with respect to DNNs, particularly due to the vulnerability
of such systems to adversarial attacks. Adversarial attacks refer to
intentionally created inputs that trick well-trained DNNs, especially when the
adversary needs to make only minor changes to an input that has already been
successfully processed. On a broad level, these attacks can be categorized
along 3 primary axes i.e. their effect or influence on the classifier, any
safety violations and individual specificity. These can be further sub-
categorized as White box or Black box attacks, depending on whether the
attacker has access to the model’s parameters. Moreover, these attacks
typically involve carefully constructed adversarial examples at the level of
pixels, which manipulate the physical environment and thus result in physical
consequences.
Adversarial examples represent a major area of research interest due to both
safety concerns and the need for explainability. Significant research and
developmental efforts have been carried out in the domain of Adversarial
examples for general classification tasks but not so much for off-road
autonomous driving. Further, these improvements have been oriented towards
urban contexts since addressing man-made structural entities is relatively
simple. On the other hand, great effort has been put towards segmenting paths,
trees, and plants in off-road areas, but little towards enhancing the
robustness of autonomous driving systems to adversarial examples in such
environments.
This work addresses this gap by evaluating the effects of these adversarial
attacks for semantic segmentation in the context of off-road autonomous
driving. The main contributions of the paper are summarized as follows:

  * •

An off-road robust dataset has been created (by adapting Ilyas et al. (2019))
consisting of only robust features.

  * •

Analysis of this robustified dataset on two (established) State-of-the-art
(SOTA) semantic segmentation networks i.e. U-Net and Link-Net.

  * •

Networks comparison and analysis on standard dataset vs Robustified dataset

The workflow is as follows: section II provides related works. The goal of
section III is to present the methodology, the datasets used followed by
architectural, implementation and training details. Section IV provides the
experimentation details along-with quantitative and qualitative comparison
whereas the evaluation of this work is done in section V. The strategic
transfer of this work to the Unimog U5023 is mentioned in section VI, whereas,
the work is concluded with future scope in section VII.

##  2\. Related Works

The work done in the field of adversarial examples suggests a variety of
explanations ranging from theoretical models by Schmidt et al. (2018), Bubeck
et al. (2018) to arguments based on the concentration of measure in high
dimensions Gilmer et al. (2018), Mahloujifar et al. (2018). An extensive
adversarial robustness evaluation of real-time semantic segmentation models is
presented by Rossolini et al. (2022) wherein, different attacks are evaluated.
On the contrary, certified defences against adversarial patch attacks was
presented by Yatsura et al. (2023) with the introduction of demasked smoothing
which can be applied to any semantic segmentation model. Using multi-sensor
setup, the authors of Lee and Ro (2021) proposed an adversarially robust
fusion method by developing a random feature fusion strategy for preserving
multi-sensor fusion features. However, the aforementioned researches
frequently falls short of accurately capturing the behaviours we observe in
practise. The works done focus on carefully crafting adversarial examples and
then adding it to the image but do not take into consideration the non-robust
features (inherently present in an image) which might affect the overall
performance. According to an earlier research on the subject, adversarial
examples are typically seen as aberrations caused by the large dimensions of
input space or statistical irregularities in the training data Mahloujifar et
al. (2018), Szegedy et al. (2014), Goodfellow et al. (2015). In the context of
Autonomous Driving (AD), many ideas for understanding adversarial examples
have been put forth, ranging from high-dimensional statistical phenomenon to
finite-sample overfitting Gilmer et al. (2018), Fawzi et al. (2018) and Tanay
and Griffin (2016). Hence, the robustness of the adversarial approach can be
seen as a target that can be separated from maximizing the accuracy to be
pursued either through enhanced regularization techniques or pre/post
processing of network inputs/outputs Uesato et al. (2018), Carlini and Wagner
(2017), He et al. (2017).

##  3\. Approach and implementation

The following section gives an overview on the methodology and approach taken.
It begins with an introduction of the types of adversarial attacks, followed
by dataset exploration, baseline model selection criteria, and network
training.

###  3.1. Gradient-based adversarial attacks

Gradient-based adversarial attacks exploit the use of gradients in ML models
to generate examples. These gradients have been utilized in different manners
for obtaining the perturbation vector, such that the new adversarial example
has a greater tendency towards being misclassified. Fast Gradient Sign Method
(FGSM) Huang et al. (2017) is one of the simplest and most effective
adversarial attacks wherein the gradients of the loss functions are calculated
(w.r.t the input data) and then perturbating the input data in the direction
of the sign of the gradient.

$W^{T}x^{adv}+b=W^{T}x+W^{T}\delta+b$ (1)

where, $x$ denotes normal examples without adding the perturbations, $\delta$
is referred to as perturbation in adversarial examples and indicates the
adversarial perturbations added to normal examples. The adversarial examples
are fed into the ML model and $x$ multiplies the parameter matrix $W^{T}$
given as equation 1.
Basic Iterative Method (BIM) Kurakin et al. (2017), an iterative version of
FGSM, involves applying the FGSM multiple times with a small perturbation in
each iteration (which can increase the likelihood of success). A stronger
version of BIM is Projected Gradient Descent (PGD) Kurakin et al. (2017) which
includes a projection step to ensure that the perturbed input data remains
within a certain range of values thereby making the attack more effective
against well-trained models. The basic PGD algorithm simply iterates the
updates:

$Repeat:\delta:=\mathcal{P}\left(\delta+\alpha\nabla_{\delta}\mathcal{L}\left(h_{\theta}(x+\delta),y\right)\right)$
(2)

where $\mathcal{P}$ denotes the projection onto the ball of interest,
$h_{\theta}$ is a neural network, $\delta$ denotes the lower bound or delta,
$\theta$ denotes the model parameters and $\alpha$, the step size.
Furthermore, the following loss function is minimized during training, where
$\Delta$ is a set of perturbations that we want our model to be invariant to,
such as the $L^{2}$ and $L^{\infty}$ perturbations.

$\min_{\theta}\max_{\delta\in\Delta}\mathcal{L}(x+\delta,y;\theta)$ (3)

And therefore due to this efficiency, we chose PGD attack (with $L^{2}$ and
$L^{\infty}$ bounds) for our task at hand.

###  3.2. Baseline model selection

Before deciding on the appropriate model, various network architectures were
studied. These included FCN16 Long et al. (2015), UNet Ronneberger et al.
(2015), DeepLab Chen et al. (2017), PNPNet Liang et al. (2020) and LinkNet
Chaurasia and Culurciello (2017). Comparing all, UNet and PNPNet employs novel
loss weighting algorithm that places a higher weight near the edges of
segmented objects. This weighting scheme helps the model to learn pixel-to-
pixel mapping and to segment cells in the image in a discontinuous fashion and
makes them identifiable. U-Net performs image localization by predicting the
image pixel by pixel and works with fewer training images to produce precise
segmentation results. As the network works with a smaller dataset and still
produces comparable results, the network was chosen for this study. Unlike
many DNNs used for segmentation, Link-Net links each encoder with a decoder
such that the input of each encoder layer is also bypassed to the output of
its corresponding decoder so as to recover lost spatial information that can
be used by the decoder and its up-sampling operations. Since decoder is
sharing knowledge learned by encoder at every layer, the decoder can use fewer
parameters. Based on the limitations of our small dataset and the observed
performance of various networks with small datasets, U-Net and LinkNet were
selected as the baseline models.

###  3.3. Datasets

A good dataset provides the foundation for the model to learn and make
accurate predictions and thereby should contain a wide variety of data
(representative) that reflects an entire range of variability in the problem
domain. When compared to widely used urban datasets for autonomous driving
such as KITTI (12919 images) Geiger et al. (2013), Cityscapes Cordts et al.
(2016) (2500 images), Waymo Sun et al. (2020) (1000 images), off-road datasets
lack distinctiveness and quantity. Prominent challenges when working with off-
road environments include occlusion, changing light conditions, and objects
with in-definitive shapes and sizes. Freiburg forest Valada et al. (2017) (366
pixel-wise annotated images) dataset is widely used for semantic scene
understanding of off-road environments with 6 classes. The Yamaha CMU Off-road
dataset (YCOR) Maturana et al. (2018), on the other hand, contains (1076
images with 7 classes) collected in 4 different seasons.

Table 1. Combined Off-road dataset classes Class | Label
---|---
Background | 0
Vegetation | 1
Traversable grass | 2
Smooth trail | 3
Obstacle | 4
Sky | 5
Rough trail | 6
Puddle | 7
| Non Traversable
---
Vegetation
8
Tree | 9

To increase diversity, sample size, and model transferability, datasets can be
carefully combined to improve the ML model’s generality. But this should be
done carefully so as to negate the effects of introducing biases and
inconsistencies. For our study, the above-mentioned two off-road datasets were
carefully combined (a total of 1442 images) with the final 10 classes as seen
from table 1. For model training, the train/val/test distribution was selected
as 794/360/288 images respectively.

###  3.4. Network training

After deciding the networks, adversarial attacks and datasets, the models were
trained with resnet50 backbone. For this, 4 Nvidia GeForce RTX 2080 Ti GPU
(total capacity of 44 GB) were used with an Intel(R) Xeon (R) Gold 6126 CPU
@2.60 GHz. For the task of adversarial attacks, the segmentation model module
in Tensorflow is used in order to understand the performance of the training
both quantitatively and qualitatively. The experiments however, were divided
in 4 stages i.e.

  * •

Standard training on the merged dataset \- This involves training the models
on the merged dataset using standard training configurations, without any
adversarial attacks. This allows us to assess the models performance on an
unbiased environment.

  * •

Adversarial training (PGD ($L^{2}$ and $L^{\infty}$)) \- wherein the networks
are trained with adversarial attacks with the adversarial training
configurations.

  * •

Robustifying the merged dataset \- i.e. dividing the dataset according to
robust and non-robust features and training the models only on the robust
features. The goal here is to remove any noise or irrelevant features that may
affect the models performance.

  * •

Standard training on Robustified dataset \- wherein the standard training is
carried out on the robustified dataset with the goal to assess the models
performance when trained on a dataset that has been modified to remove non-
robust features.

The configurations for training, testing, adversarial attacks can be seen from
table 2.

Table 2. Hyperparameter setup for different experiments | Unet/Linknet
---|---
Training Config. |  |  Optimizer \- Adam
---
Learning Rate \- 1$\times$$e^{-4}$
Loss \- CategoricalCrossEntropy
Metrics \- IoU, Accuracy
Augmentation \- Random flip,
Random hue, Contrast, Brightness
Epochs \- 100
Test Config. |  |  Loss \- CategoricalCrossEntropy
---
Metrics \- IoU, Accuracy
Adversarial Training Config. |  |  Optimizer \- Adam
---
Learning Rate \- 1$\times$$e^{-4}$
Loss \- CategoricalCrossEntropy
Metrics \- IoU, Accuracy
Attack \- PGD $L^{2}$, PGD $L^{\infty}$
epsilon \- 10
alpha \- 0.1
Robustifier/Non-Robustifier |  |  Attack \- PGD $L^{2}$, PGD $L^{\infty}$,
---
alpha \- 0.1

##  4\. Experimentation

###  4.1. Standard training on the merged dataset

Before starting the subsequent studies, it is important to perform
experimentation for baseline results comparison. Having no comparison measure
(with merged dataset), this helps us to derive the baseline with which we can
compare the later results. The models (both UNet and LinkNet) are trained for
100 epochs with the configurations mentioned in table 2. Table 3 shows the
quantitative results, wherein, UNet achieves an IoU of 73% during training and
69% during validation. LinkNet achieves an IoU of 71% and 69% during training
and validation respectively. For baselining of the results, these trained
models (UNet and LinkNet) are then evaluated against PGD $L^{2}$ and PGD
$L^{\infty}$ attacks on the test dataset. The results are presented in table 3
wherein it can be observed that the performance of both the networks went
considerably down. The networks fails to learn the non adversarial features
and therefore predicts the wrong output.
Having baseline results for comparison we now train the networks with the
aforementioned attacks, the details of which are mentioned in the next
section.

Table 3. Model is trained in standard setup for 100 epochs. The loss represents the average CategoricalCross entropy loss after 100 epochs and IoU represents the average after 100 epochs. | UNet | LinkNet
---|---|---
Train IoU | 0.73 | 0.71
Train Loss | 0.16 | 0.17
Val IoU | 0.69 | 0.69
Val Loss | 0.32 | 0.37
Test IoU (PGD $L^{2}$) | 0.48 | 0.60
Test IoU (PGD $L^{\infty}$) | 0.39 | 0.58
Test Loss (PGD $L^{2}$) | 0.59 | 0.76
Test Loss (PGD $L^{\infty}$) | 9.73 | 5.98
Table 4. Networks performance after different attacking strategies. Both the networks were trained on PGD $L^{2}$ and PGD $L^{\infty}$ attacking strategies. Input image | Ground truth | UNet (PGD $L^{2}$) | UNet (PGD $L^{\infty}$) | LinkNet (PGD $L^{2}$) | LinkNet (PGD $L^{\infty}$)
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |

###  4.2. Adversarial training

The goal of adversarial training is to obtain a classifier which will be
robust to adversarial attacks and will serve as a basis for robustification
procedure (see section 4.3). The adversarial training method (as suggested in
Ilyas et al. (2019)) is adapted to produce robust classifiers. Specifically,
training against a PGD opponent is limited in the 2-norm starting from the
original image. The gradient is calculated at each PGD step to make sure we
move a specific amount in the 2-norm per step. At the end of this experiment,
4 robust models are obtained i.e. (2 models for PGD $L^{2}$ and 2 models for
PGD $L^{\infty}$). The quantitative evaluation of this training is further
shown in table 5.

Table 5. Performance after 100 epochs of adversarial training with the attacking strategy. | UNet | LinkNet
---|---|---
Train IoU (PGD $L^{2}$) | 0.68 | 0.68
Train IoU (PGD $L^{\infty}$) | 0.50 | 0.45
Train Loss (PGD $L^{2}$) | 0.33 | 0.37
Train Loss (PGD $L^{\infty}$) | 1.30 | 1.32
Val IoU (PGD $L^{2}$) | 0.68 | 0.68
Val IoU (PGD $L^{\infty}$) | 0.52 | 0.46
Val Loss (PGD $L^{2}$) | 0.28 | 0.32
Val Loss (PGD $L^{\infty}$) | 1.36 | 1.35

####  4.2.1. UNet trained with PGD $L^{2}$ and PGD $L^{\infty}$ attacks

The experiment details are as follows. UNet, with ResNet50 as backbone is
trained for 100 epochs (with PGD $L^{2}$ and PGD $L^{\infty}$ attacks)
individually.
When UNet (trained on PGD $L^{2}$) is attacked independently with PGD $L^{2}$
and PGD $L^{\infty}$, the results are presented in table 6. When compared with
the standard training (test dataset) evaluation (table 3), an improvement is
seen. The PGD $L^{2}$ loss for standard training was 0.59 whereas, it
decreased to 0.37 during adversarial training. However, it is still evident
that the model is susceptible to other attacks (e.g. performs badly with PGD
$L^{\infty}$ attack). When compared with the results from standard training,
it can be seen that, even though, quantitatively the model performs better,
qualitatively, the model still lacks precision (see table 4). The network
performs comparatively poor when separating different surface types.

Table 6. Adversarially trained Unet and LinkNet models on two attack
strategies are evaluated against both of these attack strategies.

|  | Test
---
IOU (PGD $L^{2}$)
| Test
---
IOU (PGD $L^{\infty}$)
| Test
---
Loss (PGD $L^{2}$)
| Test
---
Loss (PGD $L^{\infty}$)
Unet trained on PGD $L^{2}$ | 0.71 | 0.60 | 0.37 | 6.71
UNet trained on PGD $L^{\infty}$ | 0.52 | 0.56 | 1.27 | 1.35
Linknet trained on PGD $L^{2}$ | 0.61 | 0.70 | 0.41 | 5.18
Linknet trained on PGD $L^{\infty}$ | 0.55 | 0.58 | 1.28 | 1.31

When UNet (trained on PGD $L^{\infty}$), is attacked similarly, the
quantitative results are presented in table 6 and the qualitative results can
be seen from table 4. It is visible that the loss (Train loss ($L^{\infty}$))
has increased whereas, the IoU has dropped significantly. On the contrary, it
can be seen that the model trained on PGD $L^{\infty}$ also performs better
with PGD $L^{2}$. This is due to the fact that, PGD $L^{2}$ is a constrained
form of PGD $L^{\infty}$ attacks and thus, the model shows more capability to
learn constrained attacks automatically, if trained on broader attacks.

####  4.2.2. LinkNet with PGD $L^{2}$ and PGD $L^{\infty}$ attacks

Similarly, the procedure is carried out with LinkNet and the results are
presented in table 6. It is visible that the model performed better than the
standard training (see table 3).

When LinkNet (trained on PGD $L^{2}$) is attacked with both PGD $L^{2}$ and
PGD $L^{\infty}$, the results are presented in table 6. It can be seen that
the test loss has reduced significantly and the IoU has increased. At the same
time, it also shows that the test loss for PGD $L^{\infty}$ is still
significantly high, thereby confirming that the model does not learn the
attacks if trained on one specific attack.
Similarly, LinkNet (trained on PGD $L^{\infty}$) is attacked during test time
and the results are presented in table 6. As visible, the test loss has
decreased for both PGD $L^{2}$ and PGD $L^{\infty}$ thereby showing that
training on broader attack families can help in acquiring immunity against
smaller attacks of the same family.

###  4.3. Dataset robustification

With the adversarially trained network as the classifier, in this experiment,
the training dataset is robustified to construct a dataset that contains
features relevant only to a given (standard/robust) model.
According to Ilyas et al. (2019), any given training example can be decomposed
into robust and non-robust features. These subsets of features correspond to
those that remain useful after adversarial perturbation and those that do not.
The robust features are the ones that are truly important for determining the
class of an object (e.g. eyes and ears of an animal). The non-robust features
(according to humans) are more like background noise that can be easily
changed without affecting the object’s class. Generating a robust dataset
begins by first creating a robust model through adversarial training as
described in 4.2. The robustified model includes all but the output layer of
the adversarially trained model. The representation for a data point is the
output of this penultimate layer. For each data point in the original dataset,
a random starting input is iteratively updated to create a representation that
is as similar to the original input as possible. The perturbation process aims
to minimize the $L^{2}$ distance between the label and the sample’s
representation. Because the model itself is robust, the adjustments to the
random input only reflect the robust features of the original data point.

Algorithm 1 Pseudo-code for creating robust dataset (D). Ilyas et al. (2019)

1:$C_{R}\leftarrow\text{AdversarialTraining}(D)$

2:$g_{R}\leftarrow\text{mapping learned by $C_{R}$ from input to the
representation layer}$

3:$D_{R}\leftarrow\\{\\}$

4:for $(x,y)\in D$ do

5: $x^{\prime}\sim D$

6: $x_{R}\leftarrow\text{arg min}_{z\in[0,1]^{d}}\lVert
g_{R}(z)-g_{R}(x)\rVert_{2}$

7: $D_{R}\leftarrow D_{R}\cup\\{(x_{R},y)\\}$

8:end for

9:Return $D_{R}$

Once this robustified model has been created, the next step is to make a
dataset containing only the features relevant to the specific model. To
achieve this, the input $x_{r}$ is initialized as a different randomly chosen
sample from the training set. Normalized gradient descent is then performed on
$x_{r}$ while fixing the $L^{2}$-norm of the gradient at each step to a
constant value. This helps to ensure that the perturbations made to $x_{r}$
are not too large and do not cause significant changes to the image. At each
step of the process, the input $x_{r}$ is clipped so that it falls within the
valid image range of [0, 1] thereby maintaining the integrity of the images
and ensuring that it remains recognizable as an object within the given
dataset.
To perform this step, several iterations (100) based on batch size are
performed. For this, the configurations mentioned in table 3 are taken. In
each iteration, a batch of training data is passed through the robust model to
obtain the goal representation. The goal representation is then attacked using
the PGD attack, resulting in a delta on the input image batch. The delta is
then concatenated with the original image and saved to create a robust dataset
that includes both the training images and their corresponding labels. This is
saved and evaluated on the standard models against the PGD attacks.

At the end of the experiment, we obtain robustified dataset against PGD
$L^{2}$ and PGD $L^{\infty}$ attack. With 4 adversially trained models , 4
robustified datasets are obtained i.e.

  * •

Adversarially trained UNet + PGD $L^{2}$ robustified dataset

  * •

Adversarially trained UNet + PGD $L^{\infty}$ robustified dataset

  * •

Adversarially trained LinkNet + PGD $L^{2}$ robustified dataset

  * •

Adversarially trained LinkNet + PGD $L^{\infty}$ robustified dataset

Table 7. Networks performance on the robustified dataset. R represents robustified dataset. Networks struggle to capture the details of an ambiguous environment. Input image | Ground truth | UNet (RPGD $L^{2}$) | UNet (RPGD $L^{\infty}$) | LinkNet (RPGD $L^{2}$) | LinkNet (RPGD $L^{\infty}$)
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Table 8. 16 Feature activations of the second last convolution layer depicting the network’s attention during standard training, adversarial training and standard training on robustified dataset. Robustified dataset being depicted by R. UNet-standard training | UNet-PGD $L^{2}$ | UNet-PGD $L^{\infty}$ | UNet-RPGD $L^{2}$ | UNet-RPGD $L^{\infty}$
---|---|---|---|---
|  |  |  |

###  4.4. Standard training on Robustified dataset

In the final experiment, the robustified dataset obtained from the
adversarially trained classifiers is taken for standard training with a total
of 4 networks (obtained from adversarial training). No attack (of any kind) is
employed during training as well as test time. The models are trained for 100
epochs on the robust dataset and validated on the original validation dataset.

####  4.4.1. UNet

Both the UNet robustified datasets are selected for standard training. As seen
from table 9, the results are comparable to the standard training results
(from experiment 4.1) thereby depicting that the model learns the features
from the robustified training set. However, the losses and IoU values for the
original validation dataset are not efficient (for both the robustified
datasets). Qualitatively (table 7 (column c and d), the model fails to
converge and predict proper segmentation, thereby depicting that even though,
the quantitative results are promising, the actual qualitative results are
misleading.

Table 9. Standard training of robustified UNet and LinkNet model. The first &
third row shows results on the robustified dataset with PGD $L^{2}$ and the
second & fourth row shows results on the robustified dataset with PGD
$L^{\infty}$ at train time.

|  | Train IOU
---
| Val IOU
---
| Train Loss
---
| Val Loss
---
UNet (PGD $L^{2}$) | 0.68 | 0.58 | 0.22 | 2.44
UNet (PGD $L^{\infty}$ | 0.72 | 0.57 | 0.26 | 2.40
LinkNet (PGD $L^{2}$) | 0.70 | 0.60 | 0.26 | 2.02
LinkNet (PGD $L^{\infty}$) | 0.71 | 0.58 | 0.29 | 2.06

####  4.4.2. LinkNet

Similar procedure for LinkNet is performed, the results of which can be seen
from table 7 (column e and f). The model shows similar performance as UNet
wherein, the qualitative and quantitative results are misleading.

##  5\. Findings

The work explored whether the dataset has any effect in countering the
adversarial effect on semantic segmentation tasks and whether it can be
changed (based on attacks) and still maintain the same performance w.r.t
state-of-the-art metrics. One conclusion, which is evident, is the direct
unreliable relation between qualitative and quantitative performance. The
study done by Ilyas et al. (2019) depicts that the robustified model performs
better, however, the qualitative results were not talked about since it
focused on classification rather than segmentation. From off-road autonomous
driving point-of-view, some attributes towards the performance of our chosen
models could be:

  * •

Presence of Multi-class in input images \- The previous study by Ilyas et al.
(2019), in this regard, concentrated on binary classification. However, in our
study, the presence of ambiguous (non-rich features) multi-classes of the
input space can present fundamental barriers to classifier’s robustness. A
classifier cannot be resistant against tiny perturbations since, at the
highest level, for certain data distributions, any decision boundary will be
near a significant portion of inputs. Further details of these can be
visualized by (see table 8) where is clearly visible that due to the presence
of many ambiguous classes, the network fails to be persistent.

  * •

Insufficient data \- Another major problem was the unavailability of a huge,
distinct dataset with higher variability. Authors of Schmidt et al. (2018)
present that for appropriate learning, a good robust classifier requires
(O$\sqrt{d}$) samples ($d$ being the dimentionality of the data). In this
paradigm, adversarial examples appear as a result of insufficient knowledge of
the real data distribution. In particular, since training models robustly
reduce the effective amount of information in the training data (as non-robust
features are discarded), more samples should be required to generalize
robustly.

##  6\. What does it mean for the Unimog U5023?

To accurately analyse the developed solutions, Unimog U5023 has been used as a
demonstrator vehicle. The vehicle has the capability of accommodating adverse
off-road conditions and can therefore be also used as a method of
methodological transfer (to other smaller robots). The vehicle has additional
D.O.F (Degrees of freedom), a controllable gearbox, differential locks, or
adaptable tire pressure. For encompassing sudden terrains, the frame has the
capability to bend diagonally up to 60 cm while the axle can be articulated up
to 30 degrees.

Previous works done Wolf et al. (2023), Wolf (2022), Wolf and Berns (2021)
towards safety evaluation have focused on robust environment perception for
safe navigation in off-road unstructured environments. In Wolf et al. (2023),
a methodology was developed for traction optimization for Unimog based on OSM
maps and semantic segmentation (shown in figure 1). With the help of a neural
network, the slip coefficient of the segmented pathway is calculated which is
compared with the OSM tag for the corresponding surface. With the help of
precise segmentation, the Unimog can be traversed through an unstructured
environment with the correct traction control ultimately providing an extra
layer of safety.

Figure 1. The results for surface type detection in Wolf et al. (2023). Image
represents input image, segmentation result, Region of Interest and the
dominant class taken.

The Unimog is equipped to perform various tasks in an off-road environment
such as forest path making with a shredder, monotonous tasks in a landfill and
performing rescue operations after natural calamities. The most important
factor when performing these operations is the appropriate understanding of
the surface type, since a small slip factor (traction) change might result in
a sudden speed shift. The pre-existing surface type detection module [24] has
the capabilities of separating surfaces according to its surface properties.
However, to make this more robust, a subset of this research will be used to
focus on neural network training only on robust features to make better
surface type detection. Since off-road environment is ambiguous in nature
(i.e. poor distinction between foreground and background) training on less no.
of classes (e.g. 2 or 3 types of surfaces) will force the network to
understand the salient features within the surface paradigm thereby providing
improved robustness during autonomous navigation tasks and subsequently
improving the safety of the Unimog.

##  7\. conclusions

In conclusion, the work explored the role of datasets for achieving the
segmentation robustness in off-road environments. Two SOTA neural networks
along-with a combined dataset was utilised to see the effects when these are
presented by adversarial attacks. Even though, the work presented a method to
generalize the classifier’s robustness, the analysis is still at a nascent
stage as seen from the unreliable results between qualitative and quantitative
results. Even while these evaluation criteria exceeded industry standards and
performed well when compared to other studies in related fields, it was
discovered that they do not accurately reflect the complexity of the off-road
environment. To mimic human visual system, classifiers require large-norm
perturbations to be fooled. Reaching such levels require careful selection of
prior distribution, smoothness and dimension attributes. Further, intelligent
redundancy and ensembles of model-agnostic defences should be analysed in
fending off-adversarial attacks.

## References

  * (1)
  * Bubeck et al. (2018) Sébastien Bubeck, Eric Price, and Ilya Razenshteyn. 2018.  Adversarial examples from computational constraints.  arXiv:1805.10204 [stat.ML]
  * Carlini and Wagner (2017) Nicholas Carlini and David Wagner. 2017.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods.  arXiv:1705.07263 [cs.LG]
  * Chaurasia and Culurciello (2017) Abhishek Chaurasia and Eugenio Culurciello. 2017.  LinkNet: Exploiting encoder representations for efficient semantic segmentation. In _2017 IEEE Visual Communications and Image Processing (VCIP)_. IEEE, honolulu, 1–4.  https://doi.org/10.1109/vcip.2017.8305148
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. 2017.  DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs.  arXiv:1606.00915 [cs.CV]
  * Cordts et al. (2016) Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016.  The Cityscapes Dataset for Semantic Urban Scene Understanding.  arXiv:1604.01685 [cs.CV]
  * Fawzi et al. (2018) Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. 2018.  Adversarial vulnerability for any classifier.  arXiv:1802.08686 [cs.LG]
  * Geiger et al. (2013) A Geiger, P Lenz, C Stiller, and R Urtasun. 2013.  Vision meets robotics: The KITTI dataset.  _The International Journal of Robotics Research_ 32, 11 (2013), 1231–1237.  https://doi.org/10.1177/0278364913491297 arXiv:https://doi.org/10.1177/0278364913491297
  * Gilmer et al. (2018) Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S. Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. 2018.  Adversarial Spheres.  arXiv:1801.02774 [cs.CV]
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015.  Explaining and Harnessing Adversarial Examples.  arXiv:1412.6572 [stat.ML]
  * He et al. (2017) Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. 2017.  Adversarial Example Defenses: Ensembles of Weak Defenses are not Strong.  arXiv:1706.04701 [cs.LG]
  * Huang et al. (2017) Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. 2017.  Adversarial Attacks on Neural Network Policies.  arXiv:1702.02284 [cs.LG]
  * Ilyas et al. (2019) Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. 2019.  Adversarial Examples Are Not Bugs, They Are Features.  arXiv:1905.02175 [stat.ML]
  * Kurakin et al. (2017) Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2017.  Adversarial examples in the physical world.  arXiv:1607.02533 [cs.CV]
  * Lee and Ro (2021) Hong Lee and Yong Ro. 2021.  Adversarially Robust Multi-Sensor Fusion Model Training Via Random Feature Fusion For Semantic Segmentation.  , 339-343 pages.  https://doi.org/10.1109/ICIP42928.2021.9506748
  * Liang et al. (2020) Ming Liang, Bin Yang, Wenyuan Zeng, Yun Chen, Rui Hu, Sergio Casas, and Raquel Urtasun. 2020.  PnPNet: End-to-End Perception and Prediction with Tracking in the Loop.  arXiv:2005.14711 [cs.CV]
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015.  Fully Convolutional Networks for Semantic Segmentation.  arXiv:1411.4038 [cs.CV]
  * Mahloujifar et al. (2018) Saeed Mahloujifar, Dimitrios I. Diochnos, and Mohammad Mahmoody. 2018.  The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measure.  arXiv:1809.03063 [cs.LG]
  * Maturana et al. (2018) Daniel Maturana, Po-Wei Chou, Masashi Uenoyama, and Sebastian Scherer. 2018.  Real-Time Semantic Mapping for Autonomous Off-Road Navigation. In _Field and Service Robotics_ , Marco Hutter and Roland Siegwart (Eds.). Springer International Publishing, Cham, 335–350.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  arXiv:1505.04597 [cs.CV]
  * Rossolini et al. (2022) Giulio Rossolini, Federico Nesti, Gianluca D’Amico, Saasha Nair, Alessandro Biondi, and Giorgio Buttazzo. 2022.  On the Real-World Adversarial Robustness of Real-Time Semantic Segmentation Models for Autonomous Driving.  arXiv:2201.01850 [cs.CV]
  * Schmidt et al. (2018) Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Mądry. 2018.  Adversarially Robust Generalization Requires More Data.  arXiv:1804.11285 [cs.LG]
  * Sun et al. (2020) Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Sheng Zhao, Shuyang Cheng, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. 2020.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset.  arXiv:1912.04838 [cs.CV]
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014.  Intriguing properties of neural networks.  arXiv:1312.6199 [cs.CV]
  * Tanay and Griffin (2016) Thomas Tanay and Lewis Griffin. 2016.  A Boundary Tilting Persepective on the Phenomenon of Adversarial Examples.  arXiv:1608.07690 [cs.LG]
  * Uesato et al. (2018) Jonathan Uesato, Brendan O’Donoghue, Aaron van den Oord, and Pushmeet Kohli. 2018.  Adversarial Risk and the Dangers of Evaluating Against Weak Attacks.  arXiv:1802.05666 [cs.LG]
  * Valada et al. (2017) Abhinav Valada, Gabriel Oliveira, Thomas Brox, and Wolfram Burgard. 2017.  Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion. In _2016 International Symposium on Experimental Robotics_. Springer International Publishing, Cham, 465–477.  https://doi.org/10.1007/978-3-319-50115-4_41
  * Wolf (2022) Patrick Wolf. 2022.  _Cognitive Processing in Behavior-Based Perception of Autonomous Off-Road Vehicles_.  Dr. Hut verlag, Kai.
  * Wolf and Berns (2021) Patrick Wolf and Karsten Berns. 2021.  Data-fusion for robust off-road perception considering data quality of uncertain sensors. In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, 6876–6883.  https://doi.org/10.1109/IROS51168.2021.9636541
  * Wolf et al. (2023) Patrick Wolf, Pankaj Deoli, Satish Kumar Thangellapally, and Karsten Berns. 2023.  Traction Optimization for Robust Navigation in Unstructured Environments Using Deep Neural Networks on the Example of the Off-Road Truck Unimog. In _Intelligent Autonomous Systems 17_ , Ivan Petrovic, Emanuele Menegatti, and Ivan Marković (Eds.). Springer Nature Switzerland, Cham, 561–579.
  * Yatsura et al. (2023) Maksym Yatsura, Kaspar Sakmann, N. Grace Hua, Matthias Hein, and Jan Hendrik Metzen. 2023.  Certified Defences Against Adversarial Patch Attacks on Semantic Segmentation.  arXiv:2209.05980 [cs.CV]


# Factuality of Large Language Models in the Year 2024

Yuxia Wang1,3 Minghan Wang2 Muhammad Arslan Manzoor1 Fei Liu4
Georgi Georgiev5 Rocktim Jyoti Das1 Preslav Nakov1
1MBZUAI  2Monash University  3LibrAI  4Google  5Sofia University {yuxia.wang,
<EMAIL_ADDRESS>
###### Abstract

Large language models (LLMs), especially when instruction-tuned for chat, have
become part of our daily lives, freeing people from the process of _searching_
, _extracting_ , and _integrating_ information from multiple sources by
offering a straightforward answer to a variety of questions in a single place.
Unfortunately, in many cases, LLM responses are factually incorrect, which
limits their applicability in real-world scenarios. As a result, research on
evaluating and improving the factuality of LLMs has attracted a lot of
research attention recently. In this survey, we critically analyze existing
work with the aim to identify the major challenges and their associated
causes, pointing out to potential solutions for improving the factuality of
LLMs, and analyzing the obstacles to automated factuality evaluation for open-
ended text generation. We further offer an outlook on where future research
should go.

##  1 Introduction

Large language models (LLMs) have become an integral part of our daily lives.
When instruction-tuned for chat, they have enabled digital assistants that can
free people from the need to _search_ , _extract_ , and _integrate_
information from multiple sources by offering straightforward answers in a
single chat. While people naturally expect LLMs to always present reliable
information that is consistent with real-world knowledge, LLMs tend to
fabricate ungrounded statements, resulting in misinformation Tonmoy et al.
(2024), which limits their utility. Thus, assessing and improving the
factuality of the text generated by LLMs has become an emerging and crucial
research area, aiming to identify potential errors and to advance the
development of more reliable LLMs Chen et al. (2023).

To this end, researchers have collected multiple datasets, introduced a
variety of measures to evaluate the factuality of LLMs, and proposed numerous
strategies leveraging external knowledge through retrieval, self-reflection,
and early refinement in model generation to mitigate factual errors Tonmoy et
al. (2024). A number of surveys, discussing LLM hallucinations, have also been
published, which we summarize in Table 1.

##### Why this survey?

Survey | Date | Pages |  Eval-uation |  Improve-ment |  Multi-modal |  Contributions and limitations
---|---|---|---|---|---|---
This work | 01-Feb-2024 | 9 |  ✓ |  ✓ |  ✓ |  Discusses ambiguous concepts in LLM factuality, compares and analyzes evaluation and enhancement approaches from academic and practical perspectives, outlining major challenges and promising avenues to explore.
Tonmoy et al. (2024) | 08-Jan-2024 | 19 |  ✗ |  ✓ |  ✓ |  Summarizes recent work in terms of mitigating LLM hallucinations, but lacks comparison between different approaches and discussions to identify open questions and challenges.
Gao et al. (2023b) | 18-Dec-2023 | 26 |  ✗ |  ✓ |  ✗ |  Summarizes three RAG paradigms: naïve, advanced, and modular RAG, with key elements and evaluation methods for the three major components in RAG (retriever, generator, and augmentation).
Huang et al. (2023) | 09-Nov-2023 | 49 |  ✓ |  ✓ |  ✗ |  Analyzes the reasons for hallucinations, and presents a comprehensive overview of hallucination detection methods, benchmarks, and approaches to mitigate hallucinations.
Wang et al. (2023b) | 18-Oct-2023 | 44 |  ✓ |  ✓ |  ✗ |  Detailed literature review of factuality improvement and enhancement methods covering both retrieval augmentation and non-retrieval augmentation, missing discussion of major bottleneck issues in LLM factuality and promising directions to investigate.
Rawte et al. (2023b) | 18-Sept-2023 | 11 |  ✗ |  ✗ |  ✓ |  Extensively elucidates the problem of hallucination across all major modalities of foundation models, including text (general, multilingual, domain-specific LLMs), image, video, and audio. However, inadequate coverage of approaches, in-depth categorization and comparison between methods.
Zhang et al. (2023c) | 03-Sept-2023 | 32 |  ✓ |  ✓ |  ✗ |  Organized by different training stages of LLMs, discusses potential sources of LLM hallucinations and in-depth review of recent work on addressing the problem.
Guo et al. (2022) | Feb-2022 | 29 |  ✓ |  ✗ |  ✗ |  Focused on the automated fact-checking pipeline

Table 1: Comparison of different surveys on the factuality of LLMs.

We can see in Table 1 that the majority of recent surveys on the factuality of
LLMs or hallucination evaluation and improvements are rather lengthy, with
most over 20 pages. While they attempt to provide a comprehensive literature
review by summarizing interesting ideas and findings, organized by their
intended use at different stages in the training cycle of an LLM covering pre-
training, supervised fine-tuning (SFT), reinforcement learning with human or
automatic feedback (RLXF), and inference, it is difficult for readers to
understand the differences between some ambiguous concepts (e.g., LLM
factuality vs. hallucination), fundamental challenges and promising solutions
in factuality evaluation and enhancement. This survey, instead, focuses on
recent novel and representative works in each category, summarizes their
common bottlenecks and solutions to mitigate such issues while highlighting
our personal stance based on practical experience and observations.

In addition to LLM hallucination detection and mitigation strategies, Huang et
al. (2023); Zhang et al. (2023c) analyze potential reasons for LLM
hallucination. Gao et al. (2023b) focus on retrieval augmentation generation
(RAG) techniques whereas Guo et al. (2022) target automated fact-checking
systems. The work of Tonmoy et al. (2024) compiles many recent research
studies up to November 2023 albeit lacking an in-depth discussion. Rawte et
al. (2023b) review numerous efforts exploring hallucinations in foundational
models across various modalities, offering an overview of considerable scope
and breadth, but with limited depth. Our objective is to address these
challenges by concisely providing comprehensive insights into LLM factuality,
prioritizing recent studies to accommodate the rapidly evolving landscape.

##### Scope

We outline the scope of this survey below:

  * •

Unlike most prior surveys, we discuss factuality across a number of
modalities, including vision and speech, with an emphasis on factual errors in
text.

  * •

We focus on factuality of open-ended text generation without references, as
opposed to generation tasks with references (e.g., summarization or dialogue
with source documents or user-input contexts).

  * •

We pay particular attention to factual errors involving world knowledge, and
we leave other types (e.g., coding, reasoning, conflicting with user input or
previously generated contexts) for future work.

  * •

In addition to improving factuality and correcting factual errors, another
line of research investigates improving LLM calibration, making models aware
of their limitations in its knowledge and confidence in its output, enabling
it to either reject to respond to instructions or express uncertainties in its
answers when unsure. Given the in-depth discussion in Geng et al. (2023), We
exclude in this survey.

##### Structure

We first compare and contrast factuality in the context of LLMs against three
closely related concepts: hallucination, relevance and trustworthiness in §2.
Next, we outline benchmarks and evaluation metrics based on data format in §3.
We then discuss recent works on addressing factual errors in §4 for text and
other modalities in §5. Lastly, we highlight open questions and their
promising solutions in §6.

##  2 Background

Hallucination and factuality, while conceptually distinct, often occur in
similar contexts and are sometimes used interchangeably, rendering them
intricately intertwined, posing a challenge in discerning their distinct
boundaries, and causing a considerable amount of misconception. In this
section, we seek to disambiguate and refine our understanding of these two
closely aligned concepts, thereby preventing misinterpretation and reducing
potential confusion. Additionally, we further include two closely-related
axes: relevance and trustworthiness for LLM evaluation to illustrate their
nuance in relation to factuality.

##### Hallucination vs. Factuality

The concept of hallucination in the context of traditional natural language
generation tasks is typically referred to as the phenomenon in which the
generated content appears nonsensical or unfaithful to the provided source
content Ji et al. (2023). One concrete example is made-up information in an
abstractive summary with additional insights beyond the scope of the original
source document.

In the age of LLMs, the term hallucination has been reimagined, encompassing
any deviation from factual reality or the inclusion of fabricated elements
within generated texts Tonmoy et al. (2024); Rawte et al. (2023b). Zhang et
al. (2023c) define hallucination as the characteristic of LLMs to generate
content that diverges from the user input, contradicts previously generated
context, or mis-aligns with established world knowledge. Huang et al. (2023)
merge the input- and context-conflicting types of hallucinations and further
take logical inconsistency into account to form faithfulness hallucination.
Another category is factuality hallucination, referring to the discrepancy
between generated content and verifiable real-world facts, manifesting as (1)
factual inconsistency and (2) factual fabrication.

Factuality, on the other hand, is concerned with a model’s ability to learn,
acquire, and utilize factual knowledge. Wang et al. (2023b) characterize
factuality issues as the probability of LLMs producing content inconsistent
with established facts. It is important to note that hallucination content may
not always involve factual missteps. Though a piece of generated text may
exhibit divergence from the initial prompt’s specifics, it falls into
hallucinations, not necessarily a factual issue if the content is accurate.

It is crucial to distinguish between factual errors and instances of
hallucination. The former involves inaccurate information whereas the latter
can present unanticipated and yet factually substantiated content Wang et al.
(2023b).

Summary: Factuality is the ability of LLMs to generate content consistent with
factual information and world knowledge. Although both hallucinations and
factuality issues may impact the credibility of LLMs in the context of content
generation, they present distinct challenges. Hallucinations occur when LLMs
produce baseless or untruthful content, not grounded in the given source. In
contrast, factuality errors arise when the model fails to accurately learn and
utilize factual knowledge. It is possible for a model to be factually correct
yet still produce hallucinations by generating content that is either off-
topic or more detailed than what is requested.

##### Relevance vs. Factuality

FELM Chen et al. (2023) categorizes factual errors into four groups to better
understand and identify LLM vulnerabilities: knowledge, irrelevance, reasoning
and math, and misunderstanding falsehoods and jokes in prompts. Irrelevant
error refers to that the generated content is unrelated to the prompt. For
example, if the prompt is What’s a country where most people love playing
rugby? a response like New Zealand is a country where rugby is considered a
national passion and is deeply ingrained in the culture … would be labeled as
irrelevant though it is factually true. While not factually incorrect, this
response does not provide much helpful information.

##### Trustworthiness/Reliability vs. Factuality

In the context of LLMs, factuality Wang et al. (2023b) refers to a model’s
capability of generating contents of factual information, grouneded in
reliable sources (e.g., dictionaries, Wikipedia or textbooks), with
commonsense, world and domain-specific knowledge taken into account. In
contrast, “trustworthiness” Sun et al. (2024) extends beyond mere factual
accuracy and is measured on six dimensions: truthfulness, safety, fairness,
robustness, privacy, and ethics.

Type | Dataset | Topic | Size | ER% |  Evaluation and Metrics used in Original Paper | Freq
---|---|---|---|---|---|---
I |  FactScore-Bio Min and et al. (2023) | Biography | 549 | 42.6 | Human annotation and automated fact-checkers | 4
Factcheck-GPT Wang et al. (2023c) | Open-ended questions | 94 | 64.9 | Human annotation | 1
FacTool-QA Chern et al. (2023) | Knowledge-based QA | 50 | 54.0 | Human annotation and automated fact-checkers | 2
FELM-WK Chen et al. (2023) | Knowledge-based QA | 184 | 46.2 | Human annotation, Accuracy and F1 score | 1
HaluEval Li and et al. (2023a) | Open-ended questions | 5000 | 12.3 | Human annotation, AUROC + LLM judge + PARENT | 3
FreshQA Vu et al. (2023) | Open-ended questions | 499 | 68.0 | Human annotation | 2
SelfAware Yin et al. (2023b) | Open-ended questions | 3369 | – | Evaluate the LLM awareness of unknown by F1-score | 1
II |  Snowball Zhang et al. (2023b) | Yes/No question | 1500 | 9.4 | Exact match + Accuracy/F1-score | 1
III |  Wiki-category List Dhuliawala et al. (2023) |  Name some [Mexican films] | 55 | – | Precision/recall@5 | 1
Multispan QA Dhuliawala et al. (2023) | Short-term Answer | 428 | – | Exact match + F1 score | 1
IV |  TruthfulQA Lin et al. (2022) | False belief or misconception | 817 | – | Accuracy | 5
HotpotQA Yang and et al. (2018) | Multi-step reasoning | 113k | – | Exact match + F1 score | 11
StrategyQA Geva et al. (2021) | Multi-step reasoning | 2780 | – | Recall@10 | 3
MMLU Hendrycks et al. (2021) | Knowledge | 15700 | – | Accuracy | 4

Table 2: Four types of datasets used to evaluate LLM factuality. I: open-ended
generation; II: Yes/No answer; III: short-term or list of entities answer; IV:
A, B, C, D multiple Choice QA. Labeled datasets under type I are mostly
generated by ChatGPT, and FactScore-Bio (ChatGPT, InstGPT and PerplexityAI).
ER: Human-annotated Error Rate. Freq: usage frequency as evaluation set in our
first 50 references.

##  3 Evaluating Factuality

Evaluating LLM factuality on open-ended generations presents a non-trivial
challenge, discerning the degree to which a generated textual statement aligns
with objective reality. Studies employ various benchmarks, evaluation
strategies and metrics to achieve this goal.

###  3.1 Datasets and Metrics

While Zhang et al. (2023c) outlined tasks and measures for hallucination
evaluation, there is no comparative analysis of existing datasets to assess
various aspects in regards to model factuality (e.g., knowledge grounding,
fast-changing facts, snowballing hallucinations, robustness to false premises,
and uncertainty awareness). We categorize the datasets in the format of
discrimination or generation, and highlights the challenges in automatic
evaluation for long-form open-ended generations.

Current benchmarks largely assess factuality in LLMs based on two primary
capabilities: proficiency in distinguishing factual accuracy within context
and ability to generate factually sound content.

The former typically comes in the form of a multi-choice question, with the
expected response being a label of one of A, B, C, and D. For instance,
HotpotQA, StrategyQA, MMLU. This form of evaluation has been widely used to
measure the general knowledge proficiency and factual accuracy of LLMs,
largely thanks to its automation-friendly nature. Under this evaluation
formulation, model responses are easily parsed and compared with gold standard
labels, enabling the calculation of accuracy or F1 scores against established
benchmarks.

Precisely assessing the factuality of free-form LLM outputs remains a
significant challenge due to the inherent limitations of automatic methods in
the face of open-ended generation and the absence of definitive gold standard
responses within an expansive output space. To make automatic evaluation
feasible, many studies constrain the generation space to (1) Yes/No; (2)
short-form phrase; and (3) a list of entities through controlling the
categories of questions and generation length.

Perhaps the most demanding, yet inherently realistic scenario is free-form
long text generation, such as biography generation. For such generations, the
most commonly used and reliable methods rely on human experts following
specific guidelines, and automatic fact-checkers based on retrieved
information, such as FactScore, Factool and Factcheck-GPT, to facilitate
efficient and consistent evaluation.

These automatic fact-checkers generally first decompose a document into a set
of atomic claims, and then verify one by one whether the claim is true or
false based on the retrieved evidence, either from offline Wikipedia or online
Web pages. The percentage of true claims over all statements in a document is
used to reflect the factual status of a response (refer to FactScore). The
averaged Factscore over a dataset is in turn used to assess a model’s
factuality accuracy. However, there is no guarantee that automatic fact-
checkers are 100% accurate in their verification process. Wang et al. (2023c)
show that even the state-of-the-art verifier, equipped with GPT-4 and
supporting evidence retrieved with Google search, can only achieve an F1 score
of $0.63$ in identifying false claims and F1=0.53 using PerplexityAI (compared
with human-annotated labels for claims: true or false).

Summary: We categorize datasets that evaluate LLM factuality into four types,
depending on the answer space and the difficulty degree on which accurate
automatic quantification can be performed (see Table 2). They are: (I) open-
domain, free-form, long-term responses (FactScore: the percentage of the
correct claims verified by human or automated fact-checker); (II) Yes/No
answer w/wt explanation (extract Yes/No, metrics for binary classification);
(III) short-form answer (Exact match the answer with gold labels and calculate
accuracy) or the listing answer (recall@K); and (IV) multi-choice QA (metrics
for multi-class classification).

###  3.2 Other Metrics

In addition to evaluation methods discussed above, Lee et al. (2022) quantify
hallucinations using two metrics, both requiring document-level ground-truth:
(1) hallucinated named entities error measures the percentage of named
entities in generations that do not appear in the ground-truth Wikipedia
document; (2) entailment ratio evaluates the number of generations that can be
entailed by the ground-truth reference, against the number of all generations.

Rawte et al. (2023a) define hallucination vulnerability index (HVI) to
evaluate and rank LLMs based on their vulnerability to producing
hallucinations, which takes a spectrum of factors into account.

Some factuality measurement tasks, such as claim extraction and evidence
retrieval are non-trivial to be resolved automatically. Rawte et al. (2023a)
curate publicly available LLM hallucination mitigation benchmark, where LLM
generations are scored by humans when automated external knowledge retrieval
fails to resolve a claim clearly. While widely used for factuality evaluation,
this hybrid approach (i.e., retrieval + human) may suffer from human
annotation bias.

##  4 Improving Factuality

Improving the factuality of an LLM often requires updating its internal
knowledge, editing fake, outdated and biased elements, thereby making its
output reflect a revised collection of facts, maximizing the probability of
$P(\textrm{truth}|\textrm{prompt})$. One option is to adopt gradient-based
methods to update model parameters to encourage desired model output. This
includes pre-training, supervised fine-tuning and RLXF. We can also explore
injecting a new fact into LLMs or overwriting the false knowledge stored in
LLM memory by in-context learning (ICL). When models store factually correct
knowledge but produce errors, they can in some cases rectify them through
self-reasoning, reflection, and multi-agent debates.

We discuss these methods throughout the lifecycle of an LLM, ranging from pre-
training, to inference, to post-processing. Another important element is
retrieval augmentation, which enhances the generation capabilities of LLMs by
anchoring them in external knowledge that may not be stored or contradict the
information in LLM parametric memory. It can be incorporated at various stages
throughout model training and the subsequent inference process Gao et al.
(2023b), and is therefore not discussed individually.

###  4.1 Pre-training

LLMs store a vast amount of world knowledge in their parameters through the
process of pre-training. The quality of the pre-training data plays a crucial
role and misinformation could potentially cause LLMs to generate false
responses, motivating the utilization of high-quality textual corpora.
However, the prohibitively massive amount of pre-training data, typically
consisting of trillions of tokens, renders manual filtering and editing
impractically laborious. To this end, automated filtering methods have been
proposed. For instance, Brown et al. (2020) introduce a method to only focus
on a small portion of the CommonCrawl dataset that exhibits similarity to
high-quality reference corpora. Touvron et al. (2023) propose to enhance
factual robustness of mixed corpora by up-sampling documents from the most
reliable sources, thereby amplifying knowledge accuracy and mitigating
hallucinations. During the pre-training phase of phi-1.5, Li and et al.
(2023b) synthesize “textbook-like” data, consists of and rich in high-quality
commonsense reasoning and world knowledge. While careful corpus curation
remains the cornerstone of pre-training for enhanced factuality, the task
becomes increasingly challenging with the expansion of dataset scale and the
growing demand for linguistic diversity. It is therefore crucial to develop
novel strategies that guarantee the consistency of factual knowledge across
diverse cultural landscapes.

Borgeaud et al. (2021) propose RETRO, a retrieval augmented pre-training
approach. An auto-regressive LLM is trained from scratch with a retrieval
module that is practically scalable to large-scale pre-training by retrieving
billions of tokens. RETRO shows better accuracy and is less prone to
hallucinate compared to GPT Wang et al. (2023a). While limitations lie in that
RETRO performance could be compromised if the retrieval database contains
inaccurate, biased or outdated information. $\sim$25$\%$ additional
computation is required for the pre-training of LLMs with retrieval.

###  4.2 Tuning and RLXF

Continued domain-specific SFT has shown to be effective for enhancing
factuality, particularly in the absence of such knowledge during pre-training.
For instance, Elaraby et al. (2023) enhance the factual accuracy of LLMs
through knowledge injection (KI). Knowledge, in the form of entity summaries
or entity triplets, is incorporated through SFT by either intermediate tuning,
i.e. first on knowledge and then on instruction data; or combined tuning, i.e.
on the mixture of both. While some improvements are exhibited, the method
alone can be insufficient to fully mitigate factual errors.

For general-purpose LLMs, SFT is typically employed to improve the
instruction-following capabilities as opposed to factual knowledge which is
mostly learned in pre-training. However, this process may inadvertently reveal
areas of knowledge not covered in the pre-training, causing the risk of
behavior cloning, where a model feigns understanding and responds with
hallucinations to questions it has little knowledge of Torabi et al. (2018).
R-tuning Zhang et al. (2023a) is proposed to address this issue with two
pivotal steps: first, assessing the knowledge gap between the model’s
parametric knowledge and the instruction tuning data, and second, creating a
refusal-aware dataset for SFT. It enables LLMs to abstain from answering
queries beyond their parametric knowledge scope. On the other hand, BeInfo
Razumovskaia et al. (2023) improve factual alignment through the form of
behavioral fine-tuning. The creation of the behavioral tuning dataset
emphasizes two goals: selectivity (choosing correct information from the
knowledge source) and response adequacy (informing the user when no relevant
information is available or asking for clarification). Both methods
effectively control LLMs on non-parametric questions but require extra effort
in dataset curation and might hinder the models’ retention of parametric
knowledge.

Sycophancy Sharma et al. (2023), known as another source of factuality errors,
often arises from misalignments during SFT and RLHFOuyang et al. (2022). This
problem is partially attributed to human annotators’ tendency to award higher
scores to responses they like rather than those that are factually accurate.
Wei et al. (2023) explore the correlation of sycophancy with model scaling and
instruction tuning. They propose a synthetic-data intervention method, using
various NLP tasks to teach models that truthfulness is independent of user
opinions. However, one limitation is that the generalizability of their
approach remains unclear for varied prompt formats and diverse user opinions.

Tian et al. (2023) utilize direct preference optimization (DPO) Rafailov et
al. (2023) with the feedback of factuality score either from automatic fact-
checkers or LLMs predictive confidence. In-domain evaluation shows promising
results on biographies and medical queries, but generalization performance
across domains and unseen domains is under-explored. Köksal et al. (2023)
propose hallucination-augmented recitations (HAR). It encourages the model to
attribute to the contexts rather than its parametric knowledge, by tuning the
model on the counterfactual dataset created leveraging LLM hallucinations.
This approach offers a novel way to enhance LLM attribution and grounding in
open-book QA. However, challenges lie in refining counterfactual generation
for consistency and expanding its application to broader contexts.

##### Retrieval Augmentation

Incorporating retrieval mechanisms during fine-tuning has been shown to
enhance the LLM factuality on downstream tasks, particularly in open-domain
QA. DPR Karpukhin et al. (2020) refines a dual-encoder framework, consisting
of two BERT models. It employs a contrastive loss to align the hidden
representations of questions and their corresponding answers, obtained through
the respective encoder models. RAG Lewis et al. (2020) and FiD Izacard and
Grave (2020) study a fine-tuning recipe for retrieval-augmented generation
models, focusing on open-domain QA tasks. WebGPT Nakano et al. (2021) fine-
tunes GPT-3 Brown et al. (2020) by RLHF, providing questions with factually
correct long-form reference generation. The implementation in a text-based
web-browsing environment allows the model to search and navigate the web.

###  4.3 Inference

We categorize approaches to improve factuality during inference into two: (1)
optimizing decoding strategies to strengthen model factuality; and (2)
empowering LLM learned ability by either in-context learning (ICL) or self-
reasoning.

####  4.3.1 Decoding Strategy

Sampling from the top subword candidates with a cumulative probability of p,
known as nucleus sampling (top-p) Holtzman et al. (2020), sees a decrease in
factuality performance compared to greedy decoding, despite higher diversity.
This is likely due to its over-encouragement of randomness. Building on the
hypothesis that sampling randomness may damage factuality when generating the
latter part of a sentence than the beginning, Lee et al. (2022) introduce
factual-nucleus sampling, which dynamically reduces the nucleus-p value as
generation progresses to limit diversity and improve factuality, modulating
factual integrity and textual diversity.

Apart from randomness, some errors arise when knowledge conflicts, where
context contradicts information present in the model’s prior knowledge.
Context-aware decoding (CAD) Shi et al. (2023) prioritizes current context
over prior knowledge, and employs contrastive ensemble logits, adjusting the
weight of the probability distribution when predicting the next token with or
without context. Despite the factuality boost, CAD is a better fit for tasks
involving knowledge conflicts and heavily reliant on high-quality context.

In contrast, DoLa Chuang et al. (2023) takes into account both upper and lower
(earlier) layers, as opposed to only the final (mature) layer. This method
dynamically selects intermediate layers at each decoding step, in which an
appropriate premature layer contains less factual information with maximum
divergence among the subset of the early layers. This method effectively
harnesses the distinct contributions of each layer to factual generations.
However, DoLa increases the decoding time by 1.01x to 1.08x and does not
utilize external knowledge, which limits its ability to correct misinformation
learned during training.

####  4.3.2 ICL and Self-reasoning

In context learning (ICL) allows an LLM to leverage and learn from
demonstration examples in its context to perform a particular task without the
need to update model parameters. Zheng et al. (2023) present that it is
possible to perform knowledge editing via ICL through facts included in
demonstration examples, thereby correcting fake or outdated facts. The
objective of demonstration examples is to teach LLMs how to: (1) identify and
copy an answer; (2) generalize using in-context facts; (3) ignore irrelevant
facts in context.

While it is rather easy for LLMs to copy answers from contexts, changing
predictions of questions related to the new facts accordingly, and keeping the
original predictions if the question is irrelevant to the modified facts,
remains tough.

Another line of research leverages the self-reasoning capability of LLMs. Du
et al. (2023) improve LLM factuality through multi-agent debate. This approach
first instantiates a number of agents and then makes them debate over answers
returned by other agents until a consensus is reached. One interesting finding
is that more agents and longer debates tend to lead to better results. This
approach is orthogonal and can be applied in addition to many other generation
methods, such as complex prompting strategy (e.g., CoT Wei et al. (2022),
ReAct Yao et al. (2023), Reflexion Shinn et al. (2023)) and retrieval
augmentation.

Take-away: Zheng et al. (2023) evaluate the effectiveness of knowledge editing
on subject-relation-object triplets, an unrealistic setting compared to open-
ended free-form text assessment. we seek answers to two research questions:
(1) What types of facts and to what extent can facts be edited effectively,
learned by LLMs through ICL? (2) Would SFT do a better job at learning from
examples that are difficult for ICL? More broadly, what is the best way to
insert new facts or edit false knowledge stored in LLMs. The community may
also benefit from an in-depth comparative analysis of the effectiveness of
improving factuality between SFT and ICL (perhaps also RLXF).

##### Retrieval Augmentation

can be applied before, during, and after model generation.

One commonly used option is to apply retrieval augmentation prior to response
generation. For questions requiring up-to-date world knowledge to answer, Vu
et al. (2023) augment LLM prompts with web-retrieved information and
demonstrate the effectiveness on improving accuracy on FreshQA, where ChatGPT
and GPT-4 struggle due to their lack of up-to-date information. Gao et al.
(2023a) place all relevant paragraphs in the context and encourage the model
to cite supporting evidence, instructing LLMs to understand retrieved
documents and generate correct citations, thereby improving reliability and
factuality.

Pre-generation retrieval augmentation is beneficial as the generation process
is conditioned on the retrieval results, implicitly constraining the output
space. While improving factual accuracy, this comes at the cost of spontaneous
and creative responses, largely limiting the capabilities of LLMs. An
alternative method is to verify and rectify factual errors after the model
generates all content. However, LLMs have been shown to be susceptible to
hallucination snowballing Zhang et al. (2023b), a common issue where a model
attempts to make its response consistent with previously generated content
even if it is factually incorrect.

Striking a balance between preserving creative elements and avoiding the
propagation of potential errors, EVER Kang et al. (2023) and “a stitch in time
saves nine” Varshney et al. (2023) actively detect and correct factual errors
during generation sentence by sentence. The former leverages retrieved
evidence for verification, and the latter further incorporates the probability
of dominant concepts in detection. Their findings suggest that timely
correcting errors during generation can prevent snowballing and further
improve factuality. Nonetheless, the primary concern for this iterative
process of generate-verify-correct in real-time systems is latency, making it
difficult to meet the high-throughput and responsiveness demand Kang et al.
(2023).

###  4.4 Automatic Fact Checkers

Figure 1: Fact-checker framework: claim processor, retriever, and verifier,
with optional step of summarizing and explaining in gray.

An automatic fact-checking framework typically consists of three components:
claim processor, retriever, and verifier as shown in Figure 1, though the
implementation of verification pipelines may differ. For example, FACTOR
Muhlgay et al. (2023) and FactScore Min and et al. (2023) only detect
falsehoods without correction. While RARR depends on web-retrieved information
Gao et al. (2022), and CoVe Dhuliawala et al. (2023) only relies on LLM
parametric knowledge Dhuliawala et al. (2023) to perform both detection and
correction, albeit at a coarse granularity, editing the entire document.
Compared to fine-grained verification over claims, it is unable to spot false
spans precisely and tends to result in poor preservation of the original
input. FacTool Chern et al. (2023) and Factcheck-GPT Wang et al. (2023c) edit
atomic claims. While the former breaks a document down to independent
checkworthy claims with three steps: decomposition, decontextualization and
checkworthiness identification, the latter employs GPT-4 to extract verifiable
claims directly. Evaluating the effectiveness of fact-checkers remains
challenging, making the improvement of such systems a difficult task.

##### Engineering and Practical Considerations

Automatic fact-checking involve tasks of extracting atomic check-worthy
claims, collecting evidence either by leveraging the knowledge stored in the
model parameters or retrieved externally, and verification. While
straightforward to implement, this pipeline may be susceptible to error
propagation. Major bottleneck lies in the absence of automatic evaluation
measures to assess the quality of intermediate steps, in particular, the claim
processor and evidence retriever as there is no gold standard.

The input to a claim processor is a document to check and its expected output
is a list of atomic check-worthy claims or atomic verifiable facts. There is
no consensus on the granularity of “atomic claims”, making consistent
decomposition difficult in a practical setting. Additionally, the concept of
check-worthy and verifiable claims are subjective. Consequently, the
definition of an atomic check-worthy claim remains a highly debatable topic.

This naturally leads to different “gold” human-annotated atomic claims
annotated following various guidelines and distinct implementation approaches
to decompose a document.

Given a document, even if assuming a ground-truth list of atomic claims, it
remains an open question how to assess the quality of automatically derived
decomposition results. Wang et al. (2023c) start by assessing the agreement in
the number of claims between the ground truth and predictions, followed by
examining the semantic similarity between two claims at the same index when
the claim count aligns. Entailment ratio presented in Section 3.2 is also
applicable Lee et al. (2022).

While it is much simpler when the evidence is constrained (e.g., to Wikipedia
documents as is the case for FEVER Thorne et al. (2018)), accurate retrieval
of evidence from the Internet and subsequently quantifying the quality of such
retrieval results remain challenging. Similar to the assessment of atomic
claims, gold-labeled evidence is unavailable and infeasible to obtain in the
expansive open search space.

The only step where we can confidently evaluate its quality is the accuracy of
verification, a simple binary true/false label given a document/claim. In
conclusion, perhaps the most significant hurdle limiting the development and
improvement of automatic fact-checkers lies in the automated assessment and
quantification of quality at intermediate stages.

##  5 Factuality of Multimodal LLMs

More recently, multimodal LLMs (MLLMs), capable of processing visual and audio
contexts (in addition to textual input), have become a new research focus.
Despite impressive results, MLLMs also hallucinate and existing research
primarily focuses on addressing grounding errors.

For tuning-based approaches, Sun et al. (2023) apply RLHF to vision-language
alignment tasks. They introduce factually-augmented RLHF, which augments a
reward model with additional factual information, such as image captions and
ground-truth options, to alleviate reward hacking. Zhao et al. (2023)
introduce hallucination-aware DPO by first constructing a hallucination-aware
dataset consisting of style-consistent pairs with positive and negative
samples curated automatically with the help of GPT-4, and then, performing DPO
training with MLLMs. While these methods enhance truthfulness, challenges
remain, in the form of expensive and intricate data generation pipelines
needed for creating tuning datasets.

Another line of research focuses on post-hoc approaches. Yin et al. (2023a)
introduce a training-free pipeline of five steps: concept extraction, question
formulation, visual validation, claim generation, and hallucination
correction. They utilize textual LLMs, object detection, and visual QA (VQA)
models to fix grounding errors in MLLM responses, framing detected objects and
attributes as evidence. Rawte et al. (2023b) indicate that adapting textual
factuality methods to MLLMs often requires conversion from various media to
text (e.g. captioning). However, with increased complexity in multimodal
contents, despite the application of various techniques such as VQA, OCR and
ASR, fact-checking pipelines for images, audio, and video and text are largely
similar. Challenges remain in accurate claim extraction and evidence retrieval
across different modalities Akhtar et al. (2023).

##  6 Challenges and Future Directions

We first identify three major challenges for improving the factuality of LLMs,
and then we point to several promising directions for future work.

##### Language models learn the distribution of a language, not facts.

The training objective of language modeling is to maximize the probability of
a sentence, as opposed to that of a factual statement. While capable of
generating seemingly coherent and fluent outputs upon convergence, models are
not guaranteed to always return factually accurate contents.

##### Automatic evaluation of factual accuracy of open-ended generations
remains challenging.

Existing studies on factuality enhancement use different benchmarks and
evaluation metrics, making fair comparisons difficult, motivating the need for
a unified automated evaluation framework utilizing the same collection of
datasets and metrics.

This can be further decomposed into two questions: (1) how to select a set of
datasets that can effectively evaluate LLMs factuality? and (2) how to measure
the percentage of factual claims given a model response with minimal time
expenditure? Current approaches rely on either human evaluation or results of
automated fact-checkers, e,g., FactScore and FacTool Min and et al. (2023);
Chern et al. (2023). However, automatically quantifying the quality of
automated fact-checkers is itself an open question, resulting in a chicken and
egg situation.

##### Latency and multi-hop reasoning could be the bottleneck of RAG systems.

Retrievers serve as the core component in RAG systems, and the effectiveness
of RAG is largely influenced by the quality (coverage and relevance) of the
retrieved documents. Latency and difficulties in gathering the most pertinent
evidence are the primary challenges in retrieval. While this is partly due to
the inability of ranking algorithms to retrieve such documents, certain facts
require information gathered from various sources and multi-hop reasoning.

##### Potential Future Directions

Mitigation in inference: We observe that models can often generate the correct
answer in multiple trials even if some of those attempts are incorrect Tian et
al. (2023). This motivates us to ask how to provide an anchor that can guide
LLM decoding to the factually correct path?

Iteratively detecting, correcting and generating during generation has been
demonstrated to be effective to mitigating hallucinations. If simply
correcting the first one or two sentences, how much improvements can we expect
for subsequent generations? Can factually correct and relevant sentences,
phrases or concepts serve as the anchors?

Timely detection of factual errors: Another interesting option to explore is
to only detect falsehoods without consideration of correction, allowing LLMs
to incorporate uncertainty information and the contrastive probability
leveraged in decoding strategies (e.g., CAD and DoLa) for timely detections
without evidence.

Improving efficiency and accuracy of automated fact-checkers: The key
breakthrough in effectively evaluating the factual accuracy of LLMs lies in
establishing accurate and efficient fact-checkers, in which how to evaluate
the quality of retrieved evidence in open search space is of significant
importance. Current methods rely on the accuracy of claim verification to
deduce the quality of evidence by prompting LLMs. Not only is this
computationally expensive, it also tends to be highly sensitive to minor
prompt changes and LLM updates (e.g., the same prompt may result in
inconsistent outputs). Instead, a small task-specific and well fine-tuned NLI
model presents a more viable, robust and cost-efficient option.

##  7 Conclusion

We presented an overview on the factuality of LLMs, surveying a number of
studies covering topics such as evaluation in Section 3 and improvement
methods (applicable at various stages: pre-training, SFT, inference and post-
processing) in Section 4, along with their respective challenges. We also
identified three major issues and additionally pointed out promising future
research directions in Section 6.

## References

  * Akhtar et al. [2023] Mubashara Akhtar, Michael Schlichtkrull, and Zhijiang et al. Guo.  Multimodal automated fact-checking: A survey.  In Findings of EMNLP 2023, pages 5430–5448, 2023.
  * Borgeaud et al. [2021] Sebastian Borgeaud, Arthur Mensch, and Jordan Hoffmann et al.  Improving language models by retrieving from trillions of tokens.  In ICML, 2021.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, and Melanie Subbiah et al.  Language models are few-shot learners.  In NeurIPS 2020, 2020.
  * Chen et al. [2023] Shiqi Chen, Yiran Zhao, Jinghan Zhang, and et al.  Felm: Benchmarking factuality evaluation of large language models.  arXiv preprint arXiv:2310.00741, 2023.
  * Chern et al. [2023] I-Chun Chern, Steffi Chern, and Shiqi Chen et al.  Factool: Factuality detection in generative AI - A tool augmented framework for multi-task and multi-domain scenarios.  CoRR, abs/2307.13528, 2023.
  * Chuang et al. [2023] Yung-Sung Chuang, Yujia Xie, and Hongyin Luo et al.  Dola: Decoding by contrasting layers improves factuality in large language models.  CoRR, abs/2309.03883, 2023.
  * Dhuliawala et al. [2023] Shehzaad Dhuliawala, Mojtaba Komeili, and et al.  Chain-of-verification reduces hallucination in large language models.  arXiv preprint arXiv:2309.11495, 2023.
  * Du et al. [2023] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch.  Improving factuality and reasoning in language models through multiagent debate.  CoRR, abs/2305.14325, 2023.
  * Elaraby et al. [2023] Mohamed Elaraby, Mengyin Lu, and Jacob Dunn et al.  Halo: Estimation and reduction of hallucinations in open-source weak large language models.  CoRR, abs/2308.11764, 2023.
  * Gao et al. [2022] Luyu Gao, Zhuyun Dai, and Panupong et al. Pasupat.  Attributed text generation via post-hoc research and revision.  arXiv preprint arXiv:2210.08726, 2022.
  * Gao et al. [2023a] Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen.  Enabling large language models to generate text with citations.  In EMNLP, pages 6465–6488, 2023.
  * Gao et al. [2023b] Yunfan Gao, Yun Xiong, and et al.  Retrieval-augmented generation for large language models: A survey.  CoRR, abs/2312.10997, 2023.
  * Geng et al. [2023] Jiahui Geng, Fengyu Cai, and Yuxia Wang et al.  A survey of language model confidence estimation and calibration.  CoRR, abs/2311.08298, 2023.
  * Geva et al. [2021] Mor Geva, Daniel Khashabi, and et al.  Did aristotle use a laptop? A question answering benchmark with implicit reasoning strategies.  TACL, 9:346–361, 2021.
  * Guo et al. [2022] Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos.  A survey on automated fact-checking.  TACL, 10:178–206, 2022.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, and et al.  Measuring massive multitask language understanding.  In ICLR 2021, 2021.
  * Holtzman et al. [2020] Ari Holtzman, Jan Buys, and Li et al.  The curious case of neural text degeneration.  In ICLR, 2020.
  * Huang et al. [2023] Lei Huang, Weijiang Yu, Weitao Ma, and Weihong Zhong et al.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  CoRR, abs/2311.05232, 2023.
  * Izacard and Grave [2020] Gautier Izacard and Edouard Grave.  Leveraging passage retrieval with generative models for open domain question answering.  ArXiv, abs/2007.01282, 2020.
  * Ji et al. [2023] Ziwei Ji, Nayeon Lee, and Rita Frieske et al.  Survey of hallucination in natural language generation.  ACM Comput. Surv., 55(12):248:1–248:38, 2023.
  * Kang et al. [2023] Haoqiang Kang, Juntong Ni, and Huaxiu Yao.  Ever: Mitigating hallucination in large language models through real-time verification and rectification.  CoRR, abs/2311.09114, 2023.
  * Karpukhin et al. [2020] Vladimir Karpukhin, Barlas Oğuz, and Sewon Min et al.  Dense passage retrieval for open-domain question answering.  ArXiv, abs/2004.04906, 2020.
  * Köksal et al. [2023] Abdullatif Köksal, Renat Aksitov, and Chung-Ching Chang.  Hallucination augmented recitations for language models.  arXiv preprint arXiv:2311.07424, 2023.
  * Lee et al. [2022] Nayeon Lee, Wei Ping, and Peng et al. Xu.  Factuality enhanced language models for open-ended text generation.  NeuralPS, 35:34586–34599, 2022.
  * Lewis et al. [2020] Patrick Lewis, Ethan Perez, and et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  ArXiv, abs/2005.11401, 2020.
  * Li and et al. [2023a] Junyi Li and Xiaoxue Cheng et al.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  CoRR, abs/2305.11747, 2023.
  * Li and et al. [2023b] Yuanzhi Li and Sébastien Bubeck et al.  Textbooks are all you need II: phi-1.5 technical report.  CoRR, abs/2309.05463, 2023.
  * Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans.  Truthfulqa: Measuring how models mimic human falsehoods.  In ACL, pages 3214–3252, 2022.
  * Min and et al. [2023] Sewon Min and Kalpesh Krishna et al.  Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.  CoRR, abs/2305.14251, 2023.
  * Muhlgay et al. [2023] Dor Muhlgay, Ori Ram, and Inbal Magar et al.  Generating benchmarks for factuality evaluation of language models.  CoRR, abs/2307.06908, 2023.
  * Nakano et al. [2021] Reiichiro Nakano, Jacob Hilton, and et al.  Webgpt: Browser-assisted question-answering with human feedback.  ArXiv, abs/2112.09332, 2021.
  * Ouyang et al. [2022] Long Ouyang, Jeff Wu, and Xu Jiang et al.  Training language models to follow instructions with human feedback.  ArXiv, abs/2203.02155, 2022.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, and et al.  Direct preference optimization: Your language model is secretly a reward model.  CoRR, abs/2305.18290, 2023.
  * Rawte et al. [2023a] Vipula Rawte, Swagata Chakraborty, and Agnibh et al. Pathak.  The troubling emergence of hallucination in large language models - an extensive definition, quantification, and prescriptive remediations.  In EMNLP 2023, pages 2541–2573, 2023.
  * Rawte et al. [2023b] Vipula Rawte, Amit P. Sheth, and Amitava Das.  A survey of hallucination in large foundation models.  CoRR, abs/2309.05922, 2023.
  * Razumovskaia et al. [2023] Evgeniia Razumovskaia, Ivan Vulic, and Pavle Markovic et al.  Dial beinfo for faithfulness: Improving factuality of information-seeking dialogue via behavioural fine-tuning.  CoRR, abs/2311.09800, 2023.
  * Sharma et al. [2023] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, and Amanda Askell et al.  Towards understanding sycophancy in language models.  CoRR, abs/2310.13548, 2023.
  * Shi et al. [2023] Weijia Shi, Xiaochuang Han, and et al.  Trusting your evidence: Hallucinate less with context-aware decoding.  arXiv preprint arXiv:2305.14739, 2023.
  * Shinn et al. [2023] Noah Shinn, Federico Cassano, and Gopinath et al.  Reflexion: Language agents with verbal reinforcement learning.  In NeuralPS, 2023.
  * Sun et al. [2023] Zhiqing Sun, Sheng Shen, and Shengcao Cao et al.  Aligning large multimodal models with factually augmented RLHF.  CoRR, abs/2309.14525, 2023.
  * Sun et al. [2024] Lichao Sun, Yue Huang, and Haoran Wang et al.  Trustllm: Trustworthiness in large language models.  ArXiv, abs/2401.05561, 2024.
  * Thorne et al. [2018] James Thorne, Andreas Vlachos, and et al.  FEVER: a large-scale dataset for fact extraction and VERification.  In NAACL, pages 809–819, 2018.
  * Tian et al. [2023] Katherine Tian, Eric Mitchell, and et al.  Fine-tuning language models for factuality.  arXiv preprint arXiv:2311.08401, 2023.
  * Tonmoy et al. [2024] S. M. Towhidul Islam Tonmoy, S. M. Mehedi Zaman, and et al.  A comprehensive survey of hallucination mitigation techniques in large language models.  CoRR, abs/2401.01313, 2024.
  * Torabi et al. [2018] Faraz Torabi, Garrett Warnell, and Peter Stone.  Behavioral cloning from observation.  In Jérôme Lang, editor, IJCAI, pages 4950–4957. ijcai.org, 2018.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, and et al.  Llama 2: Open foundation and fine-tuned chat models.  CoRR, abs/2307.09288, 2023.
  * Varshney et al. [2023] Neeraj Varshney, Wenlin Yao, and et al.  A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation.  CoRR, abs/2307.03987, 2023.
  * Vu et al. [2023] Tu Vu, Mohit Iyyer, and et al.  Freshllms: Refreshing large language models with search engine augmentation.  arXiv preprint arXiv:2310.03214, 2023.
  * Wang et al. [2023a] Boxin Wang, Wei Ping, and et al.  Shall we pretrain autoregressive language models with retrieval? a comprehensive study.  In EMNLP, 2023.
  * Wang et al. [2023b] Cunxiang Wang, Xiaoze Liu, and et al.  Survey on factuality in large language models: Knowledge, retrieval and domain-specificity.  ArXiv, abs/2310.07521, 2023.
  * Wang et al. [2023c] Yuxia Wang, Revanth Gangi Reddy, and et al.  Factcheck-gpt: End-to-end fine-grained document-level fact-checking and correction of LLM output.  CoRR, abs/2311.09000, 2023.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, and Dale et al.  Chain-of-thought prompting elicits reasoning in large language models.  In NeurIPS 2022, 2022.
  * Wei et al. [2023] Jerry W. Wei, Da Huang, and Yifeng Lu et al.  Simple synthetic data reduces sycophancy in large language models.  CoRR, abs/2308.03958, 2023.
  * Yang and et al. [2018] Zhilin Yang and Peng Qi et al.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In EMNLP 2018, pages 2369–2380, 2018.
  * Yao et al. [2023] Shunyu Yao, Jeffrey Zhao, and Dian et al.  React: Synergizing reasoning and acting in language models.  In ICLR, 2023.
  * Yin et al. [2023a] Shukang Yin, Chaoyou Fu, and et al.  Woodpecker: Hallucination correction for multimodal large language models.  CoRR, abs/2310.16045, 2023.
  * Yin et al. [2023b] Zhangyue Yin, Qiushi Sun, and Qipeng Guo et al.  Do large language models know what they don’t know?  In ACL, pages 8653–8665, 2023.
  * Zhang et al. [2023a] Hanning Zhang, Shizhe Diao, and et al.  R-tuning: Teaching large language models to refuse unknown questions.  CoRR, abs/2311.09677, 2023.
  * Zhang et al. [2023b] Muru Zhang, Ofir Press, and et al.  How language model hallucinations can snowball.  CoRR, abs/2305.13534, 2023.
  * Zhang et al. [2023c] Yue Zhang, Yafu Li, and et al.  Siren’s song in the AI ocean: A survey on hallucination in large language models.  CoRR, abs/2309.01219, 2023.
  * Zhao et al. [2023] Zhiyuan Zhao, Bin Wang, and Linke Ouyang et al.  Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization.  CoRR, abs/2311.16839, 2023.
  * Zheng et al. [2023] Ce Zheng, Lei Li, and et al.  Can we edit factual knowledge by in-context learning?  In EMNLP, pages 4862–4876, 2023.