# Localization and delocalization of light in photonic moiré lattices

Peng Wang,1 Yuanlin Zheng,1 Xianfeng Chen,1 Changming Huang,2 Yaroslav V.
Kartashov,3,4 Lluis Torner,3,5 Vladimir V. Konotop,6 Fangwei Ye1∗
_1 School of Physics and Astronomy, Shanghai Jiao Tong University, Shanghai
200240, China_
_2 Department of Electronic Information and Physics, Changzhi University,
Shanxi 046011, China_
_3 ICFO-Institut de Ciencies Fotoniques, The Barcelona Institute of Science
and Technology,_
_08860 Castelldefels (Barcelona), Spain_
_4 Institute of Spectroscopy, Russian Academy of Sciences, Troitsk, Moscow,
108840, Russia_
_5 Universitat Politecnica de Catalunya, 08034 Barcelona, Spain_
_6 Departamento de Física, and Centro de Física Teórica e Computacional,_
_Faculdade de Ciências, Universidade de Lisboa, Campo Grande, Ed. C8, Lisboa
1749-016, Portugal_
_∗ Corresponding author<EMAIL_ADDRESS>
Moiré lattices consist of two identical periodic structures overlaid with a
relative rotation angle. Observed even in everyday life, moiré lattices have
been also produced with coupled graphene-hexagonal boron nitride monolayers
GrapheneMoire ; Woods14 ; Thermal , graphene-graphene layers BandFlatGraphene
; UnconvenSupercond , and layers on a silicon carbide surface Science . The
recent surge of interest in moiré lattices is connected with a possibility to
explore in such structures a rich variety of unusual physical phenomena, such
as commensurable-incommensurable transitions and topological defects Woods14 ,
emergence of insulating states due to band flattening BandFlatGraphene ;
flat_band , unconventional superconductivity UnconvenSupercond  controlled by
the rotation angle between monolayers magic ; flat_hexagon2 , quantum Hall
effect Dean13 , realization of non-Abelian gauge potentials Gauge , as well as
quasicrystals appearing at special rotation angles Stampfli . However, a
fundamental question that remains unexplored is the evolution of waves in the
potentials defined by the moiré lattices. Here we experimentally create two-
dimensional photonic moiré lattices, which, unlike their crystalline
predecessors, have controllable parameters and symmetry allowing to explore
transitions between structures with fundamentally different periodicity:
periodic, general aperiodic and quasi-crystal ones. Equipped with such
realization, we observe localization of light in deterministic linear
lattices. Such localization is based on the flat-band physics flat_band , thus
contrasting with previous schemes based on light diffusion in optical
quasicrystals Freeman , where disorder is required Levi  for the onset of
Anderson localization Anderson2D . Using commensurable and incommensurable
moiré patterns, we also provide the first experimental demonstration of two-
dimensional localization-delocalization-transition (LDT) of wavepacket in an
optical setting. Moiré lattices may feature almost arbitrary geometry
consistent with the crystallographic symmetry groups of the sublattices, they
allow comprehensive exploration of the physics of transitions between periodic
and aperiodic phases and offer an alternative way for manipulating light by
light.

One of the most important properties of an engineered optical system is its
capability to affect the light beam in a prescribed manner, in particular to
localize it or to control its diffraction. The importance of the problem of
wave localization extends far beyond optics. It was studied in all branches of
physics dealing with wave phenomena. In particular, it is well known that
homogeneous or strictly periodic linear systems cannot result in localization
of waves, and that inhomogeneities, random or regular, or nonlinear effects
are required for localization. Wave localization in a random medium, alias
Anderson localization Anderson , is a hallmark discovery of the condensed-
matter physics. All electronic states in one- and two-dimensional potentials
with uncorrelated disorder are localized. Three-dimensional systems with
disordered potentials are known to have both localized and delocalized
eigenstates Anderson2D , separated by an energy known as the mobility edge
mobility .

If disorder is correlated, a mobility edge may exist in one-dimensional
systems too, as it was observed experimentally for a Bose-Einstein condensate
in a speckle potential Aspect . Furthermore, coexistence of localized and
delocalized eigenstates has been predicted also in regular quasiperiodic one-
dimensional systems, first in the discrete Aubri-André AA  model and later in
continuous optical and matter-wave systems Boers ; Modugnho ; LocDelocSoc .
Quasiperiodic (or aperiodic) structures, even those that possess long-range
order, fundamentally differ both from periodic systems where all eigenmodes
are delocalized Bloch waves, and from disordered media where all states are
localized (in one or two dimensions). Upon variation of the parameters of a
quasiperiodic system, it is possible to observe transition between localized
and delocalized states. Such LDT has been observed in one-dimensional
quasiperiodic optical Lahini  and in atomic systems AL_BEC_bichromat ;
Luschen18 .

Wave localization is sensitive to dimensionality of the problem, irrespective
of the type of the medium. Anderson localization and mobility edge in two-
dimensional disordered systems were first reported in the experiment with
bending waves Sheng  and later in optically induced disordered lattices
anderson1 . In quasicrystals localization has been observed only under the
action of nonlinearity Freeman  and in the presence of strong disorder Levi .
Localization and delocalization of light in two-dimensional systems without
any type of disorder and nonlinearity, have never been observed so far.

Here we report on the first experimental realization of reconfigurable
photonic moiré lattices with controllable parameters and symmetry. The
lattices are induced by two superimposed periodic patterns Fleischer ; GauM
(sublattices) with either square or hexagonal primitive cells LDTSR . The
sublattices of each moiré pattern have tunable amplitudes and twist angle.
Depending on the twist angle a photonic moiré lattice may have different
periodic (commensurable) structure or aperiodic (incommensurable) structure
without translational periodicity, but they always feature the same rotational
symmetry as the symmetry of the sublattices. Moiré lattice can also transform
into quasicrystals with higher rotational symmetry (e.g. the case of the
Stampfli pattern Stampfli ). The angles at which a commensurable phase
(periodicity) of an optically-induced moiré lattice is achieved, are
determined by the Pythagorean triples in the case of square sublattices or by
another Diophantine equation, when the primitive cell of the sublattices is
not square (see Methods). For all other rotation angles the structure is
aperiodic albeit regular (i.e., it is not disordered). Importantly, changing
the relative amplitudes of the sublattices allows to smoothly tune the shape
of the lattice without affecting its rotational symmetry. In contrast to
crystalline moiré lattices GrapheneMoire ; Woods14 ; Thermal ;
BandFlatGraphene ; UnconvenSupercond ; Science , optical patterns are
monolayer structures, i.e., both sublattices interfere in one plane. As a
consequence, light propagating in such media is described by a one-component
field [see Eq. (1) below], rather than by the spinors characterizing
electronic states in the tight-binding approximation applicable for material
moiré lattices magic ; flat_hexagon2 .

In the paraxial approximation, the propagation of an extraordinary polarized
light beam in a photorefractive medium with optically induced refractive index
landscape is governed by the Schrödinger equation for the dimensionless signal
field amplitude $\psi({\bf r},z)$ Fleischer ; Efremidis :

$\textit{i}\frac{\partial{\psi}}{\partial\textit{z}}=-\frac{1}{2}\nabla^{2}\psi+\frac{E_{0}}{1+I({\bf
r})}{\psi}.$ (1)

Here $\nabla=(\partial/\partial x,\partial/\partial y)$; ${\bf r}=(x,y)$ is
the radius-vector in the transverse plane, which is scaled to the wavelength
$\lambda=632.8~{}\textrm{nm}$ of the beam used in our experiments; $z$ is the
propagation distance scaled to the characteristic diffraction length $2\pi
n_{\rm e}\lambda$; $n_{\rm e}$ is the refractive index of the homogeneous
crystal for extraordinary-polarized light, $E_{0}>0$ is the dimensionless
applied dc field; $I({\bf r})\equiv\left|p_{1}V({\bf r})+p_{2}V(S{\bf
r})\right|^{2}$ is the moiré lattice induced by two ordinary polarized
mutually coherent periodic sublattices $V({\bf r})$ and $V(S{\bf r})$
interfering in the $(x,y)$ plane and rotated by the angle $\theta$ with
respect to each other (see Methods and Fleischer  for details of the optical
induction technique); $S=S(\theta)$ is the operator of the two-dimensional
rotation; $p_{1}$ and $p_{2}$ are the amplitudes of the first and second
sublattices, respectively. The number of laser beams creating each sublattice
$V({\bf r})$ depends on the desired lattice geometry. The form in which the
lattice intensity $I({\bf r})$ enters Eq. (1) is determined by the mechanism
of photorefractive response.

To visualize the formation of moiré lattices it is convenient to associate a
continuous sublattice $V({\bf r})$ with a discrete one with lattice vectors
determined by the locations of the absolute maxima of $V({\bf r})$. The
resulting moiré pattern inherits the rotational symmetry of $V({\bf r})$. At
specific angles some nodes of different sublattices may coincide thereby
leading to translational symmetry of the moiré patterns $I({\bf
r})\equiv\left|p_{1}V({\bf r})+p_{2}V(S{\bf r})\right|^{2}$. Here we focus on
square and hexagonal sublattices $V({\bf r})$, whose primitive translation
vectors, in commensurable phases, are illustrated by blue arrows,
respectively, in Fig. 1 (the first and third columns) and in Fig. 4 (the first
and second columns). The rotation angles at which the periodicity of $I({\bf
r})$ is achieved are determined by triples of positive integers
$(a,b,c)\in\mathbb{Z}^{+}$ related by a Diophantine equation characteristic
for a given sublattice LDTSR  (see Methods).

Figure 1: (a)-(c) Moiré lattices $\textit{I}\left({\bm{r}}\right)$ generated
by two interfering square sublattices with ${p}_{1}={p}_{2}$, whose axes are
mutually rotated by the angle indicated in each panel. First row: calculated
patterns. Second row: schematic discrete representation of two rotated
sublattices. Third row: experimental patterns observed at the output face of
the crystal. The scale is the same for all experimental images. Comparison of
DOS calculated for moiré lattice (top) and its periodic approximation (bottom)
at $p_{2}=0.1$ (d) and $p_{2}=0.2$ The approximate Pythagorean lattice has the
lattice vectors $b_{1}=...$ (see Supplementary Information), (e). (f) Band
structures for periodic lattice approximating moiré lattice at $p_{2}=0.1$
(top, 7 upper bands are shown) and $p_{2}=0.2$ (bottom, 18 upper bands are
shown). In all cases $p_{1}=1$.

First, we consider a Pythagorean lattice created by two square sublattices.
For the rotation angles $\theta$, such that $\cos\theta=a/c$ and
$\sin\theta=b/c$, where $(a,b,c)$ is a Pythagorean triple, i.e.,
$a^{2}+b^{2}=c^{2}$, $I({\bm{r}})$ a is fully periodic (commensurable) moiré
lattice. Such angle is referred below as Pythagorean. For all other rotation
non-Pythagorean angles $\theta$, the lattice $I({\bm{r}})$ is aperiodic
(incommensurable). Figures 1(a)-(c) compare calculated $I({\bm{r}})$ patterns
with $p_{1}=p_{2}$ (first row) with lattices created experimentally in a
biased SBN:61 photorefractive crystal with dimensions $5\times 5\times 20$ mm3
(third row) for different rotation angles indicated on the panels. The lattice
was created using optical induction technique, invented in Efremidis  and
first realized experimentally in Fleischer . The second row shows the
respective discrete moiré lattices. Columns (a) and (c) show periodic
lattices, while column (b) gives an example of an aperiodic lattice. All
results here and below were obtained for $E_{0}=7$, which corresponds to a
$8\times 10^{4}~{}\text{V/m}$ dc electric field applied to the crystal. The
amplitude of the first sublattice was set to $p_{1}=1$ in all cases, which
corresponds to an average intensity $I_{\rm av}\approx
3.8~{}\text{mW}/\text{cm}^{2}$. For such parameters, the actual refractive
index modulation depth in the moiré lattice is of the order of $\delta n\sim
10^{-4}$.

An important feature of optical moiré lattices that distinguishes them from
their crystalline counterparts, is that rotated sublattices have tunable
amplitudes. The ratio $p_{2}/p_{1}$ can be changed without affecting the
symmetry of the entire pattern, except for the case $p_{1}=p_{2}$, when
rotation symmetry changes, as discussed below. Thus, LDT in optical setting
can be studied either by varying the twist angle $\theta$ at fixed amplitudes
$p_{1,2}$, or by varying one or both amplitudes of the sublattices, at a fixed
angle $\theta$.

From the mathematical point of view, incommensurable lattices are almost
periodic functions Franklin . Like any irrational number can be approached by
a rational one Hurwitz , any non-Pythagorean twist angle can be approached by
a Pythagorean one with any prescribed accuracy (as shown in Supplemental
Information). Thus, any finite area of an incommensurable moiré lattice can be
approached by a primitive effective cell of some periodic Pythagorean lattice,
and more accurate approximation requires larger primitive cell of the
Pythagorean lattice. This property (derived in Supplemental Information) is
illustrated by quantitative similarities between densities of states (DOSs)
calculated for an incommensurable lattice and its Pythagorean approximation
and compared in Fig. 1(d,e) for different $p_{2}/p_{1}$ ratios. A remarkable
property of Pythagorean lattices is extreme flattening of the higher bands
that occurs when $p_{2}/p_{1}$ ratio exceeds certain threshold, as illustrated
in Fig. 1(f). The number of flat bands rapidly grows with the size of the area
of the primitive cell of the approximating Pythagorean lattice. Thus, an
incommensurable moiré lattice can be viewed as the large-area limit of
periodic Pythagorean lattices with extremely flat higher bands. Since flat
bands support quasi-nondiffracting localized modes, the initially localized
beam launched into such moiré lattice will remain localized. This flat-band
physics of moiré lattices, earlier discussed for twisted bilayer graphene
magic ; flat_hexagon1 ; flat_hexagon2 , allows us to predict beam localization
above some threshold $p_{2}/p_{1}$ value, whose physics is different from that
of Anderson localization in random media. Furthermore, flat bands support
states, which are exponentially localized in the primitive cell. Such states
can be well approximated by exponentially localized two-dimensional Wannier
functions Wannier  (see Fig. 2(c) and Supplementary Information).

To elucidate the impact of the sublattice amplitudes and rotation angle
$\theta$ on the light beam localization, we calculated the linear eigenmodes
$\psi({\bm{r}},z)=w({\bm{r}})e^{i\beta z}$, where $\beta$ is the propagation
constant and $w({\bm{r}})$ is the mode profile, supported by the moiré
lattices. To characterize the localization of the eigenmodes we use the
integral form-factor $\chi$ defined by
$\chi^{2}=U^{-2}\iint|\psi|^{4}d^{2}{\bm{r}}$, where
$U=\iint|\psi|^{2}d^{2}{\bm{r}}$ is the energy flow (the integration is over
the transverse area of the crystal). Form-factor is inversely proportional to
the mode width: the larger $\chi$ the stronger the localization. The
dependence of the form-factor of the most localized mode of the structure
(mode with largest $\beta$) on $\theta$ and $p_{2}$ is shown in Fig. 2(a) (for
modes with lower $\beta$ values these dependencies are qualitatively similar).
One observes a sharp LDT above a certain threshold depth $p_{2}^{\rm LDT}$ of
the second sublattice, at fixed amplitude of the first sublattice: $p_{1}=1$.
This corroborates with strong band flattening of approximating Pythagorean
lattice at $p_{2}>p_{2}^{\rm LDT}$ depicted in Fig. 1(f). Below $p_{2}^{\rm
LDT}$ all modes are found to be extended [Fig. 2(b)] and above the threshold,
some modes are localized [Fig. 2(c)]. Inset in Fig. 2(c) reveals exponential
tails for $p_{2}>p_{2}^{\rm LDT}$ from which localization length for most
localized mode can be extracted. In Fig. 2(a) one also observes sharp
delocalization for angles $\theta$ determined by the Pythagorean triples, when
all modes are extended regardless of $p_{2}$ value.

We observe in Fig. 2(a) that $p_{2}^{\rm LDT}$ is practically independent of
the rotation angle. This is explained by the fact that a large fraction of the
power in a localized mode resides in the vicinity of a lattice maximum (i.e.,
at ${\bm{r}}={\bf 0}$). In an incommensurable phase $I({\bm{r}})<I({\bm{0}})$
for all ${\bm{r}}\neq{\bm{0}}$ and the optical potential locally can be
approximated by the Taylor expansion of $E_{0}/[1+I({\bm{r}})]$ with respect
to ${\bm{r}}$ near the origin. Such expansion includes the rotation angle
$\theta$ only in the fourth order (see Supplementary Information). Thus,
locally, the optical potential can be viewed as almost isotropic.

Figure 2: (a) Form-factor (inverse width) of the eigenmodes with largest
$\beta$ versus rotation angle $\theta$ and versus amplitude of the second
sublattice $p_{2}$ at $p_{1}=1$. The horizontal dashed line indicates the
sublattice depth ${p}_{2}^{\textrm{LDT}}$ at which LDT occurs. The vertical
dashed line shows one of the Pythagorean angles
$\theta_{\textrm{p}}=\text{arctan(3/4)}$. Examples of mode profiles with
largest $\beta$ for $p_{2}<{p}_{2}^{\textrm{LDT}}$ (b) and
$p_{2}>{p}_{2}^{\textrm{LDT}}$ (c). Insets show cuts of
$\textrm{ln}|\psi|^{2}$ distribution along the $x$ and $y$ axes.

To study the guiding properties of the Pythagorean moiré lattices
experimentally and to expose the two-dimensional LDT, we measured the
diffraction outputs for beams propagating in lattices corresponding to
different rotation angles $\theta$ for fixed input position of the beam,
centered or off-center. The diameter of the Gaussian beam focused on the input
face of the crystal was about $23~{}\mu\textrm{m}$. Such a beam covers
approximately one bright spot (channel) of the lattice profile. The intensity
of the input beam was about $10$ times lower than the intensity of the
lattice-creating beam, $I_{\rm av}$, to guarantee that the beam does not
distort the induced refractive index and that it propagates in the linear
regime. When a beam enters moiré lattice it either diffracts, if there are no
localized modes, or it remains localized if such modes exist (notice that no
averaging is used, because lattice is deterministic). Extended Data Fig. 2
compares experimental and theoretical results for propagation dynamics in
three different regimes at fixed $p_{1}=1$. In the incommensurable lattice at
small $p_{2}<p_{2}^{\rm LDT}$ one observes beam broadening and no traces of
localization (top row). Localization takes place at sufficiently high
amplitude $p_{2}>p_{2}^{\rm LDT}$ of the second sublattice in the
incommensurable case, e.g., for $p_{2}=1$ (middle row). However, at a twist
angle $\theta$ corresponding to a commensurable moiré lattice, localization
does not occur even for $p_{2}=p_{1}=1$ due to the restoration of lattice
periodicity (bottom row). Simulations of propagation up to much larger
distances beyond the available sample length presented in Extended Data Fig. 3
fully confirm localization of the beam in incommensurable lattice at any
distance at $p_{2}>p_{2}^{\rm LDT}$ and its expansion at $p_{2}<p_{2}^{\rm
LDT}$.

Figure 3: Observed output intensity distributions illustrating LDT with
increasing amplitude $p_{2}$ of the second sublattice for rotation angle
$\theta=\arctan 3^{-1/2}=\pi/6$ (left and right columns) and absence of LDT
for the Pythagorean angle $\theta=\arctan(3/4)$ (central column). Insets show
initial excitation position, central for the left and central columns, and
off-center for the right column.

Extended experimental evidence of LDT in the two-dimensional lattice is
presented in Fig. 3, where we compare output patterns for the low-power light
beam in the incommensurable ($\tan\theta=3^{-1/2}$, left and right columns for
central and off-center excitations, respectively) and commensurable
($\tan\theta=3/4$, central column) moiré lattices, tuning in parallel the
amplitude $p_{2}$ of the second sublattice. When
$p_{2}<{p}_{2}^{\textrm{LDT}}$ (in Fig. 3 ${p}_{2}^{\textrm{LDT}}\approx
0.15$), the light beam in the incommensurable lattice notably diffracts upon
propagation and expands across multiple local maxima of $I({\bm{r}})$ in the
vicinity of the excitation point. However, when $p_{2}$ exceeds the LDT
threshold, it is readily visible that diffraction is arrested for both central
(left column) and off-center (right column) excitations and a localized spot
is observed at the output over a large range of $p_{2}$ values. In clear
contrast, localization is absent for any $p_{2}$ value in the periodic lattice
associated with the Pythagorean triple (central column).

Figure 4: First row: moiré lattices produced by interference of two hexagonal
patterns rotated by the angle $\theta$: $p_{2}=1$ in the first, second, and
fourth columns, while in the third column $p_{2}=0.18$. Second row: schematic
discrete representation of two rotated hexagonal sublattices. Third row:
measured output intensity distributions for signal beam at the output face of
the crystal. In all cases $p_{1}=1$.

The mutual rotation of two identical sublattices allows generation of
commensurable and incommensurable moiré patterns with sublattices of any
allowed symmetry. To illustrate the universality of LDT, we induced hexagonal
moiré lattices (the technique of induction is similar to that used for single
hexagonal photonic lattices photon-graphen ). For such lattices, the rotation
angles producing commensurable patterns are given by the relation
$\tan\theta=b\sqrt{3}/(2a+b)$, where the integers $a$ and $b$ solve the
Diophantine equation $a^{2}+b^{2}+ab=c^{2}$. Two examples are presented in the
first and second columns of Fig. 4. In such periodic structures, the signal
light beam experiences considerable diffraction for any amplitude of the
sublattices, shown in the bottom row. To observe LDT one has to induce
aperiodic structures. To such end, we considered the rotation angle of
$30^{o}$. In such incommensurable case we did observe LDT by increasing the
amplitude of the second sublattice, keeping the amplitude $p_{1}$ fixed.
Delocalized and localized output beams are shown in the lower panels of the
third and fourth columns of Fig. 4. In the third column the ideal 6-fold
rotation symmetry of the output pattern is slightly distorted, presumably due
to the intrinsic anisotropy of the photorefractive response. At $p_{1}=p_{2}$
the moiré pattern acquires the 12-fold rotational symmetry (shown in the
fourth column of Fig. 4) as it was proposed in Stampfli  as a model of a
quasicrystal and is similar to the twisted bilayer graphene quasicrystal
reported in Science .

In closing, we have created first photonic moiré lattices by imprinting two
mutually rotated square and hexagonal sublattices into the photorefractive
crystal. These structures with highly controllable parameters and symmetry
have enabled systematic investigation of evolution and localization of
wavepackets. Namely, we report the first experimental demonstration of two-
dimensional LDT of a wavepacket in an optical setting. Our findings uncover
novel mechanism for wave localization in regular incommensurable systems based
on the physics of flat-band structures. LDT can occur according to different
scenarios: via commensurable-incommensurable transition and via tuning of
relative depths of optically-induced lattices. In addition to new
possibilities for control of light by light, i.e. control of propagation
paths, symmetry of diffraction patterns, diffraction rate of light beams, and
formation of self-sustained excitations with new symmetries, photonic moiré
patterns allows to study phenomena relevant to other areas of physics,
particularly from condensed matter systems, which are much harder to explore
directly. Such lattices can be used to study the relations between
conductivity/transport and symmetry of incommensurable patterns with long-
range order. Being tunable, optical moiré patterns can be created in
practically any arbitrary configurations consistent with two-dimensional
symmetry groups, thus allowing the exploration of potentials that may not be
easily produced in tunable form using material moiré structures. The setting
reported here is also reproducible for atomic potentials, also created by the
standing beams using similar geometries, and hence the effects observed here
can be obtained in atomic systems, including BECs, both linear and nonlinear
ones. They can be also induced in absorbing resonant atomic media, such as hot
vapors. Furthermore, while most of previous studies of moiré lattices were
focused on graphene structures and quasicrystals, nowadays widely used in
material science, our results suggest that a somewhat simpler twisted square
lattice is a promising alternative geometry for creation of synthetic
materials allowing investigation of two-dimensional physics, including physics
of localization and physics of flat-band materials.

## References

  * (1) R. Decker, Y. Wang, V. W. Brar, W. Regan, H.-Z. Tsai, Q. Wu, W. Gannett, A. Zettl, and M. F. Crommie, Local Electronic Properties of Graphene on a BN Substrate via Scanning Tunneling Microscopy, Nano Lett. 11, 2291-2295 (2011).
  * (2) C. R. Woods, L. Britnell, A. Eckmann, R. S. Ma, J. C. Lu, H. M. Guo, X. Lin, G. L. Yu, Y. Cao, R. V. Gorbachev, A. V. Kretinin, J. Park, L. A. Ponomarenko, M. I. Katsnelson, Yu. N. Gornostyrev, K. Watanabe, T. Taniguchi, C. Casiraghi, H-J. Gao, A. K. Geim and K. S. Novoselov, Commensurate–incommensurate transition in graphene on hexagonal boron nitride, Nat. Phys. 10, 451 (2014).
  * (3) D. Wang, G. Chen, C. Li, M. Cheng, W. Yang, S. Wu, G. Xie, J. Zhang, J. Zhao, X. Lu, P. Chen, G. Wang, J. Meng, J. Tang, R. Yang, C. He, D. Liu, D. Shi, K. Watanabe, T. Taniguchi, Ji Feng, Y. Zhang, and G. Zhang, Thermally Induced Graphene Rotation on Hexagonal Boron Nitride Duoming, Phys. Rev. Lett. 116, 126101 (2016).
  * (4) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Correlated insulator behaviour at half-filling in magic-angle graphene superlattices, Nature 556, 80 (2018).
  * (5) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Unconventional superconductivity in magic-angle graphene superlattices, Nature 556, 43 (2018).
  * (6) S. J. Ahn, P. Moon, T.-H. Kim, H.-W. Kim, H.-C. Shin, E. H. Kim, H. W. Cha, S.-J. Kahng, P. Kim, M. Koshino, Y.-W. Son, C.-W. Yang, and J. R. Ahn, Dirac electrons in a dodecagonal graphene quasicrystal. Science 786, 782-786 (2018).
  * (7) A. H. MacDonald, Bilayer Graphene’s Wicked, Twisted Road, Physics,12, 12 (2019)
  * (8) R. Bistritzer and A. H. MacDonald, Moiré bands in twisted double-layer graphene, PNAS 108, 12233-12237 (2011).
  * (9) G. Tarnopolsky, A. J. Kruchkov, and A. Vishwanath, Origin of magic angles in twisted bilayer graphene, Phys. Rev. Lett. 122, 106405 (2019).
  * (10) C. R. Dean, L. Wang, P. Maher, C. Forsythe, F. Ghahari, Y. Gao, J. Katoch, M. Ishigami, P. Moon, M. Koshino, T. Taniguchi, K. Watanabe, K. L. Shepard, J. Hone, and P. Kim, Hofstadter’s butterfly and the fractal quantum Hall effect in moiré superlattices, Nature 497, 598-602 (2013).
  * (11) P. San-Jose, J. González, and F. Guinea, Non-Abelian Gauge Potentials in Graphene Bilayers, Phys. Rev. Lett. 108, 216802 (2012).
  * (12) P. Stampfli, A dodecagonal quasiperiodic lattice in two dimensional, Helv. Phys. Acta 59, 1260 - 1263 (1986).
  * (13) L. Ye, G. Cody, M. Zhou, and P. Sheng, Observation of Bending Wave Localization and Quasi Mobility Edge in Two Dimensions, Phys. Rev. Lett. 69, 3080-3083 (1992).
  * (14) B. Freedman, G. Bartal, G; M. Segev, R. Lifshitz, D. N. Christodoulides, and J. W. Fleischer, Wave and defect dynamics in nonlinear photonic quasicrystals, Nature 440, 1166 (2006).
  * (15) L. Levi, M. Rechtsman, R. Freedman, T. Schwartz, O. Manela, and M. Segev, Disorder-enhanced transport in photonic quasicrystals, Science 332, 1541 (2011).
  * (16) E. Abrahams, P. W. Anderson, D. C. Licciardello, and T. V. Ramakrishnan, Scaling theory of localization: absence of quantum diffusion in two dimensions, Phys. Rev. Lett. 42, 673–676 (1979).
  * (17) T. Brandes, and S. Kettemann, The Anderson transition and its Ramifications-localization, quantum interference, and interactions (Springer Verlag, Berlin, 2003).
  * (18) N. Mott, The mobility edge since 1967, J. Phys. C: Solid State Phys. 20, 3075-3102 (1987).
  * (19) J. Billy, V. Josse, Z. Zuo, A. Bernard, B. Hambrecht, P. Lugan, D. Clément, L. Sanchez-Palencia1, P. Bouyer, and A. Aspect, Direct observation of Anderson localization of matter waves in a controlled disorder, Nature 453, 891–894 (2008).
  * (20) S. Aubry and G. André, Analyticity breaking and Anderson localization in incommensurate lattices, Ann. Israel Phys. Soc. 3, 133 (1980).
  * (21) D. J. Boers, B. Goedeke, D. Hinrichs, and M. Holthaus, Mobility edges in bichromatic optical lattices, Phys. Rev. A 75, 063404 (2007).
  * (22) M. Modugno, Exponential localization in one-dimensional quasi-periodic optical lattices, New J. Phys. 11, 033023 (2009).
  * (23) C. Li, F. Ye, Y. V. Kartashov, V. V. Konotop, and X. Chen, Localization-delocalization transition in spin-orbit-coupled Bose-Einstein condensate, Sci. Rep. 6, 31700 (2016).
  * (24) Y. Lahini, R. Pugatch, F. Pozzi, M. Sorel, R. Morandotti, N. Davidson, and Y. Silberberg, Observation of a Localization Transition in Quasiperiodic Photonic Lattices, Phys. Rev. Lett. 103, 013901 (2009).
  * (25) G. Roati, C. D’Errico, L. Fallani, M. Fattori, C. Fort, M. Zaccanti, G. Modugno, M. Modugno, and M. Inguscio, Anderson localization of a non-interacting Bose-Einstein condensate, Nature 453, 895 (2008).
  * (26) H. P. Lüschen, S. Scherg, Th. Kohlert, M. Schreiber, P. Bordia, X. Li, S. Das Sarma, and I. Bloch, Single-Particle Mobility Edge in a One-Dimensional Quasiperiodic Optical Lattice, Phys. Rev. Lett. 120, 160404 (2018).
  * (27) L. Ye, , G. Cody, M. Zhou, P. Sheng, and A. N. Norris, Observation of bending wave localization and quasi mobility edge in two dimensions. Phys. Rev. Lett. 69, 3080–3083 (1992).
  * (28) T. Schwartz, G. Bartal, S. Fishman and M. Segev, Transport and Anderson localization in disordered two-dimensional photonic lattices, Nature 446, 52-55 (2007).
  * (29) R. C. Gauthier and K. W. Mnaymneh, Design of photonic band gap structures through a dual-beam multiple exposure technique, Opt. and Laser Techn. 36, 625 (2004).
  * (30) J. Fleischer, M. Segev, N. Efremidis, and D. Christodoulides, Observation of two-dimensional discrete solitons in optically induced nonlinear photonic lattices, Nature 442, 147 (2003).
  * (31) C. Huang, F. Ye, X. Chen, Y. Kartashov, V. V. Konotop, and L. Torner, Localization delocalization wavepacket transition in Pythagorean aperiodic potentials, Sci. Rep. 6, 32546 (2016).
  * (32) N. K. Efremidis, S. Sears, D, N. Christodoulides, J. W. Fleischer, and M. Segev, Discrete solitons in photorefractive optically induced photonic lattices, Phys. Rev. E 66, 046602 (2002).
  * (33) O. Peleg, G. Bartal, B. Freedman, O. Manela, M. Segev, and D. N. Christodoulides, Conical Diffraction and Gap Solitons in Honeycomb Photonic Lattices, Phys. Rev. Lett. 98, 103901 (2007).
  * (34) E. Suárez Morell, J. D. Correa, P. Vargas, M. Pacheco, & Z. Barticevic, Flat bands in slightly twisted bilayer graphene: Tight-binding calculations. Phys. Rev. B, 82, 121407 (2010).
  * (35) P. Franklin, The Elementary Theory of Almost Periodic Functions of Two Variables. J. Math. Phys., 5, 40–54 (1926).
  * (36) G. H. Hardy & E. M. Wright. An Introduction to the Theory of Numbers. 5th ed. Oxford: Clarendon Press, 1984.
  * (37) C. Brouder, G. Panati, M. Calandra, C. Mourougane, and N. Marzar, Exponential Localization of Wannier Functions in Insulators Christian, Phys. Rev. Lett. 98, 046402 (2007).

### Acknowledgments

P. W and F. Y acknowledge support from the NSFC (No. 61475101 and No.
11690033). Y.V.K. and L.T. acknowledge support from the Severo Ochoa
Excellence Programme (SEV-2015-0522), Fundacio Privada Cellex, Fundacio
Privada Mir-Puig, and CERCA/Generalitat de Catalunya.

### Author contributions

P. W and Y. Z contributed equally to this work. All authors contributed
significantly to this work.

### Competing interests

The authors declare no competing interests.

## METHODS

### Experimental setup.

The experimental setup is illustrated in Extended Data Fig. 1. A cw frequency-
doubled Nd:YAG laser at wavelength $\lambda=532~{}\text{nm}$ is divided by a
polarizing beam splitter into two polarization components, which are sent to
Path a and Path b separately. Light in Path a is extraordinarily polarized and
it is used to image the induced potential in the photorefractive crystal (see
the third row of Fig. 1 in the main text). Light in Path b is ordinarily
polarized and it is used to write the desirable potential landscape in the
photorefractive SBN:61 crystal with dimensions $5\times 5\times
20~{}\text{mm}^{3}$ and extraordinary refractive index $n_{\rm e}=2.2817$.
Before entering the crystal ordinarily polarized light beam in Path b is
modulated by Masks 1 and 2 transforming this beam into superposition of two
rotated periodic patterns. Their relative strength $p_{2}/p_{1}$, or, more
precisely, the strength of the second lattice, as well as the twist angle
$\theta$ are controlled by the polarizer-based Mask 1 and amplitude Mask 2.
The He-Ne laser with wavelength $\lambda=633~{}\text{nm}$ shown in path c
provides extraordinarily polarized beam focused onto the front facet of the
crystal, that serves as a probe beam for studying light propagation in the
induced potential. We record the output light intensity pattern by a CCD at
the exit facet of the crystal after propagation distance of $20~{}\text{mm}$.

Characteristics of moiré lattices used in experiment. In the experiments there
have been used two types of moiré lattices, whose characteristics are
summarized in Extended Data Table 1. In all the cases the center of rotation
in the $(x,y)$ plane was chosen coincident with a node of one of the
sublattices.

## DATA AVAILABILITY

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## EXTENDED DATA

Extended Data Fig. 1: Experimental setup. $\lambda/2$, half-wave plate; PBS,
polarizing beam splitter; SF, spatial filter; L, lens; BS, beam splitter; ID,
iris diaphragm; M, mirror; P, Polarizer; SBN, strontium barium niobate
crystal; CCD, charged-coupled device. Mask 2 is an amplitude mask to produce
two group of sub-lattices with a rotation angle $\theta$, and Mask 1 is made
of a polarizer film. Extended Data Fig. 2: Experimentally observed intensity
distributions of the probe beam (color-surface plots) and corresponding
theoretically calculated distributions (insets), at different propagation
distances $z$, for $\tan\theta=3^{-1/2}$, $p_{2}=0.1$, which falls below the
LDT point (top row), $\tan\theta=3^{-1/2}$, $p_{2}=1$, which falls above LDT
point (middle row), and $\tan\theta=3/4$, $p_{2}=1$ (bottom row). The first
two rows correspond to the incommensurable Pythagorean lattice shown in the
central column of Fig. 1 of the main text. The third row corresponds to the
commensurable lattice shown in the last column of Fig. 1 of the main text.

Extended Data Fig. 3: (a),(b) Numerical simulations of the light beam propagation in the incommensurable moiré lattice for central excitation, corresponding to top and middle rows of Extended Data Fig. 2, but for larger distances, notably exceeding sample length. (c),(d) Similar numerical results, but for off-center excitation position in moiré lattice. Parameter $p_{2}=0.1$ for (a),(c) and $p_{2}=1.0$ for (b),(d), while rotation angle $\theta=\pi/6$. In all cases Gaussian beam exciting a single site of the potential is assumed. Moiré lattice $I({\bm{r}})$ | Sublattice $V({\bm{r}})$ | Diophantine equation | $\tan\theta$
---|---|---|---
Pythagorean | $\cos(2x)+\cos(2y)$ | $a^{2}+b^{2}=c^{2}$ | $b/a$
hexagonal | $\sum_{n=1}^{3}\cos\left[2(x\cos\theta_{n}+y\sin\theta_{n})\right]$ | $a^{2}+b^{2}+ab=c^{2}$ | $\sqrt{3}b/(2a+b)$
Extended Data Tab. 1: Characteristics of the moiré lattices used in
experiments. For hexagonal lattices $\theta_{1}=0$, $\theta_{2}=2\pi/3$, and
$\theta_{3}=4\pi/3$.


# SAC3: Reliable Hallucination Detection in Black-Box Language Models via
Semantic-aware Cross-check Consistency

Jiaxin Zhang1, Zhuohang Li2, Kamalika Das1, Bradley Malin2,3, Sricharan Kumar1
1Intuit AI Research 2Vanderbilt University 3Vanderbilt University Medical
Center
{jiaxin_zhang, kamalika_das<EMAIL_ADDRESS>
{zhuohang.li<EMAIL_ADDRESS>
###### Abstract

Hallucination detection is a critical step toward understanding the
trustworthiness of modern language models (LMs). To achieve this goal, we re-
examine existing detection approaches based on the self-consistency of LMs and
uncover two types of hallucinations resulting from 1) question-level and 2)
model-level, which cannot be effectively identified through self-consistency
check alone. Building upon this discovery, we propose a novel sampling-based
method, i.e., semantic-aware cross-check consistency (SAC3) that expands on
the principle of self-consistency checking. Our SAC3 approach incorporates
additional mechanisms to detect both question-level and model-level
hallucinations by leveraging advances including semantically equivalent
question perturbation and cross-model response consistency checking. Through
extensive and systematic empirical analysis, we demonstrate that SAC3
outperforms the state of the art in detecting both non-factual and factual
statements across multiple question-answering and open-domain generation
benchmarks.111All resources are available at https://github.com/intuit/sac3.

##  1 Introduction

Large-scale pre-trained language models (LMs) have demonstrated exceptional
adaptability across a diverse array of natural language tasks that require
generating open-ended responses based on user prompt comprehension Zhao et al.
(2023). However, prominent LMs like GPT Brown et al. (2020) and PaLM Chowdhery
et al. (2022), often exhibit a tendency to produce exceedingly confident, yet
erroneous, assertions commonly referred to as hallucinations. This phenomenon
significantly impedes their applicability in domains where factual accuracy is
of utmost importance.

Hallucinations can be detected through the assistance of metrics that capture
the uncertainty about the output sequences. However, these metrics require
access to token-level log probabilities, which are not available in commercial
black-box LMs like ChatGPT or Bard that only offer limited API access. To
overcome this limitation, recent studies explore sampling-based approaches for
approximating uncertainty estimation Lin et al. (2023) through establishing a
connection between confidence and self-consistency Manakul et al. (2023);
Mündler et al. (2023). The underlying premise of this principle is that LMs
are more inclined to generate consistent responses when high probabilities are
assigned to tokens in the answer, which, in turn, implies a level of
factuality. In contrast, inconsistent responses are more likely to contain
hallucinations. To operationalize this concept, current approaches are
designed to sample multiple responses from the LMs for a given question and
then compose a hallucination score for each sentence based on the level of
response consistency.

Figure 1: Key observation: solely checking the self-consistency of LLMs is not
sufficient for deciding factuality. Left: generated responses to the same
question may be consistent but non-factual. Right: generated responses may be
inconsistent with the original answer that is factually correct.

##### Key Observations.

In this work, we investigate the relationship between the self-consistency of
LMs and the occurrence of hallucinations in a diverse range of tasks. Our
investigation indicates that while self-inconsistency in LMs often coincides
with hallucination, self-consistency does not necessarily guarantee factual
answers, as shown in Figure 1. Our findings challenge the notion that self-
consistency alone can serve as a reliable indicator of veracity, as it is
demonstrated that LMs can exhibit various tiers of hallucination that elude
detection through self-consistency checks. One such tier is question-level
hallucination, where LMs consistently generate incorrect answers in response
to specific questions (e.g., Table 1). We reveal that by reformulating the
questions, it is possible to mitigate such instances of hallucinations.
Additionally, our work further reveals the existence of model-level
hallucinations, whereby different LMs show discrepancies in their propensity
for hallucination. Surprisingly, we even observe cases where smaller LMs are
capable of correctly answering questions for which larger LMs hallucinate.
Together, these findings accentuate the need to consider model-specific
characteristics when assessing the occurrence of hallucinations.

##### Proposed Approach.

Motivated by these observations, we introduce SAC3, a new sampling-based
approach utilizing semantic-aware cross-check consistency to improve the
detection of hallucinations in black-box LMs. An overview of our approach is
provided in Fig. 2. To address question-level hallucination, we introduce a
mechanism that perturbs semantically equivalent questions to evaluate the
consistency of LMs’ responses across variants of the same question. By
examining the generated answers to these perturbed questions, we are able to
identify cases where the LM consistently provides incorrect responses to a
specific question, which is indicative of a question-level hallucination.
Furthermore, we address model-level hallucination by introducing cross-model
response consistency checking, which involves comparing the responses of
different LMs to the same set of questions. By identifying discrepancies
between the responses of different models, we are able to pinpoint cases where
certain models exhibit hallucinations while others provide correct answers.
Integrating these cross-checking extensions into our approach significantly
improves its ability to detect hallucinations that go beyond self-consistency,
thereby providing a more comprehensive assessment of the presence of question-
level and model-level hallucinations.

##### Effectiveness of SAC3.

We demonstrate the effectiveness of SAC3 in multiple classification and open-
domain generation QA datasets. In the classification QA tasks, our approach
achieves a high AUROC score of 99.4% and 97.0% respectively, which
significantly outperforms the self-consistency baseline. In the case of open-
domain generation QA tasks, our approach achieves an AUROC score of 88.0% and
77.2%, which substantially improves upon the self-consistency baseline by a
large margin (+13.8%, and +6.7% respectively).

Method | Input | Model | Output |  Consistency | Factuality
---|---|---|---|---|---
Self- consistency | Q: ‘‘Is 3691 a prime number?’’ | Target |  ‘‘No, 3691 is not a prime number as it is divisible by 7 and 13.’’ | ✓ | ✗
Q: ‘‘Is 3691 a prime number?’’ | Target |  ‘‘No, 3691 is not a prime number as it is divisible by 7 and 13.’’
Q: ‘‘Is 3691 a prime number?’’ | Target |  ‘‘No, 3691 is not a prime number. It can be divided evenly by 7 and 19.’’
Cross-Q consistency | Q1: ‘‘Is the number 3691 prime?’’ | Target | ‘‘Yes, the number 3691 is prime.’’ | ✗ | ✓
Q2: ‘‘Are the only factors of 3691 itself and one?’’ | Target |  ‘‘Yes, the only factors of 3691 are itself and one.’’
Q3: ‘‘Can 3691 only be divided by 1 and 3691?’’ | Target |  ‘‘Yes, 3691 is a prime number and can only be divided by 1 and 3691.’’
Cross-M consistency | Q: ‘‘Is 3691 a prime number?’’ | Verifier |  ‘‘Yes, 3691 is a prime number.’’ | ✗ | ✓
Cross-M & Q consistency | Q1: ‘‘Is the number 3691 prime?’’ | Verifier | ‘‘Yes.’’ |  ✗ |  ✓
Q2: ‘‘Are the only factors of 3691 itself and one?’’ | Verifier | ‘‘Yes, the only factors of 3691 are 1 and itself.’’
Q3: ‘‘Can 3691 only be divided by 1 and 3691?’’ | Verifier | ‘‘Yes, 3691 can only be divided by 1 and 3691.’’

Table 1: An illustrative example of self-consistency, cross-question
consistency, and cross-model consistency check. The original question and
answer are ‘‘Is 3691 a prime number?’’ and ‘‘No, 3691 is not a prime number.
It is divisible by 7 and 13’’, respectively. Each row presents a set of
sampled QA pairs along with its consistency regarding the original answer, and
the predicted factuality of the original answer.

##  2 Related Work

##### Hallucination in LMs.

The issue of hallucination in language models (LMs) has gained significant
attention due to its negative impact on performance and the risks it
introduces in various natural language processing (NLP) tasks, such as machine
translation Zhou et al. (2020), summarization Cao et al. (2022), dialogue
generation Das et al. (2023), and question answering Zhang et al. (2023a);
Zheng et al. (2023b); Dhuliawala et al. (2023). Recent survey Ji et al.
(2023); Zhang et al. (2023c); Ye et al. (2023) and evaluation benchmarks Liu
et al. (2021); Li et al. (2023a); Yang et al. (2023) have highlighted the
importance of addressing this issue. Previous research has explored
hallucination evaluation using confidence-based approaches Xiao and Wang
(2021); Varshney et al. (2023); Chen and Mueller (2023) that require access to
token-level log probability Kuhn et al. (2023); Cole et al. (2023) or
supervised tuning Agrawal et al. (2023); Li et al. (2023b) that relies on
internal states of the LM. However, these methods may not be applicable when
only API access to the LM is available Agrawal et al. (2023). Another approach
involves retrieving knowledge from external databases to tackle hallucinations
Ji et al. (2022); Zheng et al. (2023a); Peng et al. (2023); Zhang et al.
(2023b). In contrast to these studies, our work focuses on detecting
hallucinations in open-domain QA tasks using black-box LMs, without relying on
external resources.

##### Consistency Evaluation of LMs.

An essential characteristic of logically valid intelligent systems is self-
consistency, which entails that no two statements provided by the system
contradict each other. Self-consistency is defined by Elazar et al. (2021) as
the invariance of an LM’s responses across various types of semantics-
preserving prompt transformations. This definition is further enriched by
multiple other consistency categories proposed by Jang et al. (2022). Wang et
al. (2022) demonstrates that self-consistency can significantly enhance the
chain of thought reasoning in LMs. Without self-consistency, it becomes
challenging to regard LMs as reliable or trustworthy systems. Recent studies
employ self-consistency to detect hallucinations based on pretrained LMs
Manakul et al. (2023) and instruction-tuned LMs Mündler et al. (2023).
Although these methods exhibit promising accuracy on several specific tasks,
potential failures Chen et al. (2023) of self-consistency are overlooked in
the current settings, as existing LMs frequently provide inconsistent
responses to questions Mitchell et al. (2022) and factual knowledge inquiries
Elazar et al. (2021); Tam et al. (2023); Gekhman et al. (2023). Our work
addresses these concerns by introducing a cross-check consistency approach,
aiming to bridge the gap between self-consistency and factual assessment.

##  3 Self-consistency Limitations in Factuality Assessment

The essential assumption of self-consistency in factuality assessment is that
if the LM has the knowledge of the concept, responses sampled from its output
distribution, should be similar and consistent; conversely, if the LM lacks
corresponding knowledge, the sampled responses would contain hallucinated
facts that are diverged and contradictory. Although this assumption may seem
reasonable, it does not always hold in practice (more details are provided in
the Appendix A). Specifically, we argue that solely checking the LM’s self-
consistency is insufficient for detecting hallucination or verifying
factuality under the following two circumstances:

1\. LMs may produce consistently hallucinated facts. We observe that for
certain questions, LMs may output consistently wrong answers. For instance, as
shown in Fig. 1, when prompted with the question ‘‘Is pi smaller than 3.2?’’,
ChatGPT consistently generates incorrect answers. In this case, where the
generated responses are consistent but non-factual, solely relying on self-
consistency checking of a single model would yield false negative
hallucination detection results.

2\. Even in cases when LMs generate factual statements in their original
response, the stochastic sampled responses may lack veracity. For example, the
original answer (Answer 1) of ChatGPT under zero temperature is correct
regarding the senator search question as shown in Fig. 1. However, when
sampled with a higher temperature, ChatGPT generates multiple incorrect
responses (Answer 2 and Answer $m$). In this scenario, where the sampled
responses are inconsistent and disagree with the original response which
itself is factually correct, methods that rely solely on model self-checking
would produce false positives.

In summary, although the inconsistency of sampled responses has been
empirically demonstrated to be correlated with hallucinated facts on certain
tasks, in general, self-consistency is neither necessary nor sufficient to
verify the veracity of large LMs’ statements. Therefore, methods based solely
on self-consistency checking may not be able to accurately detect
hallucinations in complex QA and open-domain generation tasks, which motivates
us to design a more reliable and robust factuality assessment method that
extends this idea.

Figure 2: Overview of the proposed semantic-aware cross-check consistency
(SAC3) method.

##  4 SAC3 : Semantic-Aware Cross-check Consistency

This section describes the proposed semantic-aware cross-check consistency
approach, a high-level overview of which is provided in Fig. 1. Additionally,
an illustrative example of each component is presented in Table 1. Here, we
walk through each component in detail.

###  4.1 Stage 1: Question-level Cross-checking via Semantically Equivalent
Perturbations

Contrary to existing techniques that assess semantic equivalence through
entailment or paraphrasing, our approach involves rephrasing the input query
by generating alternative inputs that preserve semantic equivalence, i.e.,
semantically equivalent input perturbation. To achieve this, we leverage
advances in LLM prompting. Starting with a queried input $\boldsymbol{x}_{0}$,
we acquire a set of $k$ semantically equivalent inputs
$\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{k}\\}$ through
the prompt: ‘‘For the question [QUERIED QUESTION], provide $k$ semantically
equivalent questions’’.

To ensure the quality of the generated inputs in this step, we further double-
check the semantic equivalence between the generated inputs
$\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{k}\\}$ and the
queried input $\boldsymbol{x}_{0}$ in a pair-wise manner using the prompt
‘‘Are the following two inputs semantically equivalent? [QUERIED INPUT]
[GENERATED INPUT]’’ and filtering out the inputs that do not share the same
semantic meaning as the original input. The complete prompt templates used in
this work are provided in the Appendix B.

###  4.2 Stage 2: Model-level Cross-check with Additional Verifier LM

Let $\boldsymbol{s}_{0}$ denote the original response from a target LM
$\mathcal{T}$ based on a given query $\boldsymbol{x}_{0}$. Our objective is to
detect whether $\boldsymbol{s}_{0}$ is hallucinated by sampling responses from
the predictive distribution of $\mathcal{T}$. To avoid model-level
hallucination, we introduce an additional verifier LM denoted as $\mathcal{V}$
for model-level cross-checking. We define the responses from both models as:

$\boldsymbol{s}_{\mathcal{T}_{j}}=\mathcal{T}(\boldsymbol{x}_{j}),\
\boldsymbol{s}_{\mathcal{V}_{j}}=\mathcal{V}(\boldsymbol{x}_{j}),\ j=1,...,k,$
(1)

where $k$ is the length of the generated semantically equivalent inputs
$\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{k}\\}$ in stage
1. To assess the factuality of $\boldsymbol{x}_{0}$, the self-checking
mechanism operates by drawing a set of $n_{s}$ stochastic response samples
from the target LM:
$\mathcal{S}_{\mathcal{T}_{0}}=\\{\boldsymbol{s}_{\mathcal{T}_{0}}^{1},\boldsymbol{s}_{\mathcal{T}_{0}}^{2},...,\boldsymbol{s}_{\mathcal{T}_{0}}^{n_{s}}\\}$.
Similarly, we can apply the same self-checking mechanism to the verifier LM to
generate another set of $n_{m}$ responses:
$\mathcal{S}_{\mathcal{V}_{0}}=\\{\boldsymbol{s}_{\mathcal{V}_{0}}^{1},\boldsymbol{s}_{\mathcal{V}_{0}}^{2},...,\boldsymbol{s}_{\mathcal{V}_{0}}^{n_{m}}\\}$.
To perform question-level cross-check, for each perturbed input
$\boldsymbol{x}_{k}$, we generate $n_{q}$ sampled response sequences
$\mathcal{S}_{\mathcal{T}_{k}}=\\{\boldsymbol{s}_{\mathcal{T}_{k}}^{1},\boldsymbol{s}_{\mathcal{T}_{k}}^{2},...,\boldsymbol{s}_{\mathcal{T}_{k}}^{n_{q}}\\}$
from the target LM $\mathcal{T}$ and $n_{qm}$ sampled responses
$\mathcal{S}_{\mathcal{V}_{k}}=\\{\boldsymbol{s}_{\mathcal{V}_{k}}^{1},\boldsymbol{s}_{\mathcal{V}_{k}}^{2},...,\boldsymbol{s}_{\mathcal{V}_{k}}^{n_{qm}}\\}$
from the verifier LM $\mathcal{V}$.

Finally, we collect the total sampled sets
$\mathcal{S}=\\{\mathcal{S}_{\mathcal{T}_{0}},\mathcal{S}_{\mathcal{V}_{0}},\mathcal{S}_{\mathcal{T}_{k}},\mathcal{S}_{\mathcal{V}_{k}}\\}$
by combining all samples drawn from self-checking and cross-checking, which
will be used next for calculating a consistency score.

###  4.3 Stage 3: Consistency Score Calculation

This stage uses the generated sample sets in all previous stages to calculate
a numerical consistency score that captures the question-level and model-level
cross-checking paradigm.

####  4.3.1 Semantic-aware Consistency Check of QA Pairs

Most of the existing works mainly focus on examining the consistency of LM
outputs while ignoring the effect of the inputs. However, in QA tasks, it is
important to consider both inputs and outputs when measuring semantic
equivalence, as the same question can be rephrased in many ways. Although the
answers to these questions (e.g., ‘‘no’’ and ‘‘yes’’) may not be lexically
equivalent, the QA pairs as a whole can be semantically equivalent. In light
of this, we propose to check the semantic consistency of the QA pairs instead
of the answer only.

####  4.3.2 Self-checking Consistency (SC2) Score

Let $\mathcal{C}(\cdot,\cdot)$ denote a semantic equivalence checking operator
that takes two QA pairs as inputs. The operator $\mathcal{C}$ returns ‘‘Yes’’
if the two QA pairs are semantically equivalent, and ‘‘No’’ otherwise. This
operator should be reflexive, symmetric, and transitive. We implement the
checking operator using an LM by leveraging the prompt: ‘‘Are the following
two Question-Answering (QA) pairs semantically equivalent? [QA PAIR 1] [QA
PAIR 2]’’. We then map the best guess to a numerical semantic equivalent
score: {‘‘Yes’’ $\rightarrow$ 0.0, ‘‘No’’ $\rightarrow$ 1.0}. We use
$\mathcal{P}_{0}=(\boldsymbol{x}_{0},\boldsymbol{s}_{0})$ to denote the
original QA pair. The self-checking score $\mathcal{Z}_{\texttt{SC${}^{2}$}}$
of the target LM $\mathcal{T}$ can be calculated by

$\mathcal{Z}_{\texttt{SC${}^{2}$}}=\frac{1}{n_{s}}\sum_{i=1}^{n_{s}}\mathcal{C}(\mathcal{P}_{0},\mathcal{P}_{\mathcal{S}_{\mathcal{T}_{0}}}^{i}),$
(2)

where
$\mathcal{P}_{\mathcal{S}_{\mathcal{T}_{0}}}=\\{(\boldsymbol{x}_{0},\boldsymbol{s}_{\mathcal{T}_{0}}^{1}),...,(\boldsymbol{x}_{0},\boldsymbol{s}_{\mathcal{T}_{0}}^{n_{s}})\\}$
represents the QA pairs generated in the self-checking scenario.

####  4.3.3 Question-level Consistency (SAC3-Q) Score

Besides self-checking the original question $\boldsymbol{x}_{0}$, SAC3 further
assesses cross-check consistency of perturbed questions
$\\{\boldsymbol{x}_{1},\boldsymbol{x}_{2},...,\boldsymbol{x}_{k}\\}$. The
corresponding QA pairs compose a two-dimensional matrix, where each row
corresponds to a perturbed question ($k$ in total), and each column
corresponds to a sampled response ($n_{q}$ in total):

$\mathcal{P}_{\mathcal{S}_{\mathcal{T}_{j}}}^{i}=\begin{bmatrix}(\boldsymbol{x}_{1},\mathcal{S}_{\mathcal{T}_{1}}^{1})&...&(\boldsymbol{x}_{1},\mathcal{S}_{\mathcal{T}_{1}}^{n_{q}})\\\
...&...&...\\\
(\boldsymbol{x}_{k},\mathcal{S}_{\mathcal{T}_{k}}^{1})&...&(\boldsymbol{x}_{k},\mathcal{S}_{\mathcal{T}_{k}}^{n_{q}})\end{bmatrix}.$
(3)

Therefore, the question-level cross-checking consistency score
$\mathcal{Z}_{\texttt{SAC${}^{3}$-Q}}$ can be obtained by

$\mathcal{Z}_{\texttt{SAC${}^{3}$-Q}}=\frac{1}{n_{q}\cdot
k}\sum_{i=1}^{n_{q}}\sum_{j=1}^{k}\mathcal{C}(\mathcal{P}_{0},\mathcal{P}_{\mathcal{S}_{\mathcal{T}_{j}}}^{i}).$
(4)

####  4.3.4 Model-level Consistency (SAC3-M & SAC3-QM) Score

In addition to the question-level score, a model-level cross-check score is
calculated by performing cross-model checking and cross-question checking
using the verifier LM $\mathcal{V}$. Specifically, for the original question
$\boldsymbol{x}_{0}$, the model-level cross-checking consistency score
$\mathcal{Z}_{\texttt{SAC${}^{3}$-M}}$ is computed by

$\mathcal{Z}_{\texttt{SAC${}^{3}$-M}}=\frac{1}{n_{m}}\sum_{i=1}^{n_{m}}\mathcal{C}(\mathcal{P}_{0},\mathcal{P}_{\mathcal{S}_{\mathcal{V}_{0}}}^{i}),$
(5)

where
$\mathcal{P}_{\mathcal{S}_{\mathcal{V}_{0}}}=\\{(\boldsymbol{x}_{0},\boldsymbol{s}_{\mathcal{V}_{0}}^{1}),...,(\boldsymbol{x}_{0},\boldsymbol{s}_{\mathcal{V}_{0}}^{n_{m}})\\}$
is the QA pairs generated by the verified LM $\mathcal{V}$.

The cross-question consistency score on the verifier LM is computed on the QA
pairs produced by $\mathcal{V}$:

$\mathcal{P}_{\mathcal{S}_{\mathcal{V}_{j}}}^{i}=\begin{bmatrix}(\boldsymbol{x}_{1},\mathcal{S}_{\mathcal{V}_{1}}^{1})&...&(\boldsymbol{x}_{1},\mathcal{S}_{\mathcal{V}_{1}}^{n_{qm}})\\\
...&...&...\\\
(\boldsymbol{x}_{k},\mathcal{S}_{\mathcal{V}_{k}}^{1})&...&(\boldsymbol{x}_{k},\mathcal{S}_{\mathcal{V}_{k}}^{n_{qm}})\end{bmatrix}.$
(6)

The cross-model cross-question consistency score can thus be obtained through

$\mathcal{Z}_{\texttt{SAC${}^{3}$-QM}}=\frac{1}{n_{qm}\cdot
k}\sum_{i=1}^{n_{qm}}\sum_{j=1}^{k}\mathcal{C}(\mathcal{P}_{0},\mathcal{P}_{\mathcal{S}_{\mathcal{V}_{j}}}^{i}).$
(7)

####  4.3.5 Final Score and Model Confidence

The different variants of SAC3 capture different aspects of the uncertainty
about the original response and should complement each other. We thus consider
a combination of all variants including SAC3-Q , SAC3-M , SAC3-QM as the final
score:

$\mathcal{Z}_{\texttt{SAC${}^{3}$-all}}=\mathcal{Z}_{\texttt{SAC${}^{3}$-Q}}+\lambda(\mathcal{Z}_{\texttt{SAC${}^{3}$-M}}+\mathcal{Z}_{\texttt{SAC${}^{3}$-QM}}),$
(8)

where $\lambda$ is a weight factor for the verifier LM. Unless mentioned
otherwise, we use $\lambda=1$ by default in our experiments. In practice, as
the computation of each component is independent, they can be computed in
parallel to reduce latency. The detection prediction is made by comparing the
final score with a preset threshold. In addition to the computed score, we
also ask the target LM to generate a verbalized confidence score Tian et al.
(2023) along with its prediction when checking the semantic equivalence of QA
pairs. More discussions are offered in the Appendix C.2.

##  5 Data and Annotation

We evaluate our hallucination detection approach on two categories of QA
tasks, namely, classification QA and generation QA, with each category
containing two datasets. Following prior work Zhang et al. (2023a), we use the
following two binary classification datasets for evaluation on the
classification QA task:

  * •

Prime number: this dataset contains 500 questions that query the primality of
a randomly chosen prime number between 1,000 and 20,000, where the factual
answer is always ‘‘Yes’’. The synthesized hallucinated answers are ‘‘No, it is
not a prime number’’.

  * •

Senator search: the dataset consists of 500 questions that follow the
following template: ‘‘Was there ever a US senator that represented the state
of [US STATE NAME] and whose alma mater was [US COLLEGE NAME]?’’. The factual
answer is always ‘‘No’’. We also generate hallucinated answers: ‘‘Yes, there
was a US senator that represented the state of [US STATE NAME] and whose alma
mater was [US COLLEGE NAME].’’.

As for the generation QA tasks, we take questions from the following two open-
domain QA datasets and generate answers using LLMs. Then we manually annotate
the factuality of the answers following previous work Li et al. (2023a).

  * •

HotpotQA-halu: We randomly sample 250 examples from the training set of
HotpotQA Yang et al. (2018) and generate hallucinated answers drawn from
gpt-3.5-turbo. Then we manually annotate the answers by comparing the ground
truth and knowledge.

  * •

NQ-open-halu: Natural Questions (NQ)-open Lee et al. (2019) is a more
challenging open domain QA benchmark Kwiatkowski et al. (2019). We use the
same setting as HotpotQA-halu to create a small-scale dataset that consists of
250 non-factual and factual examples with manual annotations.

Please find more relevant details about data annotations in the Appendix C.3.

##  6 Experiments

###  6.1 Experimental Setup

##### Evaluation Models.

We use gpt-3.5-turbo from OpenAI as the target LM for our experiment. The
verifier LM is chosen from the following two models: (1) Falcon-7b-instruct
Almazrouei et al. (2023): an open-source causal decoder-only model built by
TII that is trained on 1,500B tokens of RefinedWeb Penedo et al. (2023) and
further enhanced using the curated corpora; and (2) Guanaco-33b: an open-
source instruction-following models through QLoRA Dettmers et al. (2023)
tuning of LLaMA Touvron et al. (2023) base model on the OASST1 dataset.

##### Implementation Details.

The evaluation is conducted using Azure OpenAI API. When performing semantic
perturbations and consistency checking, we set the temperature to 0.0 to get
deterministic high-quality outputs. Given a specific input query, we generate
$k=10$ semantically equivalent inputs using the prompt described in Section
4.1. For the self-checking-based method SC2 , we follow prior work Manakul et
al. (2023) to set the temperature to 1.0 and generate $n_{s}=10$ stochastic
samples. For SAC3-Q and SAC3-QM , we set $n_{q}=n_{qm}=1$ to reduce
computational cost. To further reduce the inference cost, we set $n_{m}=1$ by
default and combine SAC3-M with SAC3-QM to report the model-level results. We
use hallucination detection accuracy and area under the ROC curve (AUROC) to
evaluate the performance. In addition to the estimated hallucination score, we
also show the verbalized probabilities Tian et al. (2023) from the target LM
for comparison. We execute all experiments on 8 NVIDIA V100 32G GPUs.

Method | Prime number | Senator search
---|---|---
Score | Confidence | Score | Confidence
SC2 (gpt-3.5-turbo) | 65.9 | 67.5 | 56.1 | 53.1
SAC3-Q (gpt-3.5-turbo) | 99.4 | 99.7 | 99.7 | 99.7

Table 2: AUROC on classification QA tasks with 50% hallucinated samples and
50% factual samples.

###  6.2 Evaluation Results

####  6.2.1 Classification QA

##### Balanced Dataset.

We first experiment on balanced datasets with 50% hallucinated samples and 50%
factual samples. Table 2 compares the detection performance of SC2 and SAC3-Q
in terms of AUROC and verbalized confidence score. We observe that self-
checking (SC2) performs poorly on both datasets, with a low AUROC of $65.9\%$
and $56.1\%$, respectively. Our question-level cross-checking (SAC3-Q)
significantly outperforms the SC2 baseline achieving $>99\%$ AUROC on both
datasets and is in line with the verbalized confidence score, confirming the
effectiveness of cross-checking.

##### Unbalanced Dataset.

We further evaluate our method in a more challenging scenario where the
dataset only contains hallucinated samples. Table 3 presents the accuracy of
detecting hallucinated samples using a preset threshold of 0.5. In this case,
the performance of self-check drops significantly to 48.2% and 29.6%
respectively. SAC3-Q still outperforms SC2 by a large margin. The model-level
cross-check with verifier LMs performs well in the prime number dataset but
fails to accurately detect hallucination in the senator search dataset. This
is because both verifier LMs refuse to answer a large portion of the questions
on this dataset due to a lack of sufficient information. By combining the
target LM, SAC3-all with Guanaco-33b achieves the highest detection accuracy
compared to other baselines SC2 (+51.2%), SAC3-Q (+6.2%), and SAC3-QM (+
5.0%).

Method | Prime number | Senator search
---|---|---
Score | Confidence | Score | Confidence
SC2 (gpt-3.5-turbo) | 48.2 | 51.0 | 29.6 | 30.6
SAC3-Q (gpt-3.5-turbo) | 93.2 | 96.4 | 97.0 | 97.4
SAC3-QM (Falcon-7b) | 89.8 | 91.2 | 21.0 | 21.8
SAC3-QM (Guanaco-33b) | 94.4 | 96.3 | 45.6 | 46.2
SAC3-all (Falcon-7b) | 97.8 | 98.0 | 84.6 | 85.6
SAC3-all (Guanaco-33b) | 99.4 | 99.4 | 85.6 | 86.7

Table 3: Accuracy on classification QA tasks with 100% hallucinated samples
(with the threshold set to 0.5).

##### Impact of Threshold.

Since the choice of threshold has a significant impact on the detection
accuracy in the case where the dataset only contains positive (hallucinated)
samples, we further experiment with different thresholds and present the
results in Fig. 3. We observe that SAC3-Q and SAC3-all with Guanaco-33b are
more robust against large threshold values and outperform SC2 in most cases.

Figure 3: Impact of threshold on detection accuracy.

##### Why Does Self-checking Fail?

To further understand why self-checking methods fail to detect some
hallucinated responses, we visualize the distribution of consistency scores
using histogram plots in Fig. 4. We observe that for SC2, a significant
portion of hallucinated samples received highly consistent predictions. In
other words, the target LM made consistently wrong predictions due to a lack
of question and model diversity, which aligns with our analysis in Section 3.
On the other hand, benefiting from the semantically equivalent question
perturbation, SAC3-Q’s scores are more spread out in the inconsistent region,
which helps to improve the effectiveness of detecting hallucinations by
choosing a proper threshold.

Figure 4: Histogram of hallucination score.

####  6.2.2 Open-domain Generation QA

Compared to the classification QA tasks, detecting hallucinations in open-
domain generation QA tasks is more challenging. As shown in Table 4, SAC3-Q
exhibits better AUROC than SC2 (+7%) in both datasets. Compared to SAC3-Q ,
SAC3-QM shows 6.7% improvement in the HotpotQA-halu dataset but is slightly
worse in the NQ open dataset. SAC3-all leverages the advantages of question-
level and model-level cross-checking and achieves consistently good
performance in both datasets.

Method | HotpotQA-halu | NQ-open-halu
---|---|---
Score | Confidence | Score | Confidence
SC2 (gpt-3.5-turbo) | 74.2 | 77.0 | 70.5 | 72.7
SAC3-Q (gpt-3.5-turbo) | 81.3 | 81.4 | 77.2 | 78.5
SAC3-QM (Falcon-7b) | 83.0 | 79.5 | 67.5 | 62.0
SAC3-QM (Guanaco-33b) | 88.0 | 85.2 | 72.7 | 72.7
SAC3-all (Falcon-7b) | 84.5 | 84.5 | 77.1 | 77.2
SAC3-all (Guanaco-33b) | 87.0 | 86.8 | 77.2 | 77.8

Table 4: AUROC on open-domain generation QA tasks.

##### Effect of Verifier LM Weight.

In our previous experiments, we assigned equal importance to the consistency
score computed by the target and verifier LM. However, typically, the target
LM and the verifier LM have different architectures and scales such that the
user may have different levels of trust in their output truthfulness. This
difference in trust can be incorporated by introducing a weight $\lambda$ to
the consistency score produced by the verifier LM. For instance, if the goal
is to detect hallucination in a specialized domain and the verifier LM is a
domain-specific model developed for this domain, we can assign a large weight
to its scores (e.g., $\lambda>1.0$). In the general case, where the verifier
LM is a small-sized open-source model, we can apply a small weight value
(e.g., $\lambda<1.0$) to discount the influence of the verifier LM in the
final score. Fig. 5 visualizes the effect of various weight factors on the
generation QA datasets. We observe that SAC3-all with a higher weight would
result in a larger advantage over SAC3-Q in the HotpotQA-halu task, where the
verifier LM outperforms the target LM. On the contrary, in the NQ open task
where the target LM shows competitive performance, a smaller weight would
yield better results.

Figure 5: Effect of verifier LM weight on AUROC.

##### Effect of the Number of Perturbed Questions.

The performance of sampling-based methods is expected to improve as the sample
size increases, at the cost of higher latency and computational cost. In Fig.
6, we study this trade-off by varying the number of perturbed questions $k$
from 2 to 10. We observe that the performance of SAC3 increases as more
question samples are used but the performance gain gradually diminishes after
using more than 5 question samples. This suggests that in practice we could
use 2-4 question samples to achieve reasonably good performance at a low
computational cost.

Figure 6: Performance of SAC3 with varying number of perturbed questions.

##### Effect of the Model Type.

Our proposed SAC3 framework does not restrict the type of LLM employed and can
be naturally extended to various types of target LLMs. To verify this, in
addition to GPT-3.5 (gpt-3.5-turbo), we conduct experiments using GPT-4
(gpt-4) and PaLM 2 (chat-bison) on the considered four datasets in the setting
of the balanced dataset. The experimental results of comparing the proposed
SAC3-Q with the SC2 baseline are summarized in Table 5. We observe that the
proposed SAC3-Q consistently outperforms the SC2 baseline across all LLM
variants.

Method |  | Prime
---
Number
| Senator
---
Search
| HotpotQA
---
-halu
| NQ-open
---
-halu
SC2 (gpt-3.5-turbo) | 48.2 | 29.6 | 74.2 | 70.5
SC2 (gpt-4) | 38.3 | 18.4 | 79.7 | 76.3
SC2 (chat-bison) | 26.9 | 19.2 | 75.8 | 67.9
SAC3-Q (gpt-3.5-turbo) | 93.2 | 97.0 | 81.3 | 77.2
SAC3-Q (gpt-4) | 91.1 | 61.6 | 87.2 | 82.9
SAC3-Q (chat-bison) | 90.3 | 66.3 | 82.8 | 72.7

Table 5: Accuracy of different LLMs (GPT-3.5, GPT-4, and PaLM 2) on
classification and generation QA tasks.

##### Computational Cost.

We monitor the computational cost of our approach based on the number of model
evaluations consumed by OpenAI API and open-source LLMs inference. Assuming
the number of samples equals the number of perturbed questions, i.e.,
$n_{s}=n_{m}=n_{q}=n_{qm}$, the cost of SC2 is $n_{s}$ API calls, and our
SAC3-all needs $n_{s}$ target LM calls plus $2\times n_{s}$ verifier LM calls.
Beyond the model evaluations, SAC3 may have additional costs from question
perturbations and semantic equivalence checking via prompting. Additional
discussions can be found in the Appendix C.1.

##  7 Conclusion and Discussion

We investigate the relationship between the self-consistency of LM and the
factuality of the response and propose SAC3 as a robust hallucination
detection approach for black-box LMs. Through extensive empirical analysis,
our work highlights several findings. First, self-consistency checking alone
is insufficient to effectively detect question-level and model-level
hallucinations, where LMs generate consistently wrong responses to certain
questions. Second, cross-checking between semantically equivalent questions
can reduce the occurrence of persistent hallucinations, potentially by
reducing question-level ambiguity. Third, there exists a model-level disparity
in hallucinations, which we attribute to the inherent differences in LM
capabilities originating from different training procedures and data. Thus,
the verifier LM can be selected according to specific tasks to maximize
detection accuracy. We believe that our work is an important step towards
building reliable LLMs.

## Ethics Statement

This paper studies hallucination detection in LMs, which has significant
broader impacts in the field of natural language processing (NLP) and helps to
address ethical considerations regarding trustworthiness and reliability. The
research outcome may contribute to the development of more accurate and
reliable LMs by mitigating the risks of misinformation and biased outputs and
promoting accountability and trust in AI systems.

## Limintations

Our current experiments focus on the question-answering setting. Further
research is needed to assess the generalizability of the proposed framework
and the accuracy of semantic equivalence checks on more complex tasks such as
conversational or dialogue-based prompting. Additionally, it would be
interesting to investigate the efficiency-utility trade-off: we expect
increasing sample sizes to improve detection accuracy but may introduce
additional cost and latency. Speeding up the implementation through
parallelization is also worth exploring.

## References

  * Agrawal et al. (2023) Ayush Agrawal, Lester Mackey, and Adam Tauman Kalai. 2023.  Do language models know when they’re hallucinating references?  _arXiv preprint arXiv:2305.18248_.
  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023.  Falcon-40B: an open large language model with state-of-the-art performance.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cao et al. (2022) Meng Cao, Yue Dong, and Jackie Chi Kit Cheung. 2022.  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3340–3354.
  * Chen et al. (2023) Angelica Chen, Jason Phang, Alicia Parrish, Vishakh Padmakumar, Chen Zhao, Samuel R Bowman, and Kyunghyun Cho. 2023.  Two failures of self-consistency in the multi-step reasoning of llms.  _arXiv preprint arXiv:2305.14279_.
  * Chen and Mueller (2023) Jiuhai Chen and Jonas Mueller. 2023.  Quantifying uncertainty in answers from any language model via intrinsic and extrinsic confidence assessment.  _arXiv preprint arXiv:2308.16175_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Cole et al. (2023) Jeremy R Cole, Michael JQ Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023.  Selectively answering ambiguous questions.  _arXiv preprint arXiv:2305.14613_.
  * Das et al. (2023) Souvik Das, Sougata Saha, and Rohini K Srihari. 2023.  Diving deep into modes of fact hallucinations in dialogue systems.  _arXiv preprint arXiv:2301.04449_.
  * Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023.  Qlora: Efficient finetuning of quantized llms.  _arXiv preprint arXiv:2305.14314_.
  * Dhuliawala et al. (2023) Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023.  Chain-of-verification reduces hallucination in large language models.  _arXiv preprint arXiv:2309.11495_.
  * Elazar et al. (2021) Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021.  Measuring and improving consistency in pretrained language models.  _Transactions of the Association for Computational Linguistics_ , 9:1012–1031.
  * Gekhman et al. (2023) Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. 2023\.  Trueteacher: Learning factual consistency evaluation with large language models.  _arXiv preprint arXiv:2305.11171_.
  * Jang et al. (2022) Myeongjun Jang, Deuk Sin Kwon, and Thomas Lukasiewicz. 2022.  Becel: Benchmark for consistency evaluation of language models.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 3680–3696.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Ji et al. (2022) Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezheng Yu, Bryan Wilie, Min Zeng, and Pascale Fung. 2022.  Rho: Reducing hallucination in open-domain dialogues with knowledge grounding.  _arXiv preprint arXiv:2212.01588_.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  _arXiv preprint arXiv:2302.09664_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466.
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019.  Latent retrieval for weakly supervised open domain question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6086–6096.
  * Li et al. (2023a) Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023a.  Halueval: A large-scale hallucination evaluation benchmark for large language models.  _arXiv e-prints_ , pages arXiv–2305.
  * Li et al. (2023b) Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2023b.  Inference-time intervention: Eliciting truthful answers from a language model.  _arXiv preprint arXiv:2306.03341_.
  * Lin et al. (2023) Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023.  Generating with confidence: Uncertainty quantification for black-box large language models.  _arXiv preprint arXiv:2305.19187_.
  * Liu et al. (2021) Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021.  A token-level reference-free hallucination detection benchmark for free-form text generation.  _arXiv preprint arXiv:2104.08704_.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Mitchell et al. (2022) Eric Mitchell, Joseph J Noh, Siyan Li, William S Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D Manning. 2022.  Enhancing self-consistency and performance of pre-trained language models through natural language inference.  _arXiv preprint arXiv:2211.11875_.
  * Mündler et al. (2023) Niels Mündler, Jingxuan He, Slobodan Jenko, and Martin Vechev. 2023.  Self-contradictory hallucinations of large language models: Evaluation, detection and mitigation.  _arXiv preprint arXiv:2305.15852_.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Peng et al. (2023) Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023.  Check your facts and try again: Improving large language models with external knowledge and automated feedback.  _arXiv preprint arXiv:2302.12813_.
  * Tam et al. (2023) Derek Tam, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2023.  Evaluating the factual consistency of large language models through news summarization.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 5220–5255.
  * Tian et al. (2023) Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. 2023.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  _arXiv preprint arXiv:2305.14975_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Varshney et al. (2023) Neeraj Varshney, Wenlin Yao, Hongming Zhang, Jianshu Chen, and Dong Yu. 2023.  A stitch in time saves nine: Detecting and mitigating hallucinations of llms by validating low-confidence generation.  _arXiv preprint arXiv:2307.03987_.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.  _arXiv preprint arXiv:2203.11171_.
  * Xiao and Wang (2021) Yijun Xiao and William Yang Wang. 2021.  On hallucination and predictive uncertainty in conditional language generation.  _arXiv preprint arXiv:2103.15025_.
  * Yang et al. (2023) Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023.  A new benchmark and reverse validation method for passage-level hallucination detection.  _arXiv preprint arXiv:2310.06498_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380.
  * Ye et al. (2023) Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2023.  Cognitive mirage: A review of hallucinations in large language models.  _arXiv preprint arXiv:2309.06794_.
  * Zhang et al. (2023a) Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. 2023a.  How language model hallucinations can snowball.  _arXiv preprint arXiv:2305.13534_.
  * Zhang et al. (2023b) Shuo Zhang, Liangming Pan, Junzhou Zhao, and William Yang Wang. 2023b.  Mitigating language model hallucination with interactive question-knowledge alignment.  _arXiv preprint arXiv:2305.13669_.
  * Zhang et al. (2023c) Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023c.  Siren’s song in the ai ocean: A survey on hallucination in large language models.  _arXiv preprint arXiv:2309.01219_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zheng et al. (2023a) Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023a.  Why does chatgpt fall short in providing truthful answers.  _ArXiv preprint, abs/2304.10513_.
  * Zheng et al. (2023b) Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023b.  Why does chatgpt fall short in providing truthful answers?  _arXiv preprint arXiv:2304.10513_.
  * Zhou et al. (2020) Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad. 2020.  Detecting hallucinated content in conditional neural sequence generation.  _arXiv preprint arXiv:2011.02593_.

##  Appendix A Factuality Assessment of LLMs

This section first introduces the background of LM factuality assessment and
reviews the key components of existing black-box factuality assessment
approaches.

###  A.1 Background

Existing studies Manakul et al. (2023) showed that the factuality of LMs’
statements can be assessed by capturing the output uncertainty. If permitted
with complete white-box access, hallucinations of LMs can be detected through
uncertainty metrics such as entropy, which models the uncertainty of the
output distribution as
$\mathcal{H}(\boldsymbol{s})=-\sum_{i=1}^{N}p(s_{i}|s_{<i})\log
p(s_{i}|s_{<i})$ for a sequence of $N$ tokens
$\boldsymbol{s}=\\{s_{i}\\}_{i=1}^{N}$. The insight is that factual sentences
are likely to be composed of high-probability tokens and thus should have less
entropy, whereas non-factual statements are likely to be associated with lower
probability and higher entropy.

Although these methods provide an accurate estimate of the LM’s uncertainty,
computation of such metrics involves token-level output probability
distribution, which is typically not accessible for prominent LMs like ChatGPT
that are only available through black-box interactions.

###  A.2 Black-box Assessment via Checking Consistency in Sampled Responses

One natural way to extend the above white-box factuality assessment methods to
restrictive black-box settings is to approximate the uncertainty metrics by
sampling LM responses. Existing approaches Manakul et al. (2023) mostly focus
on assessing factuality through inspecting LM’s self-consistency. The typical
process of self-consistency check is as follows. First, for a given user-
queried prompt $\boldsymbol{x}$, let the LM generate a candidate response
$\boldsymbol{s}$ with a low (default) temperature. Next, to assess the
factuality of the candidate statements, a higher temperature value is used to
stochastically sample a diverse set of supporting responses
$\\{\boldsymbol{s}^{1},\boldsymbol{s}^{2},...,\boldsymbol{s}^{m}\\}$ from the
LM. Finally, a hallucination score $\mathcal{S}\in[0,1]$ is calculated by
measuring the consistency between the candidate response and the supporting
responses: $\mathcal{S}\rightarrow 0$ if the supporting responses agree with
the candidate response and $\mathcal{S}\rightarrow 1$ if they disagree. The
consistency between two responses is typically measured by checking the
textual entailment or lexical similarity using an additional pre-trained NLU
model such as BERT Manakul et al. (2023); Kuhn et al. (2023).

Method | Input | Model | Output |  Semantic Consistency
---|---|---|---|---
SC2 | Q | Target |  Yes, there was a US senator that represented the state of North Dakota and whose alma mater was MIT. His name was Quentin N. Burdick. | ✓
| Q | Target |  Yes, there was a US senator who represented the state of North Dakota and whose alma mater was MIT. His name was Quentin N. Burdick. | ✓
SAC3-Q | Q1 | Target |  As an AI language model, I do not have access to the most recent updates or news. However, as of my last update, there is no North Dakota US senator who graduated from MIT. | ✗
| Q2 | Target |  As an AI language model, I do not have access to real-time information or current events. However, as of my last update, I am not aware of any US senators from North Dakota who were also MIT graduates. | ✗
SAC3-M | Q | Verifier |  No, there has never been a US senator who represented the state of North Dakota and attended MIT. | ✗
SAC3-QM | Q1 | Verifier | No, a North Dakota US senator did not have MIT listed as their alma mater. | ✗
| Q2 | Verifier | There are no known US senators from North Dakota who are also MIT graduates. | ✗

Table 6:  Input Question (Q) and Answer (A): Q = “Was there ever a US senator
that represented the state of North Dakota and whose alma mater was MIT? ” A =
“Yes, there was a US senator who represented the state of North Dakota and
whose alma mater was MIT. His name was Quentin Burdick.” Semantically
rephrased questions:  Q1 = “Did a North Dakota US senator have MIT listed as
their alma mater? ” Q2 = “Were there any US senators from North Dakota who
were also MIT graduates?

##  Appendix B Complete Prompts

The prompt templates that we used are provided in Table 7, specifically the
question perturbations using semantically equivalent prompts and semantic
equivalence checks between two QA pairs.

Objective |  Template
---|---
Semantic question perturbation |  For the question [QUERIED QUESTION], provide ${$k$} semantically equivalent questions \n Are the following two inputs semantically equivalent? \n [QUERIED INPUT] \n [GENERATED INPUT]
Semantic equivalence check |  Are the following two Question-Answering (QA) pairs semantically equivalent? Provide your best guess and the probability that it is correct (0.0 to 1.0). Given ONLY the guess (Yes or No) and probability, no other words or explanation. For example: \n Guess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\n Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability! \n \n The first QA pair is: \n Q: ${THE QUESTION} \n A: ${THE ANSWER} \n The second QA pair is: \n Q: ${THE QUESTION} \n A: ${THE ANSWER}
Table 7: Prompt templates for different objectives, including semantically
equivalent perturbations of input questions and semantic equivalent check of
QA pairs.

##  Appendix C Additional Details and Discussions

###  C.1 Computational Cost

High computational cost is a common research challenge faced in many areas of
black-box LLM studies, including self-consistency Wang et al. (2022) and self-
check Manakul et al. (2023). Compared with these existing works, our framework
enables model-level parallel execution of LLM inferences (either target or
verifier LM) in the form of parallel API calls, which helps to mitigate the
overhead. Moreover, we may further employ more advanced prompt strategies for
semantic consistency checking that condense the pairwise comparisons to a
single inference call to substantially reduce the time complexity. Finally,
our framework also offers the flexibility for the user to adjust the balance
between computational cost and accuracy: users may choose to trade off
accuracy for efficiency in scenarios where low overhead is more valued than
high detection precision.

To achieve a more balanced trade-off, we primarily recommend two strategies:
(1) Utilizing a smaller sample size, such as the number of question
perturbations and self-evaluations. Our ablation experiments demonstrate that
the performance of SAC3 improves with an increase in question samples, but the
performance gain plateaus after exceeding five question samples. This
indicates that using 2-4 question samples in practice could yield satisfactory
results with minimal computational expense. (2) Implementing an optimized
prompt strategy. With the original input question, we prompt the GPT model to
generate multiple paraphrases in a single API call, i.e., $\mathcal{O}(1)$ for
question paraphrasing. Although semantic consistency checking via a pairwise
method can be slightly burdensome, we can devise an advanced prompt strategy
to parallelize the pairwise consistency check, i.e., through one API call
instead of multiple calls, effectively reducing the inference complexity from
$\mathcal{O}(n)$ to $\mathcal{O}(1)$. These strategies empower users to
optimally balance efficiency (light version) and accuracy (performance
version), facilitating more informed decisions in practice.

###  C.2 Semantic Consistency Checking

The goal of our research is to provide a flexible framework for the effective
detection of hallucinations in black-box LLMs. Compared to previous approaches
Manakul et al. (2023) based on similarity metrics such as BERTScore, employing
LLM for semantic consistency checking not only offers better accuracy but also
eliminates the need for involving additional models. We note that our design
is not dependent on any specific type of LLM and the users are allowed to
choose freely for each component. Specifically, we chose GPT as the
instantiation in our experiments due to its well-acclaimed ability to follow
human instructions which is crucial for evaluation. In practice, the user may
choose any open-sourced LLMs that have been aligned to follow instructions.

###  C.3 Data Annotations

For the two classification QA tasks (Prime Number and Senator Search), we use
the synthesized hallucinated answers following prior work Zhang et al.
(2023a). For the task of prime number, the factual/true answer is always
“Yes”, i.e., all the testing numbers are prime numbers. The synthesized
hallucinated response is “No”. The factual/true answer of the senator search
is always "No" and the synthesized hallucinated answer is “Yes”. Such an
experimental setting on the binary classification tasks is realistic since we
have verified that on these datasets most of the responses generated by
gpt-3.5-turbo are indeed “Yes” or “No” which align with the synthesized
response.

For the generation QA tasks (HotpotQA-halu and NQ-Open-halu), we used answers
generated by LLM (gpt-3.5-turbo) for experiments, which do not have pre-
defined factual/non-factual labels. Therefore, we manually annotated these
LLM-generated answers by comparing them with the ground truth.

We would like to note that in practice, a more versatile hallucination
detection approach should be able to evaluate the factuality of a sample
regardless of its origin (e.g., synthesized or generated by itself / other
LLMs). In our framework, this is achieved through semantically equivalent
question perturbation and cross-model response consistency checking. We plan
to release the annotated datasets to facilitate future research.

##  Appendix D Additional Examples

Table 6 provides another illustrative example to explain our methodology in
the senator search dataset.