# Molding light via glided photonic crystal interfaces

Zi-Mei Zhan  School of Physical Science and Technology, Guangxi Normal
University, Guilin, China, 541004  Guangxi Key Laboratory of Nuclear Physics
and Technology, Guangxi Normal University, Guilin, China, 541004  Hai-Xiao
Wang  School of Physical Science and Technology, Guangxi Normal University,
Guilin, China, 541004  Guangxi Key Laboratory of Nuclear Physics and
Technology, Guangxi Normal University, Guilin, China, 541004
<EMAIL_ADDRESS>
###### Abstract

Photonic crystals (PCs) are periodic dielectric structures that severed as an
excellent platform to manipulate light. A conventional way to guide/trap light
via PCs is to introduce a line or point defect by removing or modifying
several unit cells. Here we show that the light can be effectively guided and
trapped in the glided photonic crystal interfaces (GPCIs). The projected band
gap of GPCIs, which depends on the glide parameter, is characterized by a
Dirac mass. Interestingly, the GPCIs with zero Dirac mass is a glide-symmetric
waveguide featured with excellent transmission performance even in the
presence of sharp corners and disorders. Moreover, placing two GPCIs with
opposite Dirac mass together results in a photonic bound state due to the
Jackiw-Rebbi theory. Our work provides an alternative way towards the design
of ultracompact photonic devices such as GPCIs-induced coupled cavity-
waveguide system and waveguide splitter.

††preprint: AIP/123-QED

Molding the flow of light lies at the heart of both photonic and optics
Joannopoulos  _et al._ (2008). In the last decades, tremendous efforts have
been devoted to investigate and control the optical properties of materials,
giving birth to several important photonics and optics branches, such as
waveguide optics Jamois  _et al._ (2003); Jiang  _et al._ (2007); Luan and
Chang (2006); Song  _et al._ (2022), nonlinear optics Berger (1998); Segal
_et al._ (2015), and topological photonics Ozawa  _et al._ (2019); Kim, Jacob,
and Rho (2020); Wang and Jiang, Jian-Hua (2022). Among them, one of the most
greatest achievements is the photonic crystals (PCs) Yablonovitch (1987); John
(1987), which are periodic dielectric structures with photonic band gaps.
Similar to the formation of the electronic band structures in the solid
crystals, the electromagnetic waves propagate in a periodic dielectric
structure are subjected to Bragg scattering, forming the passing and forbidden
bands for photons. From the application perspective, the key to manipulate the
light in PC is to generate the guiding and localized modes in the photonic
band gaps, which are two fundamental modes in the design of photonic devices.
Owing to the nonequilibrium nature of photon and the design flexibility of
PCs, engineering photonic band structure get rids of the limitation of Fermi
level and hence is more accessible than that in electronics. In general, one
can realize the guiding and localized modes in the photonic band gap by simply
modifying the size/position of some unit cells or even removing them, making
PC a versatile platform to guide and trap light. To date, such an approach
have been extensively utilized to design various photonic devices, such as
waveguide bend Mekis  _et al._ (1996); Lin  _et al._ (1998), waveguide
splitter Fan  _et al._ (2001a), channel-drop filter Fan  _et al._ (1998);
Manolatou  _et al._ (1999); Fan  _et al._ (2001b).

Recently, it is recognized that glide symmetry, a composition of mirror and
translation operation, is beneficial for engineering the properties of
periodic structures. For example, the glide symmetry provides a powerful tool
to synthetic Kramers degeneracy in topological classical wave systems Lin  _et
al._ (2017); Liu, Wei, and Xiao (2022); O and Kim (2024); Wang  _et al._
(2022); Yue  _et al._ (2022). Besides, the glide symmetry has been used to
improve gain and bandwidth of leaky-wave and lens antennas Chen  _et al._
(2020); Fan  _et al._ (2020). Moreover, the combination of glide symmetry and
slow light leads to a strongly enhanced chiral light-matter interaction
Söllner  _et al._ (2015); Patil  _et al._ (2022). Very recently, a glide-
symmetric phononic crystal interface that supports wide-bandwidth, single-mode
waves is proposed Iglesias Martínez  _et al._ (2022); Laude  _et al._ (2023),
and is suggested to have potential application on the extremely sensing and
isolation Janković and Alù (2021). Nevertheless, the glide-symmetric photonic
crystal interface remains unexplored and it is unknown whether or not such an
interface can forms into waveguide bend with high transmissions, or even
cavity with high-Q factor.

In this letter, we first introduce a glided photonic crystal interface (GPCI)
by making two half PCs with relative displacement. Under the periodic boundary
condition, one half PCs coincides with another half PCs after a translation
and a reflection with respect to the GPCI. The eigen spectrum calculations
show that the dispersions of the GPCIs are closely related to the relative
displacement. A Dirac mass is employed to characterize the projected band gap
of the GPCIs. Interestingly, GPCIs with zero Dirac mass is a glide-symmetric
waveguide, and combining two GPCIs with opposite Dirac mass together leads to
a photonic bound state (PBS) due to Jackiw-Rebbi theory. Based on the GPCIs-
induced waveguide and PBS, we revisit some photonic devices, including
waveguide bend, coupled cavity-waveguide systems and waveguide splitter. Our
work suggests a new route towards the design of ultracompact integrated
photonics.

We start from a two-dimensional PCs with circular air holes arrayed in square
lattice in a dielectric background [see Fig. 1(a)]. The radius of each
circular air hole is $r=0.5a$, where $a$ is lattice constant. Throughout this
work, we use silicon ($\epsilon=11.9$) as the dielectric background, and only
consider the transverse-magnetic (TM) mode. All simulations are carried out
with the radio frequency module of commercial software COMSOL Multiphysics. A
typical photonic band structure with complete band gap is presented in Fig.
1(b), where the grey regions are the light cone of air and only those states
below the light cone are of interest.

To illustrate the GPCI, a glide parameter $g$ is employed to describe the
relative displacement of the PCs between upper and lower interfaces. As
schematically indicated by dashed rectangular box in Fig. 1(c), a GPCI forms
by moving the lower half of PCs along $x$-directions with parameter $g$ while
the upper half of PCs keeps unchanged. For simplify, the parameter $g$ ranges
from $-a/2$ to $+a/2$, which the positive and negative signs refer to the
rightward and leftward moving directions, respectively. Obviously, the initial
square PCs represent the situation with $g=0$. For $g=\pm a/2$, namely, one
half of the PCs is shifted by a half lattice constant along $x$-direction,
results in the two interface bands linearly cross at $k_{x}=\pi/a$ [see the
solid colored lines in Fig. 1(d)]. We remark that such interface states are
originated from the spatial mismatch of symmetry Wannier functions between PCs
of the upper and below interface Huang  _et al._ (2016); Chen  _et al._
(2018); Lu  _et al._ (2021); Zhang  _et al._ (2022); Vaidya  _et al._ (2023)
[see details in the Supplementary Material]. Note that we adopt a supercell
that consisting of $10\times 1$ unit cells for eigen spectrum calculation. We
remark that such a band crossing (also known as a Dirac point) is protected by
the glide-symmetry, which is crucial for designing the photonic cavity, as we
will elaborately discuss later. To understand it, we construct an anti-unitary
operator $\Theta_{x}=G_{x}\star T$, where $G_{x}:(x,y)\rightarrow(x\pm
a/2,-y)$ and $T$ are glide operator and time-reversal symmetry, respectively.
Hence, acting twice $\Theta_{x}$ on a photonic Bloch states give additional
phase factor $e^{-ik_{x}\cdot a}$. In other words, $\Theta_{x}$ transforms
$(k_{x},k_{y})$ into $(-k_{x},k_{y})$ and we have

${\Theta_{x}}^{2}=e^{-ik_{x}a}|_{k_{x}=\pi/a}=-1,$ (1)

which, as an analog of the Kramers theorem fermions, guarantees that all bands
are degenerate by pairs at $k_{x}=\pi/a$. Such gapless interface states
protected by the synthetic Kramers theorem exhibit with excellent performance
on guiding waves [also see in Fig. 2] and have been utilized as the glide-
symmetric waveguide (GSWG) Iglesias Martínez  _et al._ (2022); Laude  _et al._
(2023); Patil  _et al._ (2022) in the context of phononic crystals.

Figure 1: (a) Schematic of two-dimensional PCs consisting of circular air
holes with $r=0.5a$ arrayed in square lattice ($a$ refers to the lattice
constant). (b) The photonic band structure for a square array of air columns
drilled in a silicon substrate with permittivity $\epsilon=11.9$. The grey
area refers to photonic states that above the light line. Inset: the first
Brillouin zone. (c) Schematic of a GPCI highlighted by green wavy lines, which
is formed by moving the lower half of PCs with glide parameter $g$. (d) Left
panel: the projected band structure of GPCIs, where the solid (dashed) lines
refer to interfaces states with $g=\pm 0.5a(g=\pm 0.3a)$. Right panel: the
eigen electric fields of interface states at $k_{x}=\pi/a$ with odd and even
parities.

Inspired by the design of GSWG, which actually is a GPCI with $g=\pm a/2$, we
construct L-shaped (U-shaped) GPCIs [see Supplementary Material for the
details of the geometry designs] by shifting PCs along both $x$ and $y$
directions with half lattice constant. To examine the guiding ability of the
L-shaped (U-shaped) GPCIs, we first plot the transmission of L-shaped
(U-shaped) GPCIs, as indicated by the solid red (blue) line in Fig. 2(a). In
comparison, the transmission of GSWG represented by black line is also
presented. It is seen that all three cases exhibit perfect transmission
(nearly unity) within the whole bulk band gap along the GPCIs. Furthermore, we
also display the electric field patterns of GSWG, L-shaped, and U-shaped GPCIs
in Fig. 2(b-d), respectively, at a frequency of $0.289(c/a)$. It is observed
that electromagnetic wave propagate along the L-shape and U-shaped GPCIs
without much scatterings even in presence of sharp corner. In comparison, to
have perfect transmission in conventional waveguide bend, there is a need to
adjust the configuration around the sharp corners. We also demonstrate the
GPCIs exhibit with high fabrication tolerance by implementing numerical
simulations in GSWG with disorders [see details in Supplementary Material].
For the both L-shaped and U-shaped GPCIs, it is worth noting that there exists
other channel besides the GPCIs. Fortunately, these channels do not support
any interface states. Therefore, GPCIs have been proven to exhibit excellent
transmission performance as a waveguide bend.

Figure 2: (a) Transmission of GSWG (black line), L-shaped (red line) and
U-shaped (blue line) GPCI versus frequency. The light yellow refers to the
bulk band gap. (b-d) The electric field patterns of GSWG, (c) L-shaped and (d)
U-shaped GPCIs, respectively, at a frequency of $0.289(c/a)$.

On the other hand, when $g\neq\pm a/2$, the synthetic Kramers theorem in Eq. 1
is no longer valid and hence, the degeneracy of two interface bands at
$k_{x}=\pi/a$ shall be lifted. As is expected, two interface bands separated
by a frequency gap emerge in the bulk band gap for the case of $g=0.3a$ [see
the dashed colored lines in Fig. 1(d)]. One step further, Fig. 3(b) presents
the frequency ranges (indicated by orange and blue areas) of the two projected
bands versus the glide parameter $g$. It is observed that the frequency ranges
of the two projected interface bands are symmetric about $g=0$ since two GPCIs
with opposite $g$ share the identical eigen spectrum. Taking $g\in[0,a/2]$
into consideration, the frequency gap between two projected interface bands is
equal to the bulk gap when $g=0$, and gradually decreases and finally closed
at $g=a/2$. Meanwhile, the frequency ranges of projected interface bands
gradually increase and finally fulfill the whole bulk band gap at $g=a/2$. For
$g\in[-a/2,0]$, the processes are reversed.

During the band evolution, it is note worthing that the frequency gap between
two projected interface bands (described by Dirac mass $m$) experiences a
process of the opening and reclosing. Such a Dirac mass $m$ is proportional to
the overlapping integral of the electromagnetic field of the two interface
states at $k_{x}=\pi/a$, i.e., $m\propto\int
d\bm{r}(\bm{E}_{+}\cdot{\hat{\epsilon}}\cdot\bm{E}_{-}^{\star}+c.c.)$, where
the subscripts $\pm$ represent the upper and lower interface bands. Obviously,
for $g=\pm a/2$, the parities of $E_{\pm}$ are well defined, namely, they are
of opposite parities about $x=0$ [also see the insets of Fig. 1(d)], which
makes the zero overlapping integral, i.e., $m=0$. Remarkably,a frequency gap
with positive (negative) Dirac mass is induced by GPCIs with $g(-g)$ [see
details in Supplementary Material]. Hence, one can build a mass domain-wall
system by placing two GPCIs with opposite $g(g\neq\pm a/2)$ together.

Figure 3: (a) The evolution of the GPCI states versus the glide parameter
$g_{x}$. The orange and blue areas represent the GPCI states, while the grey
areas refer to the bulk states. The red line represent evolution of frequency
the GPCIs-induced PBS. (b) Schematic of the formation of PBS by combing two
GPCIs with $|g|$ and $-|g|$ together. Upper panel: the GPCIs with $|g|$ is on
the left side of that with $-|g|$, termed as type-I domain-wall systems. Lower
panel: the GPCIs with $|g|$ is on the right side of that with $-|g|$, termed
as type-II domain-wall systems. (c,d) Eigen spectra of the (c) type-I mass
domain-wall system, and (d) type-II mass domain-wall system, with $g=0.3281a$.

In general, there are two configurations of the mass domain-wall systems. As
schematically shown in Fig. 3(b), the first configuration (denoted as type-I,
see upper panel) is formed by shifting PCs of region II (III) with positive
(negative) $g$, while the second configuration (denoted as type-II, see lower
panel) is formed by shifting PCs of region II (III) with negative (positive)
$g$. According to the Jackiw-Rebbi theory Jackiw and Rebbi (1976), it is
predicted that a PBS shall be localized at the boundary of two GPCIs with
opposite $g$, as schematically indicated by the red and blue stars. Note that
Fig. 3(a) also plots the frequency of PBS (indicated by red and blue lines)
versus the displacement of PCs between regions I and II. As is expected, for
both type-I and type-II configurations, the PBS emerges within the frequency
gap of the projected interface bands. Moreover, we also present the eigen
spectra of two mass domain-wall systems in Fig. 3(c) and (d), respectively.
Note that these two domain-wall systems, i.e., with type-I and type-II
configurations, are formed by GPCIs with $g=\pm 0.3281a$, which gives PBS with
same frequency. The corresponding normalized electric field pattern $|E_{z}|$
are also shown in the insets. It is seen that the PBS in the mass domain-wall
system with type-I configuration is even-symmetric, while that with type-II
configuration is odd-symmetric. Remarkably, these PBSs induced by two GPCIs
with opposite $g$ can be acted as photonic cavity modes, which offer an
alternative way to realize those cavity-based photonic devices.

As an expample, The GPCIs-induced waveguide and PBS could be integrated in a
single systems, forming a GPCIs-induced coupled cavity-waveguide system. As
schematically shown in Figs. 4(a) and 4(b), based on the mass domain-wall
system with type-I and type-II configurations, which consisting of PCs of
Region II, III and IV, we further introduce a GSWG by shifting the PCs of
Region I with half lattice constant along $x$-direction and hence, forming a
GPCIs-induced type-I (upper panel) and type-II (lower panel) coupled cavity-
waveguide system. The transmission of type-I and type-II coupled cavity-
waveguide systems are displayed in Fig. 4(c) and 4(d), respectively. It is
observed that the transmissions of both systems are nearly unity except for
the resonance frequency, which possesses a Lorentzian lineshape. Nevertheless,
around the resonance frequency, the half maximum width of transmission of the
type-I coupled cavity-waveguide is much larger than that of the type-II
coupled cavity-waveguide, indicating that Q-factor of the type-I cavity mode
is smaller than that of type-II cavity mode. Moreover, we display the
normalized electric field patterns of type-I and type-II coupled cavity-
waveguide around the resonance frequency in Fig. 4(e) and 4(f), respectively.

Figure 4: (a,b) Schematic illustration of the formation of the (a) type-I, and
(b) type-II coupled cavity-waveguide systems. (c,d) Transmission of the (c)
type-I, and (d) type-II coupled cavity-waveguide systems. (e,f) The electric
field patterns of (e) type-I, and (f) type-II coupled cavity-waveguide systems
around the resonance frequencies.

In what follows, we show that the GPCIs are compatible with the conventional
waveguide. As is shown in Fig. 5(a), we design a T-shaped splitter structure
that consisting of both conventional waveguide and GSWG. To be concrete, the
PCs of Region I and IV is first moved with $g=0.4a$ and $g=0.1a$,
respectively. Then we move PCs of Region II and III with $g=-0.4a$ and
$g=-0.1a$, respectively. Hence, the horizontal interface forms into a GSWG,
which the frequency ranges matches well with that of the vertical conventional
waveguide. It is predicted the wave from input port P1 will pass through the
vertical waveguide, and then split into two branches along two GSWGs and
finally, output from ports P2 and P3, respectively.

Figure 5: (a) Schematic illustration of the T-shaped waveguide splitter
consisting of both conventional waveguide and GPCIs. (b) Transmission of the
waveguide splitter. Insets: the electric field pattern of the waveguide
splitter at a frequency of $0.2997(c/a)$.

To verify it, we present the transmission $S_{21}$ and $S_{31}$ versus
frequency in Fig. 5(b). It is observed that two transmissions are overlap
completely, i.e., $S_{21}=S_{31}$, indicating that energy output from P2 and
P3 are equivalent. On the other hand, both transmissions are lower than unity,
suggesting there exists some energy losses. A typical normalized electric
pattern at a frequency of $0.2997(c/a)$ displayed in the inset of Fig. 5(b)
shows that the normalized electric field at the turns are stronger than any
other places, which is in good agreement with the conclusion unveiled by
transmission behaviour. However, in similar to the conventional waveguide
splitter, we remark that it maybe possible to improve the performance of
waveguide splitter by making a revision of configuration, which is left for
future works.

To conclude, we propose the concept GPCIs that are formed by simply gliding
one half of the PCs. The GPCIs support two guiding mode, of which the
dispersions are closely rely on the glide parameters $g$. A Dirac mass is
employed to characterize the frequency gap of two guiding modes. We
demonstrate that GPCIs with $g=\pm a/2$ forms a GSWG and exhibit excellent
performance on the ability of guiding waves even in the presence of sharp
bends and disorders. Moreover, placing two GPCIs with opposite Dirac mass
results in a PBS due to the Jackiw-Rabbi mechanism, which can be acted as a
photonic cavity mode. We revisit two coupled GPCIs-induced cavity-waveguide
systems by studying its transmission properties. Last but not least, a
waveguide splitter integrated by GPCI-induced and conventional PC waveguides
are also discussed. Our work provides an alternative way towards the design of
ultracompact integrated photonics.

See the supplementary material for (1) the origin of the interface states in
glided PCs, (2) the formation of L-shaped and U-shaped GPCIs, (3) the
robustness of interface states and PBS in glide PCs with disorder, and (4) the
GPCIs with opposite $g$ and Dirac mass with opposite sign.

Zi-Mei Zhan and Hai-Xiao Wang thank the support from the project of all-
English course construction for graduate students in Guangxi Normal
University, and the Central Government Guidance Funds for Local Scientific and
Technological Development, China (No. Guike ZY22096024), Natural Science
Foundation of Guangxi Province (Grant No. 2023GXNSFAA026048), and National
Natural Science Foundation of China (Grant No. 11904060).

## Author declarations

### Conflict of Interest

The authors have no conflicts to disclose.

### Author Contributions

Zi-Mei Zhan performed the numerical calculations and simulations, and drew the
figures. Hai-Xiao Wang initiated the project and guided the research, and
wrote the manuscript and the Supplementary Material.

### Data availability

The data and code that support the findings of this study are available from
the corresponding author upon reasonable request.

## References

  * Joannopoulos  _et al._ (2008) J. D. Joannopoulos, S. G. Johnson, J. N. Winn, and R. D. Meade,  _Photonic Crystals: Molding the Flow of Light_ , 2nd ed. (Princeton University Press, USA, 2008).
  * Jamois  _et al._ (2003) C. Jamois, R. Wehrspohn, L. Andreani, C. Hermann, O. Hess, and U. Gösele, “Silicon-based two-dimensional photonic crystal waveguides,” Photonics and Nanostructures - Fundamentals and Applications 1, 1–13 (2003).
  * Jiang  _et al._ (2007) W. Jiang, L. Gu, X. Chen, and R. T. Chen, “Photonic crystal waveguide modulators for silicon photonics: Device physics and some recent progress,” Solid-State Electronics 51, 1278–1286 (2007).
  * Luan and Chang (2006) P.-G. Luan and K.-D. Chang, “Transmission characteristics of finite periodic dielectric waveguides,” Optics Express 14, 3263 (2006).
  * Song  _et al._ (2022) T. Song, H. Chu, J. Luo, Z. Cao, M. Xiao, R. Peng, M. Wang, and Y. Lai, “Ultracompact Photonic Circuits without Cladding Layers,” Physical Review X 12, 011053 (2022).
  * Berger (1998) V. Berger, “Nonlinear Photonic Crystals,” Physical Review Letters 81, 4136–4139 (1998).
  * Segal  _et al._ (2015) N. Segal, S. Keren-Zur, N. Hendler, and T. Ellenbogen, “Controlling light with metamaterial-based nonlinear photonic crystals,” Nature Photonics 9, 180–184 (2015).
  * Ozawa  _et al._ (2019) T. Ozawa, H. M. Price, A. Amo, N. Goldman, M. Hafezi, L. Lu, M. C. Rechtsman, D. Schuster, J. Simon, O. Zilberberg, and I. Carusotto, “Topological photonics,” Reviews of Modern Physics 91, 015006 (2019).
  * Kim, Jacob, and Rho (2020) M. Kim, Z. Jacob, and J. Rho, “Recent advances in 2D, 3D and higher-order topological photonics,” Light: Science & Applications 9, 130 (2020).
  * Wang and Jiang, Jian-Hua (2022) H.-X. Wang and Jiang, Jian-Hua, “A Short Review of All-Dielectric Topological Photonic Crystals,” Frontiers in Physics 10, 17 (2022).
  * Yablonovitch (1987) E. Yablonovitch, “Inhibited Spontaneous Emission in Solid-State Physics and Electronics,” Physical Review Letters 58, 2059–2062 (1987).
  * John (1987) S. John, “Strong localization of photons in certain disordered dielectric superlattices,” Physical Review Letters 58, 2486–2489 (1987).
  * Mekis  _et al._ (1996) A. Mekis, J. C. Chen, I. Kurland, S. Fan, P. R. Villeneuve, and J. D. Joannopoulos, “High Transmission through Sharp Bends in Photonic Crystal Waveguides,” Physical Review Letters 77, 3787–3790 (1996).
  * Lin  _et al._ (1998) S.-Y. Lin, E. Chow, V. Hietala, P. R. Villeneuve, and J. D. Joannopoulos, “Experimental Demonstration of Guiding and Bending of Electromagnetic Waves in a Photonic Crystal,” Science 282, 274–276 (1998).
  * Fan  _et al._ (2001a) S. Fan, S. G. Johnson, J. D. Joannopoulos, C. Manolatou, and H. A. Haus, “Waveguide branches in photonic crystals,” Journal of the Optical Society of America B 18, 162 (2001a).
  * Fan  _et al._ (1998) S. Fan, P. R. Villeneuve, J. D. Joannopoulos, and H. A. Haus, “Channel drop filters in photonic crystals,” Optics Express 3, 4 (1998).
  * Manolatou  _et al._ (1999) C. Manolatou, M. Khan, S. Fan, P. Villeneuve, H. Haus, and J. Joannopoulos, “Coupling of modes analysis of resonant channel add-drop filters,” IEEE Journal of Quantum Electronics 35, 1322–1331 (1999).
  * Fan  _et al._ (2001b) S. Fan, P. R. Villeneuve, J. D. Joannopoulos, and H. A. Haus, “Loss-induced on/off switching in a channel add/drop filter,” Physical Review B 64, 245302 (2001b).
  * Lin  _et al._ (2017) J. Y. Lin, N. C. Hu, Y. J. Chen, C. H. Lee, and X. Zhang, “Line nodes, Dirac points, and Lifshitz transition in two-dimensional nonsymmorphic photonic crystals,” Physical Review B 96, 075438 (2017).
  * Liu, Wei, and Xiao (2022) Z. Liu, G. Wei, and J.-J. Xiao, “Geometric anisotropy induced higher-order topological insulators in nonsymmorphic photonic crystals,” Physical Review B 106, 085415 (2022).
  * O and Kim (2024) K.-H. O and K.-H. Kim, “Multiband quadrupole topological photonic crystals with glide symmetries,” Optics & Laser Technology 168, 109901 (2024).
  * Wang  _et al._ (2022) H.-X. Wang, Y. Chen, G.-Y. Guo, H.-Y. Kee, and J.-H. Jiang, “Possible realization of optical Dirac points in woodpile photonic crystals,” Optics Express 30, 17204 (2022).
  * Yue  _et al._ (2022) Z. Yue, Z. Zhang, H.-X. Wang, W. Xiong, Y. Cheng, and X. Liu, “Glided acoustic higher-order topological insulators based on spoof surface acoustic waves,” New Journal of Physics 24, 053009 (2022).
  * Chen  _et al._ (2020) Q. Chen, O. Zetterstrom, E. Pucci, A. Palomares-Caballero, P. Padilla, and O. Quevedo-Teruel, “Glide-Symmetric Holey Leaky-Wave Antenna With Low Dispersion for 60 GHz Point-to-Point Communications,” IEEE Transactions on Antennas and Propagation 68, 1925–1936 (2020).
  * Fan  _et al._ (2020) F. Fan, M. Cai, J. Zhang, Z. Yan, and J. Wu, “Wideband Low-Profile Luneburg Lens Based on a Glide-Symmetric Metasurface,” IEEE Access 8, 85698–85705 (2020).
  * Söllner  _et al._ (2015) I. Söllner, S. Mahmoodian, S. L. Hansen, L. Midolo, A. Javadi, G. Kiršanskė, T. Pregnolato, H. El-Ella, E. H. Lee, J. D. Song, S. Stobbe, and P. Lodahl, “Deterministic photon–emitter coupling in chiral photonic circuits,” Nature Nanotechnology 10, 775–778 (2015).
  * Patil  _et al._ (2022) C. M. Patil, G. Arregui, M. Mechlenborg, X. Zhou, H. Alaeian, P. D. García, and S. Stobbe, “Observation of slow light in glide-symmetric photonic-crystal waveguides,” Optics Express 30, 12565–12575 (2022).
  * Iglesias Martínez  _et al._ (2022) J. A. Iglesias Martínez, N. Laforge, M. Kadic, and V. Laude, “Topological waves guided by a glide-reflection symmetric crystal interface,” Physical Review B 106, 064304 (2022).
  * Laude  _et al._ (2023) V. Laude, J. A. I. Martínez, N. Laforge, M. Kadic, and E. Prodan, “Glide-reflection symmetric phononic crystal interface: Variation on a theme,” Acta Mechanica Sinica 39, 723016 (2023).
  * Janković and Alù (2021) N. Janković and A. Alù, “Glide-Symmetric Acoustic Waveguides for Extreme Sensing and Isolation,” Physical Review Applied 15, 024004 (2021).
  * Huang  _et al._ (2016) X. Huang, Y. Yang, Z. H. Hang, Z.-Q. Zhang, and C. T. Chan, “Geometric phase induced interface states in mutually inverted two-dimensional photonic crystals,” Physical Review B 93, 085415 (2016).
  * Chen  _et al._ (2018) X.-D. Chen, D. Zhao, X.-S. Zhu, F.-L. Shi, H. Liu, J.-C. Lu, M. Chen, and J.-W. Dong, “Edge states in self-complementary checkerboard photonic crystals: Zak phase, surface impedance, and experimental verification,” Physical Review A 97, 013831 (2018).
  * Lu  _et al._ (2021) C. Lu, C. Wang, M. Xiao, Z. Q. Zhang, and C. T. Chan, “Topological Rainbow Concentrator Based on Synthetic Dimension,” Physical Review Letters 126, 113902 (2021).
  * Zhang  _et al._ (2022) P. Zhang, H. Jia, J. Lu, X. Yang, S. Wang, X. Xu, Y. Yang, Z. Liu, and J. Yang, “Multidimensional Interface Topology Induced by Acoustic Wannier Configurations,” Physical Review Applied 18, 064094 (2022).
  * Vaidya  _et al._ (2023) S. Vaidya, A. Ghorashi, T. Christensen, M. C. Rechtsman, and W. A. Benalcazar, “Topological phases of photonic crystals under crystalline symmetries,” Physical Review B 108, 085116 (2023).
  * Jackiw and Rebbi (1976) R. Jackiw and C. Rebbi, “Solitons with fermion number $\frac{1}{2}$,” Physical Review D 13, 3398–3409 (1976).
  * Huang  _et al._ (2014) X. Huang, M. Xiao, Z.-Q. Zhang, and C. T. Chan, “Sufficient condition for the existence of interface states in some two-dimensional photonic crystals,” Physical Review B 90, 075423 (2014).


# Factuality Challenges in the Era of Large Language Models

Isabelle Augenstein  University of Copenhagen, Nørregade 10, 1172 København,
Denmark  Timothy Baldwin  Mohamed bin Zayed University of Artificial
Intelligence, Masdar City, Abu Dhabi, 7909, United Arab Emirates  Meeyoung Cha
Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu,
Daejeon, South Korea  Tanmoy Chakraborty111Corresponding author, Email:
<EMAIL_ADDRESS>Indian Institute of Technology Delhi, New Delhi, 110016,
India  Giovanni Luca Ciampaglia  University of Maryland, College Park,
Maryland 20742, USA  David Corney  Full Fact, 17 Oval Way, London, SE11 5RR,
United Kingdom  Renee DiResta  Stanford University, 450 Jane Stanford Way,
Stanford, CA 94305, USA  Emilio Ferrara  University of Southern California,
Los Angeles, CA 90007, USA  Scott Hale  University of Oxford, Broad St, Oxford
OX1 3AZ, United Kingdom  Alon Halevy  Meta AI, 1 Hacker Way, Menlo Park, CA
94025, USA  Eduard Hovy  Carnegie Mellon University, 5000 Forbes Ave,
Pittsburgh, PA 15213, USA  Heng Ji  University of Illinois Urbana-Champaign,
506 S. Wright St. Urbana, IL 61801-3633, USA  Filippo Menczer  Indiana
University, 1015 E 11th St., Bloomington, IN 47408, USA  Ruben Miguez
Newtrales, C/Vandergoten 1, 28014 Madrid, Spain  Preslav Nakov  Mohamed bin
Zayed University of Artificial Intelligence, Masdar City, Abu Dhabi, 7909,
United Arab Emirates  Dietram Scheufele  University of Wisconsin, Madison, WI,
USA  Shivam Sharma  Indian Institute of Technology Delhi, New Delhi, 110016,
India  Giovanni Zagni  Pagella Politica/Facta, viale Monza 259/265, Milano,
20125, Italy

###### Abstract

The emergence of tools based on Large Language Models (LLMs), such as OpenAI’s
ChatGPT, Microsoft’s Bing Chat, and Google’s Bard, has garnered immense public
attention. These incredibly useful, natural-sounding tools mark significant
advances in natural language generation, yet they exhibit a propensity to
generate false, erroneous, or misleading content — commonly referred to as
_hallucinations_. Moreover, LLMs can be exploited for malicious applications,
such as generating false but credible-sounding content and profiles at scale.
This poses a significant challenge to society in terms of the potential
deception of users and the increasing dissemination of inaccurate information.
In light of these risks, we explore the kinds of technological innovations,
regulatory reforms, and AI literacy initiatives needed from fact-checkers,
news organizations, and the broader research and policy communities. By
identifying the risks, the imminent threats, and some viable solutions, we
seek to shed light on navigating various aspects of veracity in the era of
generative AI.

##  1 Introduction

In his pioneering work on information theory published in 1948[1], Claude
Shannon proposed simple statistical language models based on letter and word
frequencies capable of generating text that, although devoid of meaning,
resembled human language. With the vast amounts of digital text available for
training and thanks to advances in computation, large language models (LLMs)
today are capable of memorizing and reasoning about natural language[2]; they
can generate text that human readers find increasingly hard to distinguish
from actual human text. From early LLMs such as GPT[3] to more recent models
such as GPT-4[4] and LLaMA 2[5], several iterations of these technologies have
achieved unprecedented sophistication in natural language understanding and
generation ability. Despite the differences in their architectures, training
data, and algorithms, LLMs principally aim to address the same task, known as
“next-word prediction” — given a sequence of words as a prompt, what word(s)
should follow?

LLMs were brought to the general public’s attention in late 2022 with the
release of OpenAI’s ChatGPT[6], a chatbot based on LLM technology specifically
trained to generate responses as part of a conversation. Analyst reports found
that ChatGPT reached 100 million monthly active users only two months after
its launch, making it the fastest-growing consumer application in history[7].
Subsequent innovations by large tech companies such as Microsoft, Meta, and
Google[8] pushed the technology’s profile even further. These were then
followed by Alpaca[9] and Vicuna[10], open-source alternatives created by
instruction-tuning Meta’s LLaMA base model[5] with conversations from ChatGPT.
Similarly, Claude[11], Falcon[12], Jurassic[13], and Jais[14] were developed
through instruction-tuning of their own individual LLM models.

The proliferation of LLMs has led to concerns about an arms race in Generative
AI (GenAI) among leading tech companies. In 2020, prior to eventually open-
sourcing the second release of their generative pre-trained transformer (GPT)
series, GPT-2[15], OpenAI cited the potential for societal harm as one of the
reasons for not releasing the source code of their first model, GPT[16]. Since
then, the industry has largely abandoned such a cautious stance. As a result,
LLMs are rapidly gaining popularity among a vast number of professionals and
everyday people with limited or no technical knowledge. On the research front,
a recent survey indicated a significant uptick in the number of papers
containing the term “large language model” in their title or abstract [17].

The year 2022 brought a dramatic increase in the parameter size of these
models, reminiscent of Moore’s law, and a qualitative leap in commonsense
reasoning [17] and knowledge generalization [18]. Some studies also point out
that LLMs with billion-scale parameters have _emergent abilities_ that were
not present in smaller models[19]. Training these large models requires
extensive GPU resources. As a result, a handful of well-funded tech companies
and organizations dominate technological advances. This makes LLMs difficult
to study from outside. Nonetheless, testing and validating technology is
critical to ensuring its reliability, safety, and effectiveness, especially
for fast-adopted and widely-used technologies like LLMs.

We focus on one of the potential risks of LLMs in generating inaccurate,
misleading, false, or entirely fabricated content. Even though LLMs are
developed with good intentions and applied to perform useful tasks, they can
generate factually false statements, known as hallucinations[20] and
hallucinatory explanations in defense of such statements. Although widely
used, the term “hallucination” may be misleading and encourages
anthropomorphism[21]. Thus, discerning fact from fiction in LLM-generated
content is difficult, and it is further complicated by the use of eloquent
language and confident tone[22].

The implications of LLM hallucinations and the resulting unsolicited
justifications are especially concerning in critical and sensitive fields. For
example, chatbots may become particularly appealing in information-seeking and
question-answering tasks related to public health, given that it is easier for
many people to receive health advice from a chatbot than from a physician[23].
Yet, LLMs typically lack access to updated and reliable information sources. A
study involving 6,594 real-world chatbot interactions during the COVID-19
pandemic revealed that 30% of users queried using the keyword “COVID-19”[24].
This is concerning, given the unsuitability of present chatbot systems in
addressing emerging topics such as COVID-19, both before the pandemic[25] and
in hindsight[26]. A recent expert evaluation revealed that ChatGPT ended up
giving a “definitive answer to something that is not totally agreed on” on the
subject of dialysis for lithium poisoning[26]. Although advanced agents such
as ChatGPT, given their release timing, could potentially provide solid
answers to basic health questions with well-established scientific
consensus[27], they are ill-equipped to handle evolving scientific consensus
on controversial or emerging topics.

An even larger concern arises when LLMs are used with malicious intent. There
have been several discussions about applications of LLMs that put individuals
at risk, such as the generation of inauthentic or misleading content at
scale[28]. For example, when ChatGPT was prompted to improve the language of a
poorly written phishing email, it not only corrected it but also, without any
direct user request, added additional content at the end, asking for
money[29]. These capabilities will likely extend beyond mere “proof-of-
concept” demonstrations, underscoring the risk of propagating false or
manipulative information, deliberately or accidentally, with negative
consequences for politics, finance, health, and society at large.

LLM-based chatbots also pose challenges to fact-checking. Fact-checkers now
need to vet a surge of persuasive yet unreliable text, especially false news
generated with the help of LLMs[30] and spread through social media, including
by LLM-based social bots[31]. There are also questions about whether and how
the public can over-rely on chatbots such as ChatGPT as research tools for
verifying information[32, 33]. Search engines such as Google and Bing have
historically been seen as reliable gateways to authoritative information
sources; while in certain cases, search engines have helped amplify
disinformation[34], they are more reliable as they give a clear indication of
their sources, unlike ChatGPT. However, as LLM-based chatbots become
increasingly used for information-seeking, there is a risk that the public may
receive unreliable information through a modality that has traditionally been
trustworthy[32]. Recent incidents have highlighted the dangers of this trend,
such as Google Bard falsely claiming the James Webb telescope’s discovery of
an exoplanet image[35] and Replika disseminating misinformation by suggesting
that Bill Gates was the architect of COVID-19[36]. These occurrences
underscore the delicate balance between the risks and benefits of such
advanced technologies.

In summary, LLMs have the potential to mislead the public in novel ways, often
in combination with or within environments such as search engines, in which
users have come to expect (and assume) accuracy. Moreover, the datasets used
to train these models can introduce biases, magnifying specific viewpoints
while suppressing others[37], thus raising concerns about their widespread
adoption. This article explores how LLMs like ChatGPT affect fact-checking and
truthfulness, emphasizing the tangible risks associated with their
hallucinatory capabilities and prospects for malicious use. It also discusses
potential opportunities and strategies for addressing these challenges and the
limitations of current solutions, concluding with key objectives for
individuals, organizations, and governments, advocating the harnessing of LLM
benefits while reducing the risks.

##  2 Risks, Threats, and Challenges

There are two sides to the information veracity problem of LLMs. On the one
hand, LLMs developed with good intentions may nevertheless be plagued by
unreliable information. This could be due to unreliable, inconsistent,
incomplete, or missing training data or due to hallucinations. As a result,
chatbots or AI-enhanced search may give inaccurate responses on sensitive
topics such as finance or health, as they may generate false claims or
explanations. This is a risk because users may act upon, share, and/or quote
these responses later.

On the other hand, AI chatbots may be intentionally used for malicious
services or activities, such as scam emails, disinformation websites, or bot
feeds. AI-generated fabricated news websites, for example, are known to have
tens of thousands of followers on social media, directly reaching social media
users worldwide. The two sides are not independent of each other;
disinformation from low-credibility sources supported by malicious GenAI
applications is made available as training data for LLMs[38]. While the focus
of this article is on textual output, it is worth noting that AI tools with
visual capabilities, such as Dall-E[39], MidJourney[40] and Stable
Diffusion[41] also pose substantial challenges by enabling the creation of
deepfakes and false images at an unprecedented scale[42]. Malicious actors can
combine fake visuals with AI-generated text to support false claims or to
create fake social media profiles on a large scale.

Traditional fact-checking has evolved to ensure the reliability of
information[43], but GenAI poses real threats. The hidden, private nature of
chatbot conversations and the potential for misinformation underscore the
importance of AI literacy and user awareness. Below, we briefly outline
multiple risk factors, imminent threats, and the associated challenges that
arise from direct interaction with chatbots and their deliberate malicious
use.

###  2.1 Factuality Challenges

The ubiquitous adoption of AI chatbots can aggravate one of our most pressing
challenges — the quality and reliability of information in the digital age.
Chatbots can generate false or misleading content that can be quickly
disseminated and amplified through various platforms and channels. Some of the
main limitations and risks of chatbots are outlined below.

#### Undersourcing:

LLM-generated content tends to be coherent but lacks credible sourcing[26].
This issue extends beyond chatbots to generative search engines. Studies
scrutinizing generative search engines also found significant problems with
proper citations, with nearly half of the generated sentences lacking
citations and only three-quarters of those that have them actually supporting
their claims[44]. These findings raise concerns, considering the growing trust
of consumers in their reliability.

#### Truthfulness:

Despite their significant strides in natural language generation, LLMs tend to
generate undesirable text hallucinations that include nonsensical or
significantly divergent output, incoherent content, and factual
inaccuracies[45]. Although LLMs excel in creative writing and basic
explanations, they struggle with factual accuracy. This has led to their
temporary ban on platforms such as Stack Overflow[46]. Multiple studies
highlighted ChatGPT’s overall lack of trustworthiness[26]. In clinical
contexts, it demonstrated higher accuracy (80%) in clinical questions compared
to evidence-based ones (36%)[47]. In another study, ChatGPT was shown to
perform better in basic clinical science tests but less effectively in
specialized domains, while GPT-4 showed greater consistency between various
types of tests[48]. In summary, ChatGPT is not a reliable substitute for
actual physicians. Overall, LLMs are better at deductive reasoning and less so
at inductive reasoning[20], which could affect their reliability for fact-
based reasoning tasks[49].

#### Confident Tone:

The text generated by chatbots exudes confidence, which often makes them
appear as “authoritative liars.” Their persuasive narratives can deceive
readers into believing false or meaningless information due to the absence of
uncertainty or hedging expressions. Language models tailored for conversation
tend to maintain a confident tone even when their accuracy is compromised.
Addressing this challenge is difficult because LLMs lack measures of inherent
factuality and ways to express related uncertainty. While chatbots such as
ChatGPT attempt to emulate uncertainty, most safety checks rely on hard-coded
rules based on learned prompts, which can be easily circumvented.

#### Fluent Style:

Chatbots communicate from the first person’s perspective and even digress at
times like humans being, which encourages anthropomorphism. In addition, the
coherent and fluent writing style of chatbots can be persuasive to humans,
even on controversial political issues[50]. Numerous studies have shown that
fluency shapes the perception of truth “among intelligent people, despite
contradictory knowledge, for claims from unreliable sources” [51]. Further
studies are needed to examine how users perceive chatbots and their
capabilities in terms of AI literacy.

#### Direct Use:

In traditional settings, misinformation claims gain prominence and eventually
catch the attention of fact-checkers who can debunk them. However, chatbot
interactions are proprietary and occur privately and without mediation, which
makes it harder to discern their validity. The misinformation a chatbot
conveys may be buried among many other true and false claims, adding to the
difficulty of detection and correction. WhatsApp has partnered with a number
of fact-checking organisations to create tiplines[52], where users can
manually request verification of suspicious claims. Still, this manual process
risks being overwhelmed by GenAI content.

#### Ease of Access:

Competition in LLMs is driving fast technological advances. Meta’s LLaMA,
which came out as a competitor for ChatGPT, is now accessible for personalized
use on standard laptop hardware[53] unlike ChatGPT, which is restricted via a
rate-limited API. Recent open-sourcing of models such as LLAMA 2[5],
Falcon[54], and Jais[14] offer freely downloadable alternatives to encourage
transparent and democratized technological development. The ease of access to
LLMs can accelerate innovation, but it poses significant risks and challenges
when used with malicious intent, as discussed in Section 2.2. Tracing such
malicious use will be more challenging with the growing diversity of available
models.

#### Halo Effect:

A model’s proficiency in addressing one topic might lead users to believe it
can excel in any open-domain conversations, an example of a cognitive bias
known as the _halo effect_.[55]. Such an assumption is risky for users who
seek information on new and urgent topics. For example, a chatbot trained
before 2019 may not know anything about COVID-19 and can give wrong responses
during a health crisis. Even a large, up-to-date training dataset only
reflects a tiny and biased selection of human knowledge.

#### Public Perception:

An LLM may seem like a reliable “knowledge base” to the general public. This
can lead some users to consider LLMs as enhanced versions of search engines
capable of delivering the best answers. However, this is risky, as
demonstrated by a recent incident in which an attorney presented to the court
fake cases generated by ChatGPT[56]. Responses presented in the form of
answers, rather than a collection of diverse links offering varying
perspectives, may produce either inaccurate or biased information. Therefore,
users must learn how LLMs operate and not trust responses blindly.

#### Unreliable Evaluation:

Another pressing issue is the unreliable evaluation of LLMs[57]. Assessing
subjective factors such as “factuality” and “truthfulness” is a complex task
beyond predefined and annotated benchmarks such as BIG-bench[58], GLUE[59],
and SuperGLUE[60]. Specialized datasets have been developed to measure the
factuality of LLMs, such as TruthfulQA[61], which contains expert-crafted
questions that quantify model misconception on the topics of health, law,
finance, and politics. There have also been some specialized evaluation
measures such as FactScore, which evaluate factuality with respect to
Wikipedia (e.g., biography pages)[62]. Benchmark datasets alone are
insufficient, as LLMs may have been polluted by encountering the same or
similar data during their training [63]. Newer metrics such as GPTScore[64],
G-Eval[65], and SelfCheckGPT[66] attempt to address this shortcoming in
evaluations, albeit with limitations due to aspects such as positional and
numeric biases, stochastic inferencing, and self-preferencing[67]. As of now,
LLMs have demonstrated mixed performance on misinformation detection tasks[20,
33], involving test sets consisting of scientific and social claims related to
COVID-19[68]. Maintaining accurate ground-truth references, particularly for
factuality assessment, is costly, subject to data drift, and remains an open
research problem.

###  2.2 Threats Posed by Malicious LLM Usage

The widespread availability of LLMs empowers people with ideas to craft
compelling and elegantly written content and persuasive arguments, a skill
traditionally held by experienced writers. Even though all technologies have
both good and bad applications, it is important to foresee potential harmful
applications of LLMs, as outlined below.

#### Personalized Attacks:

LLMs can generate text that aligns with the context of the ongoing
conversation. This capability can be exploited for malicious purposes, where a
user’s prior statements, such as those from emails or social media, can be
incorporated into a prompt to generate disinformation, phishing messages,
harassment or other harmful content on a large scale. The content can be
tailored to appear credible, personalized, and targeted at specific
individuals or groups and can be mixed with factual statements to make it more
persuasive. The possibility of data leakage increases these risks. Cyberhaven,
a firm that provides data tracing and security solutions, recently analyzed
the engagements of their clients’ employees with ChatGPT[69] and found that
10.8% had used ChatGPT at work, 8.6% had shared company data via prompts, and
4.7% had accidentally disclosed confidential information. The risk of data
leakage is evidenced by the inadvertent disclosure of payment details from
1.2% of ChatGPT Plus subscribers. Open-source LLMs can be manipulated to
extract such private information, which can then be used for more effective
phishing attacks and other scams.

#### Style Impersonation:

Fine-tuned LLMs will be able to generate text that emulates the style of any
person, providing access to relevant training data. Consequently, it will be
relatively simple to train a model to generate text that mimics the writing
style of specific individuals, such as journalists, fact-checkers,
politicians, regulators, and more. This content could then be distributed on
social media platforms in an effort to undermine the credibility of perceived
adversaries.

#### Bypassing Detection:

Fact-checkers prioritize monitoring and verifying widely circulated claims.
For example, a piece of misinformation that has circulated a thousand times is
scrutinized more than once by only ten people. However, GenAI tools can
generate infinite variations of the same content. Even if each variant reaches
a small number of people, the cumulative impact could add to that of a highly
viral piece while remaining invisible to fact-checkers. This automated
diversification could undermine the safety and integrity efforts of many
social media platforms, such as Meta’s third-party fact-checking program[70].

#### Fake Profiles:

The ability to generate credible-sounding fake profiles and content at scale
will impact social media influencing, making social bots a more formidable
challenge. These fake profiles will empower bad actors to infiltrate and
manipulate online communities more easily; automation will reduce costs while
increasing output quality. Fake accounts can be used to spread illicit or
manipulative content, such as disinformation and hate speech, creating a
skewed perception of popular opinions and norms[71]. We have already observed
a large network of ChatGPT-driven fake profiles on Twitter/X[31]. These
deceptive social bots engaged with each other through replies and retweets,
created the false appearance of broad support for fake news sources and posted
harmful comments while evading detection. Social media users are less likely
to fact-check and more likely to share false claims when those claims appear
to be liked or shared by many others[72]. This creates a significant
vulnerability to manipulation by fake profiles. Potential harm could even
extend to life-threatening matters. For instance, a strong correlation has
been reported between the volume of online COVID-19 vaccine misinformation and
a reduction in vaccination rates[73]. GenAI can be quickly weaponized to
amplify such harm at scale.

##  3 Addressing the Threats

Addressing actuality-related challenges in LLMs requires a comprehensive
strategy, as no single solution can fully mitigate the adverse consequences.
Here, we outline various strategic dimensions that, when combined, may lead to
more responsible and constructive technological utilization. Some solutions
are technological and require building an entirely new LLM. Training a multi-
billion-parameter LLM from scratch takes several months and hundreds of GPUs,
which is beyond the reach of most academics. However, smaller models, such as
those of LLaMA, Falcon, or Jais, are feasible in academia. For example,
running a 7B model needs a single GPU, while a 13B model needs two GPUs.

#### Alignment and Safety:

Safety and aligning LLMs with human values and intent have become a major
concern for recent models like ChatGPT, LLaMA 2, and Jais. In fact, safety
measures are increasingly being considered in all stages of chatbot
development — data cleansing before training the base model, safety
instruction-tuning[74], safety in the hidden prompt to the chatbot, and safety
in the deployed chatbot via keywords and machine learning. The availability of
open-source LLMs suggests that the effectiveness of alignment efforts, as well
as other countermeasures, such as watermarking by large AI companies may be
severely limited in mitigating the potential looming threats. Nevertheless, it
remains crucial to make every conceivable effort that holds promise in
restraining the counterproductive consequences of LLMs.

#### Modularized Knowledge Grounded Framework:

One area where current LLMs significantly fall short is in producing timely,
thorough, and well-organized presentations of factually dense information,
such as situational and strategic reports. One way to ameliorate this
shortcoming in the context of pre-trained LLMs involves a multi-step automated
framework for gathering and organizing real-time event information. This
modular design can create factually accurate content, which can be further
refined using an LLM, as exemplified by SmartBook[75]. Initially developed for
efficient ground-level reporting during the Russia-Ukraine conflict, SmartBook
used LLMs to generate initial situational reports by streamlining event-based
timelines, structured summaries, and consolidated references.

#### Retrieval-Augmented Generation:

Retrieval-augmented generation (RAG)[76, 77] incorporates contextual
information from external sources into text generation. RAG mitigates the
challenge of LLMs producing inaccurate content by enhancing their capabilities
with external data. However, it requires efficient retrieval of grounded text
at scale and robust evaluation.

#### Hallucination Control and Knowledge Editing:

There are two types of LLM hallucinations [78]: (i) faithfulness, when the
generated text is not faithful to the input context; and (ii) factualness,
when the generated text is not factually correct with respect to world
knowledge. Most recent attempts focus on solving the hallucination problem
during the inference stage based on an LLM’s self-consistency checking [79],
cross-model verification [80, 81], or checking against related knowledge [82].
The assumption is that an LLM has knowledge of a given concept, and sampled
responses are likely to be similar and contain consistent facts. Another
promising line of research focuses on opening up LLMs for knowledge editing.
Factual errors can then be localized and fixed by injecting factual updates
into the model [83, 84, 85, 86, 87]. Existing methods focus on factual updates
for triples by precise editing. This can be extended to more complex knowledge
representations, such as logical rules in the future. Another challenge is to
evaluate the “ripple effects” of knowledge editing in language models. Current
knowledge editing benchmarks check that a few paraphrases of the original fact
are updated and some unrelated facts are untouched. More research must explore
whether other facts logically derived from the edit are also changed
accordingly.

#### Alleviating Exposure Bias:

Exposure bias, i.e., favoring preexisting inductive biases over new ones,
persists as a challenge in natural language generation, affecting the output
quality of LLMs trained on fixed datasets[88]. Solutions such as selective
upgrade, which dynamically derives relevant instruction-response pairs, aim to
address this by improving the ability of LLMs to generalize effectively beyond
their training data[89].

#### Better Evaluation:

Existing evaluation methods such as BERTScore[90] and MoverScore[91] assume
similar training and evaluation data distributions, which do not align with
the evolving capabilities and requirements of LLMs. This discrepancy is
particularly evident in scenarios involving zero-shot instructions and in-
context learning, which prominently constitute disparate data distribution
scenarios. Recently proposed evaluation measures such as GPTScore[64] and
G-Eval[65] have shown reasonable correlations with human assessments in
various tasks, including consistency, accuracy, and correctness. However, weak
correlations (around 20–25%) remain in factuality assessments[64], suggesting
that there is room for improvement. One potential direction is customizing
factuality instructions for specific domains, such as medicine or law. Similar
adjustments have demonstrated improved factuality assessment in the case of
SelfCheckGPT[66], which is based on the idea that consistently replicable
responses are rooted in factual accuracy, as opposed to those generated
through stochastic sampling or hallucination, which tend to exhibit more
variation.

#### Privacy and Data Protection:

While organizations like OpenAI have implemented privacy controls and undergo
periodic cybersecurity audits[92], user studies call for AI systems to adhere
to additional data protection regulations, emphasizing steps like data
anonymization, aggregation, and differential privacy[93]. While the European
Union’s AI Act[94] has transparency-related obligations based on identified
risk levels, stronger regulation might facilitate access to third-party fact-
verification APIs/plugins. Chatbot users could then opt-in to use such tools
whenever verification is critical to the conversation.

#### Recognizing AI-Generated Content:

LLM output is already nearly indistinguishable from human-written text[95].
For example, state-of-the-art tools to detect AI-generated content failed to
distinguish between legitimate Twitter/X accounts and those managed through
ChatGPT[31]. Furthermore, previous misinformation detectors [96] trained from
automatically generated fake news training data still perform poorly on
detecting human-generated or edited fake content [97]. Future models will
likely narrow this gap further, making attempts to label AI-generated text
unreliable. Watermarks will also be easy for malicious actors to bypass.
Therefore, it is important to study multiple generators in multiple domains,
for multiple languages, and using different detectors, and also to maintain
constantly growing collections of up-to-date machine-generated content as in
the M4 repository[95]. This issue parallels the rapid advancement of deepfake
technology in various media formats, including fake videos, manipulated
images, altered audio clips, and stylized text[98, 99]. The emergence of
sophisticated tactics, such as paraphrasing attacks aimed at evading detection
[99] and adversarial perturbations resistant to image and video compression
codecs[100], underscore the pressing need to address these concerns. Devising
a robust and fool-proof detection technique, akin to a cat-and-mouse game,
remains a formidable challenge.

#### Content Authenticity and Provenance:

As AI-generated text becomes ubiquitous, fully human-written text may become
more valuable. Technologies and standards for content authenticity and
provenance already exist for video and image content[101]. These standards
describe a way to cryptographically sign content so that the metadata around
its creation can be proven not altered. We could use similar methods for text
content to prove that they are not AI-generated. Since AI-generated content
can cause harm when it spreads on social media, provenance proofs could be
imposed to limit the spread of fake content before it has reached many
people[28].

#### Regulation:

Various regulatory efforts have emerged in light of the disruptive impact of
emerging technologies. A new rule in China requires watermarks for AI-
generated content, while ChatGPT was temporarily banned in Italy due to GDPR
compliance concerns before being finally allowed after due compliance on
transparency[102]. The European Union’s AI Act is likely the first to regulate
high-risk AI applications spanning multiple sectors[94]. In the USA, the
Federal Trade Commission has issued warnings against creating misleading
tools, emphasizing their prohibition under existing regulations and signaling
the potential application of these rules to GenAI[103]. Along similar lines,
the Canadian Directive on Automated Decision-Making has prescribed extensive
guidelines that promote data-driven practices and federal compliance,
transparency, and reduced negative algorithmic outcomes[104]. One of the
challenges to the effectiveness of AI regulation stems from rapid
technological advancements. On the one hand, controlling LLMs and their users
can be as challenging as handling individuals engaged in phishing and
misinformation. On the other hand, bad actors using open-source models will
not be bound by regulation.

#### Public Education:

Public awareness of deceptively “slick” content is essential, similar to our
skepticism towards images doctored through Photoshop. This awareness extends
to deepfake visual technology. A constructive way for experts to contribute to
such education is through resources such as tutorial videos and code,
explaining the basic technology behind ChatGPT. However, raising awareness
about LLM-based chatbots is challenging, as the risks they pose demand
immediate attention, unlike the gradual understanding of deepfakes for visual
content that has developed over the last few years. Another caveat is that
skeptical citizens may lose trust in credible, authoritative sources of
information, and thus become more vulnerable to conspiracy theories.

##  4 Fact-Checking Opportunities

Leveraging LLMs to foster factuality is as important as addressing the
challenges of LLMs. In fact, one could argue that verifying the truthfulness
of a claim may be more important than detecting whether it is AI-generated.
LLMs present several promising opportunities for fact-checkers and
journalists, some of which are highlighted below. It is important, however, to
remain vigilant about the challenges and the ethical considerations associated
with such applications of LLMs, including bias, transparency, and the
potential for misuse. Balancing the advantages of LLMs with responsible and
ethical practices is essential to harness their full potential in domains such
as fact-checking.

#### Fact-Checking Support:

Although GenAI models lack a concept of truth, they can be used to assist
fact-checkers in verifying claims. LLMs can transcribe speeches, debates,
interviews, online videos, and news broadcasts, summarize extensive documents,
and help create concise lists of crucial claims [105, 106, 107]. These claims
can be further organized into fine-grained claim frames, and the claim-claim
relations can be utilized for cross-media cross-lingual fact checking [108]
and factual error correction [109]. This capability is particularly valuable
for processing large volumes of news content, enabling fact-checkers to
monitor a vast amount of potentially misleading information. After a fact-
checked article has been published, the same claims often reappear. LLMs can
help identify sections of documents that repeat a previously fact-checked
claim or that make a claim semantically equivalent to a previously verified
one. This task does not involve determining the factuality or the falsehood of
the claim, as this was already established by fact-checkers or
journalists[43]. Such promising opportunities for LLMs to support fact-
checking come with some caveats. First, there are risks of errors when
summarizing long documents, such as omitting important context or inserting
plausible text that is outside the source material. Second, we need to
understand how humans will interact with AI-aided fact-checking. A recent
study found that fact-checks generated by ChatGPT, even when accurate, did not
significantly affect participants’ ability to discern headline accuracy or to
share accurate news[33]. Worse, they were harmful in specific cases, such as
increasing beliefs in certain false claims. These findings highlight the
importance of evaluating unintended consequences of GenAI applications.

#### Stance Detection:

There are indications that ChatGPT and other LLMs can help with downstream
fact-checking tasks, such as stance detection[110]. On the other hand, whether
the LLM has seen the test data, as it was trained on many publicly available
datasets remains as a partinent question. Recent studies suggest that ChatGPT
is more inclined to excel in stance detection tasks compared to tasks
involving emotions or pragmatic analyses, even though it might not have
undergone pre-training on evaluation sets explicitly tailored for stance
detection tasks[111].

#### Domain-Specific Chat Support:

There have been attempts to use LLMs to provide a chatbot interface to a
domain-specific controlled corpus[112]. Organizations could provide such a
service to allow users to search for information on a collection of factually
verified articles. Some caution is needed, as LLMs can insert plausible but
incorrect information even with a controlled corpus.

##  5 Conclusion

Any tool capable of generating novel content also has the potential to produce
misleading content[113]. Therefore, anyone using LLMs to compose news
articles, academic reports, emails, or any text must verify even the most
basic facts, regardless of how fluent the text appears. Given the rapid and
widespread growth in the use of LLMs, society must act quickly with
appropriate regulation, education, and collaboration. Below, we propose an
urgent agenda for individuals, governments, and democratic societies.

#### Coordination and Collaboration:

Towards ensuring the responsible development and deployment of GenAI
technologies, nations must make coordinated research investments and establish
infrastructure capable of dynamically adapting legislative safeguards, much
like the measures in place for other groundbreaking technologies such as human
germline editing[114]. Moreover, fostering collaborations between political
actors and industry leaders is essential to prevent an arms race between AI
and AI-detection tools. By coordinating these efforts globally, we can promote
transparent, constructive, and responsible technological development, ensuring
that GenAI benefits humanity while minimizing potential risks and abuses.

#### Regulations:

Key components of comprehensive regulations in the realm of GenAI include
stringent law enforcement to mitigate intentional or inadvertent harm from
technology use[115], regulatory frameworks for high-risk tech production and
sales[103], and industry-wide standards for ethical GenAI usage. Special
attention should be paid to creating guidelines for journalists who use GenAI
and for labeling public ads generated by AI. Adopting standardized evaluation
measures tailored to critical sectors will also be vital to ensuring that
technology benefits society while minimizing the risks[64, 48].

#### Promoting AI Literacy:

To promote AI literacy globally, we recommend three key actions: (i)
conducting AI literacy programs for people of all ages[100], (ii)
incorporating AI education with an emphasis on ethics into the graduate-level
curricula[116], and (iii) sensitizing digital consumers to the potential harms
and root causes of GenAI[117]. These measures will empower individuals to
engage with AI responsibly and safely, fostering a more knowledgeable and
aware global community.

#### Technological Development:

As common R$\&$D best practices, we recommend integrating clear and accessible
informative material for users upfront to clarify the limits (e.g., in terms
of factuality) and the risks (e.g., because of the “authoritative” tone) of
LLMs. To enhance safety, robust guardrails should be implemented in LLM-based
conversations[118]. Chatbots should be equipped with evidence-supporting
capabilities[119] to bolster credibility, while external knowledge sources
like retrieval-augmented LLMs and knowledge graphs can ensure factual
consistency[120]. Lastly, the development of coordination detection algorithms
is essential to identify suspiciously similar and potentially harmful
content[121] generated by malicious actors using LLMs[28].

## Author Contributions

All the authors contributed to conceptualizing, preparing, and finalizing the
manuscript. The author names are arranged alphabetically by the last names.

## Funding Information

M.C. is supported by the Institute for Basic Science (grant IBS-R029-C2) and
the National Research Foundation of Korea (grant RS-2022-00165347). T.C.
acknowledges the financial support of Wipro AI. I.A. is supported in part by
the European Union (ERC, ExplainYourself, 101077481). G.L.C. is supported by
the National Science Foundation (grant 2239194). E.F. and F.M. are partly
supported by DARPA (grant HR001121C0169). F.M. is also partly supported by the
Knight Foundation and Craig Newmark Philanthropies. G.Z.’s fact-checking
project receives funding from the European Union through multiple grants and
is part of Meta’s 3PFC Program. H.J.’s research is partially supported by U.S.
DARPA SemaFor Program No. HR001120C0123. Any opinions, findings, conclusions,
or recommendations expressed in this material are those of the authors and do
not necessarily reflect the views of the funders.

## Competing Interests

The authors declare no competing interests.

## Additional Information

Materials & Correspondence should be emailed to Tanmoy Chakraborty
(tanchak@iitd.ac.in).

## References

  * [1] Shannon, C. E.  A mathematical theory of communication.  _The Bell System Technical Journal_ 27, 379–423, DOI: 10.1002/j.1538-7305.1948.tb01338.x (1948).
  * [2] Huang, J. & Chang, K. C.-C.  Towards Reasoning in Large Language Models: A Survey.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , 1049–1065, DOI: 10.18653/v1/2023.findings-acl.67 (Association for Computational Linguistics, Toronto, Canada, 2023).
  * [3] Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. _et al._ Improving language understanding by generative pre-training (2018).
  * [4] OpenAI.  GPT-4 technical report.  _arXiv:2303.08774_ (2023).
  * [5] Touvron, H. _et al._ Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ (2023).
  * [6] Introducing ChatGPT — openai.com.  https://openai.com/blog/chatgpt.  [Accessed 27-09-2023].
  * [7] Hu, K.  ChatGPT sets record for fastest-growing user base - analyst note.  _Reuters_ (2023).
  * [8] Thoppilan, R. _et al._ Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ (2022).
  * [9] Taori, R. _et al._ Stanford Alpaca: An instruction-following LLaMA model.  https://github.com/tatsu-lab/stanford_alpaca (2023).
  * [10] Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.  https://lmsys.org/blog/2023-03-30-vicuna/.  [Accessed 15-09-2023].
  * [11] Introducing Claude.  https://www.anthropic.com/index/introducing-claude.  [Accessed 15-09-2023].
  * [12] Falcon.  https://falconllm.tii.ae/falcon.html.  [Accessed 15-09-2023].
  * [13] Jurassic-2 models.  https://docs.ai21.com/docs/jurassic-2-models.  [Accessed 20-09-2023].
  * [14] Sengupta, N. _et al._ Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models.  _arXiv preprint 2308.16149_ (2023).  2308.16149.
  * [15] Radford, A. _et al._ Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 9 (2019).
  * [16] Brown, T. _et al._ Language Models are Few-Shot Learners.  In _Advances in Neural Information Processing Systems_ , vol. 33, 1877–1901 (Curran Associates, Inc., 2020).
  * [17] Zhao, W. X. _et al._ A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).
  * [18] Li, S. _et al._ Open-domain hierarchical event schema induction by incremental prompting and verification.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 5677–5697, DOI: 10.18653/v1/2023.acl-long.312 (Association for Computational Linguistics, Toronto, Canada, 2023).
  * [19] Wei, J. _et al._ Emergent abilities of large language models.  _Trans. Mach. Learn. Res._ 2022 (2022).
  * [20] Bang, Y. _et al._ A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity.  Preprint 2302.04023, arXiv (2023).  DOI: 10.48550/arXiv.2302.04023.
  * [21] Bergstrom, C. T. & Ogbunu, C. B.  ChatGPT isn’t ‘hallucinating.’ It’s Bullshitting.  https://undark.org/2023/04/06/chatgpt-isnt-hallucinating-its-bullshitting/ (2023).  Accessed 2 October 2023.
  * [22] Sison, A. J. G., Daza, M. T., Gozalo-Brizuela, R. & Garrido-Merchán, E. C.  ChatGPT: More Than a “Weapon of Mass Deception” – Ethical Challenges and Responses from the Human-Centered Artificial Intelligence (HCAI) Perspective.  _International Journal of Human-Computer Interaction_ 0, 1–20, DOI: 10.1080/10447318.2023.2225931 (2023).  https://doi.org/10.1080/10447318.2023.2225931.
  * [23] Iftikhar, L. _et al._ Docgpt: Impact of chatgpt-3 on health services as a virtual doctor.  _EC Paediatrics_ 12, 45–55 (2023).
  * [24] Chin, H. _et al._ User-chatbot conversations during the COVID-19 pandemic: Study based on topic modeling and sentiment analysis.  _Journal of Medical Internet Research_ 25, e40922, DOI: 10.2196/40922 (2023).
  * [25] Palanica, A., Flaschner, P., Thommandram, A., Li, M. & Fossat, Y.  Physicians’ perceptions of chatbots in health care: Cross-sectional web-based survey.  _J Med Internet Res_ 21, e12887, DOI: 10.2196/12887 (2019).
  * [26] Peskoff, D. & Stewart, B.  Credible without credit: Domain experts assess generative language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 427–438, DOI: 10.18653/v1/2023.acl-short.37 (Association for Computational Linguistics, Toronto, Canada, 2023).
  * [27] Srivastava, B.  Did chatbots miss their “Apollo moment”? Potential, gaps, and lessons from using collaboration assistants during COVID-19.  _Patterns_ 2, 100308, DOI: https://doi.org/10.1016/j.patter.2021.100308 (2021).
  * [28] Menczer, F., Crandall, D., Ahn, Y.-Y. & Kapadia, A.  Addressing the harms of AI-generated inauthentic content.  _Nature Machine Intelligence_ 5, 678–680, DOI: 10.1038/s42256-023-00690-w (2023).
  * [29] Patel, A. & Sattler, J.  Creatively malicious prompt engineering.  Tech. Rep., WithSecure Labs (2023).
  * [30] Brewster, J., Wang, M. & Palmer, C.  Plagiarism-Bot? How Low-Quality Websites Are Using AI to Deceptively Rewrite Content from Mainstream News Outlets.  https://www.newsguardtech.com/misinformation-monitor/august-2023/ (2023).  [Accessed 23 Sep 2023].
  * [31] Yang, K.-C. & Menczer, F.  Anatomy of an AI-powered malicious social botnet.  Preprint 2307.16336, arXiv (2023).  DOI: 10.48550/arXiv.2307.16336.
  * [32] Verma, P. & Oremus, W.  ChatGPT invented a sexual harassment scandal and named a real law prof as the accused.  _Washington Post_ (2023).
  * [33] DeVerna, M. R., Yan, H. Y., Yang, K.-C. & Menczer, F.  Artificial intelligence is ineffective and potentially harmful for fact checking.  Preprint 2308.10800, arXiv (2023).  DOI: 10.48550/arXiv.2308.10800.
  * [34] Ferrara, E.  The history of digital spam.  _Commun. ACM_ 62, 82–91, DOI: 10.1145/3299768 (2019).
  * [35] Vincent, J.  Google’s AI chatbot Bard makes factual error in first demo (2023).
  * [36] Marcus, G.  Deep Learning Is Hitting a Wall (2022).
  * [37] Ferrara, E.  Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models.  _Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models_ (2023).
  * [38] Pan, Y. _et al._ On the risk of misinformation pollution with large language models.  _arXiv preprint 2305.13661_ (2023).  2305.13661.
  * [39] Dall.e 2.  https://openai.com/dall-e-2.  [Accessed 15-09-2023].
  * [40] Midjourney.  https://www.midjourney.com/.  [Accessed 15-09-2023].
  * [41] Stable diffusion.  https://stablediffusionweb.com/.  [Accessed 15-09-2023].
  * [42] Mirsky, Y. & Lee, W.  The Creation and Detection of Deepfakes: A Survey.  _ACM Comput. Surv._ 54, DOI: 10.1145/3425780 (2021).
  * [43] CLEF2022-CheckThat! — sites.google.com.  https://sites.google.com/view/clef2022-checkthat.
  * [44] Liu, N. F., Zhang, T. & Liang, P.  Evaluating verifiability in generative search engines (2023).  2304.09848.
  * [45] Ji, Z. _et al._ Survey of hallucination in natural language generation.  _ACM Comput. Surv._ 55, DOI: 10.1145/3571730 (2023).
  * [46] Vincent, J.  AI-generated answers temporarily banned on coding Q&A site Stack Overflow — theverge.com.  https://www.theverge.com/2022/12/5/23493932/chatgpt-ai-generated-answers-temporarily-banned-stack-overflow-llms-dangers.  [Accessed 08-09-2023].
  * [47] Kusunose, K., Kashima, S. & Sata, M.  Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension Guidelines.  _Circulation Journal_ 87, 1030–1033, DOI: 10.1253/circj.CJ-23-0308 (2023).
  * [48] Antaki, F., Touma, S., Milad, D., El-Khoury, J. & Duval, R.  Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings.  _Ophthalmology Science_ 3, 100324, DOI: https://doi.org/10.1016/j.xops.2023.100324 (2023).
  * [49] Abels, G.  Can ChatGPT fact-check? We tested. - Poynter — poynter.org.  https://www.poynter.org/fact-checking/2023/chatgpt-ai-replace-fact-checking/.  [Accessed 27-09-2023].
  * [50] Bai, H., Voelkel, J. G., Eichstaedt, j. C. & Willer, R.  Artificial intelligence can persuade humans on political issues, DOI: 10.31219/osf.io/stakv (2023).
  * [51] Brashier, N. M. & Marsh, E. J.  Judging truth.  _Annual review of psychology_ 71, 499–515 (2020).
  * [52] IFCN fact-checking organizations on WhatsApp.  https://faq.whatsapp.com/5059120540855664 (2023).  Accessed 2 October 2023.
  * [53] Running LLaMA 7B and 13B on a 64GB M2 MacBook Pro with llama.cpp — til.simonwillison.net.  https://til.simonwillison.net/llms/llama-7b-m2.  [Accessed 08-09-2023].
  * [54] Falcon LLM — falconllm.tii.ae.  https://falconllm.tii.ae/falcon-180b.html.  [Accessed 08-09-2023].
  * [55] Volunteering, C.  Halo effect — wikipedia.org.  https://en.wikipedia.org/wiki/Halo_effect.  [Accessed 27-09-2023].
  * [56] OpenAI.  Lawyer used ChatGPT in court—and cited fake cases. A judge is considering sanctions.  https://www.forbes.com/sites/mollybohannon/2023/06/08/lawyer-used-chatgpt-in-court-and-cited-fake-cases-a-judge-is-considering-sanctions/.  [Accessed 15-09-2023].
  * [57] Chakraborty, T. & Masud, S.  Judging the creative prowess of AI.  _Nature Machine Intelligence_ 1–1 (2023).
  * [58] Srivastava, A. _et al._ Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_ (2023).
  * [59] Wang, A. _et al._ GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , 353–355, DOI: 10.18653/v1/W18-5446 (Association for Computational Linguistics, Brussels, Belgium, 2018).
  * [60] Wang, A. _et al._ SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , 294 (Curran Associates Inc., Red Hook, NY, USA, 2019).
  * [61] Lin, S., Hilton, J. & Evans, O.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 3214–3252, DOI: 10.18653/v1/2022.acl-long.229 (Association for Computational Linguistics, Dublin, Ireland, 2022).
  * [62] Min, S. _et al._ FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.  _arXiv preprint 2305.14251_ (2023).  2305.14251.
  * [63] Golchin, S. & Surdeanu, M.  Time Travel in LLMs: Tracing Data Contamination in Large Language Models (2023).  2308.08493.
  * [64] Fu, J., Ng, S.-K., Jiang, Z. & Liu, P.  GPTScore: Evaluate as you desire (2023).  2302.04166.
  * [65] Liu, Y. _et al._ G-Eval: NLG evaluation using GPT-4 with better human alignment (2023).  2303.16634.
  * [66] Manakul, P., Liusie, A. & Gales, M. J. F.  SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models (2023).  2303.08896.
  * [67] Wang, P. _et al._ Large language models are not fair evaluators (2023).  2305.17926.
  * [68] Lee, N., Bang, Y., Madotto, A. & Fung, P.  Towards few-shot fact-checking via perplexity.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 1971–1981, DOI: 10.18653/v1/2021.naacl-main.158 (Association for Computational Linguistics, Online, 2021).
  * [69] Coles, C.  11% of data employees paste into ChatGPT is confidential - Cyberhaven — cyberhaven.com.  https://www.cyberhaven.com/blog/4-2-of-workers-have-pasted-company-data-into-chatgpt/.  [Accessed 08-09-2023].
  * [70] Meta’s Third-Party Fact-Checking Program | Meta Journalism Project — facebook.com.  https://www.facebook.com/formedia/mjp/programs/third-party-fact-checking.
  * [71] Truong, B. T., Lou, X., Flammini, A. & Menczer, F.  Vulnerabilities of the online public square to manipulation.  Preprint 1907.06130, arXiv (2023).  DOI: 10.48550/arXiv.1907.06130.
  * [72] Avram, M., Micallef, N., Patil, S. & Menczer, F.  Exposure to social engagement metrics increases vulnerability to misinformation.  _HKS Misinformation Review_ 1, DOI: 10.37016/mr-2020-033 (2020).
  * [73] Pierri, F. _et al._ Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal.  _Scientific Reports_ 12, 5966, DOI: 10.1038/s41598-022-10070-w (2022).
  * [74] Wang, Y., Li, H., Han, X., Nakov, P. & Baldwin, T.  Do-not-answer: A dataset for evaluating safeguards in llms.  _arXiv preprint 2308.13387_ (2023).  2308.13387.
  * [75] Reddy, R. G. _et al._ Smartbook: AI-assisted situation report generation (2023).  2303.14337.
  * [76] Yu, W. _et al._ A survey of knowledge-enhanced text generation.  _ACM Comput. Surv._ 54, DOI: 10.1145/3512467 (2022).
  * [77] Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M.-W.  Realm: retrieval-augmented language model pre-training.  _ArXiv_ (2020).
  * [78] Filippova, K.  Controlled hallucinations: learning to generate faithfully from noisy data.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 864–870, DOI: 10.18653/v1/2020.findings-emnlp.76 (Association for Computational Linguistics, Online, 2020).
  * [79] Gou, Z. _et al._ Critic: large language models can self-correct with tool-interactive critiquing.  In _arxiv_ (2023).
  * [80] Cohen, R., Hamri, M., Geva, M. & Globerson, A.  Lm vs lm: detecting factual errors via cross examination.  In _arxiv_ (2023).
  * [81] Du, Y., Li, S., Torralba, A., Tenenbaum, J. B. & Mordatch, I.  Improving factuality and reasoning in language models through multiagent debate.  In _arxiv_ (2023).
  * [82] Dziri, N., Madotto, A., Zaïane, O. & Bose, A. J.  Neural path hunter: reducing hallucination in dialogue systems via path grounding.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 2197–2214, DOI: 10.18653/v1/2021.emnlp-main.168 (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021).
  * [83] De Cao, N., Aziz, W. & Titov, I.  Editing factual knowledge in language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 6491–6506, DOI: 10.18653/v1/2021.emnlp-main.522 (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021).
  * [84] Mitchell, E., Lin, C., Bosselut, A., Finn, C. & Manning, C. D.  Fast model editing at scale.  _arXiv preprint arXiv:2110.11309_ (2021).
  * [85] Mitchell, E., Lin, C., Bosselut, A., Manning, C. D. & Finn, C.  Memory-based model editing at scale.  In _International Conference on Machine Learning_ , 15817–15831 (PMLR, 2022).
  * [86] Meng, K., Bau, D., Andonian, A. & Belinkov, Y.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ 35, 17359–17372 (2022).
  * [87] Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y. & Bau, D.  Mass-editing memory in a transformer.  _arXiv preprint arXiv:2210.07229_ (2022).
  * [88] Schmidt, F.  Generalization in Generation: A closer look at Exposure Bias.  In _Proceedings of the 3rd Workshop on Neural Generation and Translation_ , 157–167, DOI: 10.18653/v1/D19-5616 (Association for Computational Linguistics, Hong Kong, 2019).
  * [89] Yu, P. & Ji, H.  Self information update for large language models through mitigating exposure bias.  _arXiv preprint arXiv:2305.18582_ (2023).
  * [90] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_ (OpenReview.net, 2020).
  * [91] Zhao, W. _et al._ MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 563–578, DOI: 10.18653/v1/D19-1053 (Association for Computational Linguistics, Hong Kong, China, 2019).
  * [92] OpenAI.  Security & privacy.  https://openai.com/security.  [Accessed 27-09-2023].
  * [93] Sebastian, G.  Privacy and data protection in chatgpt and other ai chatbots: Strategies for securing user information.  _International Journal of Security and Privacy in Pervasive Computing_ 15, 1–14, DOI: 10.4018/IJSPPC.325475 (2023).
  * [94] EUR-Lex - 52021PC0206 - EN - EUR-Lex — eur-lex.europa.eu.  https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206.
  * [95] Wang, Y. _et al._ M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection.  _arXiv:2305.14902_ (2023).
  * [96] Fung, Y. _et al._ Infosurgeon: cross-media fine-grained information consistency checking for fake news detection.  In _Proc. The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021)_ (2021).
  * [97] Huang, K.-H., McKeown, K., Nakov, P., Choi, Y. & Ji, H.  Faking fake news for real fake news detection: propaganda-loaded training data generation.  In _Proc. The 61st Annual Meeting of the Association for Computational Linguistics (ACL2023) Findings_ (2023).
  * [98] Groh, M. _et al._ Human detection of political speech deepfakes across transcripts, audio, and video (2023).  2202.12883.
  * [99] Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W. & Feizi, S.  Can ai-generated text be reliably detected?  _arXiv preprint arXiv:2303.11156_ (2023).
  * [100] Hussain, S., Neekhara, P., Jere, M., Koushanfar, F. & McAuley, J.  Adversarial deepfakes: Evaluating vulnerability of deepfake detectors to adversarial examples.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 3348–3357 (2021).
  * [101] Content Authenticity Initiative — contentauthenticity.org.  https://contentauthenticity.org/.  [Accessed 08-09-2023].
  * [102] ChatGPT: OpenAI reopens the platform in Italy guaranteeing more transparency and more rights to European users and non-users — tbs-sct.canada.ca.  https://www.garanteprivacy.it/home/docweb/-/docweb-display/docweb/9881490.  [Accessed 27-09-2023].
  * [103] Chatbots, deepfakes, and voice clones: AI deception for sale — ftc.gov.  https://www.ftc.gov/business-guidance/blog/2023/03/chatbots-deepfakes-voice-clones-ai-deception-sale.  [Accessed 08-09-2023].
  * [104] Directive on Automated Decision-Making — tbs-sct.canada.ca.  https://www.tbs-sct.canada.ca/pol/doc-eng.aspx?id=32592.  [Accessed 27-09-2023].
  * [105] Gangi Reddy, R. _et al._ NewsClaims: A new benchmark for claim detection from news with attribute knowledge.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 6002–6018, DOI: 10.18653/v1/2022.emnlp-main.403 (Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022).
  * [106] Sundriyal, M., Kulkarni, A., Pulastya, V., Akhtar, M. S. & Chakraborty, T.  Empowering the fact-checkers! automatic identification of claim spans on Twitter.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 7701–7715, DOI: 10.18653/v1/2022.emnlp-main.525 (Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2022).
  * [107] Sundriyal, M., Singh, P., Akhtar, M. S., Sengupta, S. & Chakraborty, T.  Desyr: definition and syntactic representation based claim detection on the web.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 1764–1773 (2021).
  * [108] Huang, K.-H., Zhai, C. & Ji, H.  CONCRETE: Improving cross-lingual fact-checking with cross-lingual retrieval.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , 1024–1035 (International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 2022).
  * [109] Huang, K.-H., Chan, H. P. & Ji, H.  Zero-shot faithful factual error correction.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 5660–5676, DOI: 10.18653/v1/2023.acl-long.311 (Association for Computational Linguistics, Toronto, Canada, 2023).
  * [110] Zhang, B., Ding, D. & Jing, L.  How would stance detection techniques evolve after the launch of chatgpt?  _arXiv preprint arXiv:2212.14548_ (2022).
  * [111] Kocoń, J. _et al._ ChatGPT: Jack of all trades, master of none.  _Information Fusion_ 99, 101861, DOI: https://doi.org/10.1016/j.inffus.2023.101861 (2023).
  * [112] Shankar, A.  Remembering Conversations: Building Chatbots with Short and Long-Term Memory on AWS — itnext.io.  https://itnext.io/remembering-conversations-building-chatbots-with-short-and-long-term-memory-on-aws-c1361c130046.  [Accessed 28-09-2023].
  * [113] Ferrara, E.  GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models.  _arXiv preprint arXiv:2310.00737_ (2023).
  * [114] Login | The National Academies Press — nap.nationalacademies.org.  https://nap.nationalacademies.org/download/24623.  [Accessed 08-09-2023].
  * [115] McCallum, S.  ChatGPT banned in Italy over privacy concerns.  https://www.bbc.com/news/technology-65139406.  [Accessed 08-09-2023].
  * [116] New UK university principles promote AI literacy and integrity — universityworldnews.com.  https://www.universityworldnews.com/post.php?story=20230704155107330.  [Accessed 08-09-2023].
  * [117] Department for Education.  Generative artificial intelligence in education departmental statement.  https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/1146540/Generative_artificial_intelligence_in_education_.pdf.  [Accessed 08-09-2023].
  * [118] Cohen, J.  Right on Track: NVIDIA Open-Source Software Helps Developers Add Guardrails to AI Chatbots — blogs.nvidia.com.  https://blogs.nvidia.com/blog/2023/04/25/ai-chatbot-guardrails-nemo/.  [Accessed 08-09-2023].
  * [119] Chen, A. & Chen, D. O.  Accuracy of chatbots in citing journal articles.  _JAMA Netw Open_ 6, e2327647 (2023).
  * [120] Spataro, J.  Introducing Microsoft 365 Copilot – your copilot for work - The Official Microsoft Blog — blogs.microsoft.com.  https://blogs.microsoft.com/blog/2023/03/16/introducing-microsoft-365-copilot-your-copilot-for-work/.  [Accessed 08-09-2023].
  * [121] Pacheco, D. _et al._ Uncovering coordinated networks on social media: Methods and case studies.  In _Proc. International AAAI Conference on Web and Social Media (ICWSM)_ , vol. 15, 455–466, DOI: 10.1609/icwsm.v15i1.18075 (2021).