# “Double-path” ferroelectrics and the sign of the piezoelectric response

Yubo Qi1, Sebastian E. Reyes-Lillo2, and Karin M. Rabe1 1Department of Physics
$\&$ Astronomy, Rutgers University, Piscataway, New Jersey 08854, United
States
2Departamento de Ciencias Físicas, Universidad Andres Bello, Santiago
837-0136, Chile

###### Abstract

In this work, we propose a class of ferroelectrics (which we denote “double-
path” ferroelectrics), characterized by two competing polarization switching
paths for which the change in polarization is different and in fact of
opposite sign. Depending on which path is favorable under given conditions,
this leads to different identification of up- and down-polarized states. Since
the sign of piezoelectric response depends on the assignment of up- or down-
polarized state for a specific structure, this means that the material can
exhibit different signs of the piezoelectric response under different
conditions. We focus on HfO2 as a key example. Our first-principles
calculations show that there are two competing paths in HfO2, resulting from
different displacements of the atoms from the initial to the final structures,
and the change in polarization along these two paths is of opposite sign.
These results provide a natural explanation for the recently observed
discrepancy in the signs of piezoelectric responses in HfO2 between
theoretical first-principles calculations and experimental observation.
Further, this allows predictions of how to favor one path over another by
changes in conditions and compositional tuning. This family of materials also
includes other candidates, such as CuInP2S6 and theoretically proposed
LaVO3-SrVO3 superlattice. We finally note that double-path ferroelectrics
possess novel electromechanical properties since the signs of their
piezoelectric responses can be switched.

###### pacs:

Valid PACS appear here

††preprint: APS/123-QED

Ferroelectrics have promising applications in electronic devices due to their
spontaneous polarization and switchability by applied electric fields
Haertling (1999); Setter  _et al._ (2006). Typically, a ferroelectric material
has two symmetry-related states, called variants, with polarization in
opposite directions, designated “up” and “down”, with switching from the up
state to the down state by application of a field in the down direction, and
from down to up by a field in the up direction. The ferroelectric polarization
is obtained as half of the magnitude of change in polarization on switching.

In conventional ferroelectrics such as BaTiO3 or PbTiO3, the polar states are
generated by freezing in of an unstable polar lattice mode of a closely
related high-symmetry state, such as the cubic perovskite structure Ghosez
_et al._ (1999); Sai  _et al._ (2002); Diéguez  _et al._ (2005); Qi  _et al._
(2016, 2015). In that case, the identification of a particular variant as up
or down polarized is straightforward. The displacement of each ion in the up
state to its final position in the down state is determined by reversing the
displacements of the unstable polar lattice mode distortion, with the overall
net displacement along the direction of the electric forces on the ions. This
matching up corresponds to a switching path with displacements that are in
some sense minimal, and from this path, the change in polarization can be
directly computed Bonini  _et al._ (2020).

In some ferroelectrics, the matching up of ions in the up state with final
positions in the down state can be made in two different ways with comparable
ionic motion, as illustrated in Fig. 1. The initial structures in the two
subfigures (represented by the grey circles and red solid circles) are the
same and the final structures in the two subfigures (represented by the grey
circles and red hollow circles) are also the same. However, due to the
different displacements of the red cations, the switching polarizations in
Fig. 1 (a) and (b) differ by a integer multiple of the quantum of polarization
$q{\bm{R}}/V$, where $q$ is the ionic change, $\bm{R}$ is the lattice vector,
and $V$ is the volume Bonini  _et al._ (2020); King-Smith and Vanderbilt
(1993), and in fact are opposite in sign. The switching in (a) is thus driven
by an upward-pointing electric field, as shown, leading to the identification
of the initial structure as down polarized. Conversely, the switching in (b)
is driven by an downward-pointing electric field, as shown, leading to the
identification of the same initial structure as up polarized.

Figure 1: Schematic illustrations of path dependent switching polarization.
Filled red and grey circles represent negatively and positively charged ions
in the crystal. The arrows indicate the motion of the red ions in the applied
electric field shown, and open red circles represent the final positions of
each red ion after switching.

In this work, we argue that ferroelectric HfO2 is a double-path ferroelectric.
Combining previous results Clima  _et al._ (2014, 2020); Yang  _et al._
(2020); Huan  _et al._ (2014); Maeda  _et al._ (2017); Fan  _et al._ (2019);
Choe  _et al._ (2021); Chen  _et al._ (2020); Ding  _et al._ (2020); Lee  _et
al._ (2020) with additional first-principles calculations, we will show that
there are competing paths connecting symmetry related variants that match the
ions in one state to final positions in the other state in different ways. In
the case of HfO2, we will show this changes the identification of which
structure is up and which is down. We will then discuss the implications of
this for the interpretation of observations of piezoelectricity in
ferroelectric HfO2, and in particular offer a natural explanation for the
apparent discrepancy between theoretical predictions of negative piezoelectric
response and experimental observations in which most systems show a positive
piezoelectric response Liu  _et al._ (2020); Dutta  _et al._ (2021); Schenk
_et al._ (2020); Böscke  _et al._ (2011); Starschich  _et al._ (2014);
Chouprik  _et al._ (2019). Based on these results, we propose a combined
theory-experiment approach to identify the preferred polarization switching
path in double-path ferroelectrics by computing the piezoelectric response for
a given variant and identifying it as up or down based on the shape of the
measured butterfly piezoelectric hysteresis loop.

Figure 2: Structure sequence ((a) and (d)), energy profiles ((b) and (e)), and
slopes of the energy profiles ((c) and (f)) in the two polarization switching
paths. In (a) and (c), the gray atoms are Hf and the blue and red atoms are
oxygen, with red being closer and blue being farther from this viewpoint. In
the two paths, the starting structures (structure 1) are identical, with extra
oxygen atoms shown in (d) to clarify the different structural evolution along
path II, and the final structures (structures 3 and 3’) are symmetry-related.
The initial sign of $S$ is an indication that structure 1 is the up-polarized
state for path I and the down-polarized state for path II. The maximum
magnitudes of $S$ for paths I and II are marked in (c) and (f), respectively.

The ferroelectricity in HfO2-based materials is attributed to the formation of
the $Pca2_{1}$ phase Sang  _et al._ (2015); Böscke  _et al._ (2011); Qi  _et
al._ (2020); Xu  _et al._ (2021); Raeliarijaona and Cohen (2021), which is
shown in Fig. 2. The identification of a particular variant as up or down may
at first seem as obvious as in BaTiO3 or PbTiO3, with polar states generated
by freezing in a polar lattice mode of the high-symmetry cubic fluorite
structure (Fig. 2 (a)). Since the polarization change from 1 to 3 is positive,
this identifies structure 1 as down and structure 3 as up Yang  _et al._
(2020); Huan  _et al._ (2014); Maeda  _et al._ (2017); Fan  _et al._ (2019);
Choe  _et al._ (2021); Chen  _et al._ (2020); Ding  _et al._ (2020); Lee  _et
al._ (2020). However, we note that in contrast to conventional ferroelectrics,
additional modes are needed to generate the polar structure of HfO2 and in
fact there are no unstable polar modes in the the cubic fluorite structure.
Furthermore, there is a distinct polarization switching path that has recently
been discussed Clima  _et al._ (2014, 2020); Yang  _et al._ (2020). In this
path ii@, shown in (d), the oxygen atoms move up through the Hf atomic planes,
with the polarization change then being positive. This changes the
identification of structure 1 to the up state, with structure 3’ being the
down state, noting that structure 3’ is symmetry-related to structure 3 Qi
_et al._ (2021).

In this work, we carry out density functional theory (DFT) calculations with
the Quantum–espresso package Giannozzi  _et al._ (2009) to investigate the
competition between the two paths,. Norm-conserving pseudopotentials based on
the local density approximation are generated by the Opium package Opi . The
atomic force convergence threshold is set as 1$\times$10-4 Hartree per Bohr.
Calculations are done in the conventional 12-atom cell with a
4$\times$4$\times 4$ Monkhorst–Pack $k$-point mesh to sample the Brillouin
zone Monkhorst and Pack (1976). To generate the energy profile as a function
of polarization, starting from structure 1, we move all oxygen atoms together
along the $z$ direction, then optimize the structure keeping the $z$-direction
displacement between the Hf atom and the average position of its surrounding 8
oxygen atoms fixed Reyes-Lillo  _et al._ (2014); Qi and Rabe (2020), and then
compute the polarization for the optimized structure. The polarization of each
structure is calculated with the Berry flux diagonalization method reported in
Ref. Bonini  _et al._ (2020).

The calculated energy profiles are shown in Fig. 2, with the energy of the
$Pca2_{1}$ phase taken as zero. In path i@, the zero polarization structure is
the centrosymmetric tetragonal $P$4${\rm{{}_{2}}}/nmc$ phase, obtained by
freezing the unstable $X_{2}^{-}$ phonon into the cubic fluorite structure
Huan  _et al._ (2014); Yang  _et al._ (2020). This tetragonal
$P$4${\rm{{}_{2}}}/nmc$ phase has no unstable phonons Reyes-Lillo  _et al._
(2014); Qi  _et al._ (2020). On this path, as the polarization increases from
structure 1 and reaches a critical value ($P=0.206$ C/m2), the structure
becomes unstable and collapses in a first-order phase transition to an $Aba2$
structure, which is related to the $P$4${\rm{{}_{2}}}/nmc$ phase by a polar
distortion Reyes-Lillo  _et al._ (2014); Qi and Rabe (2020). In path ii@, the
structures and energies change smoothly. The zero polarization structure on
this path is the centrosymmetric orthorhombic $Pbcm$ structure, in which the
oxygen atoms lie in the Hf atomic planes Clima  _et al._ (2014, 2020); Yang
_et al._ (2020); Chen  _et al._ (2020). This orthorhombic $Pbcm$ structure is
unstable and has a higher (153 meV vs. 51 meV per formula unit) energy than
the centrosymmetric tetragonal $P$4${\rm{{}_{2}}}/nmc$ structure in path<EMAIL_ADDRESS>
Since path i@ has a lower energy barrier and a more symmetric zero-
polarization structure, it has been generally accepted as the physically
relevant path Yang  _et al._ (2020); Huan  _et al._ (2014); Maeda  _et al._
(2017); Fan  _et al._ (2019); Choe  _et al._ (2021); Chen  _et al._ (2020);
Ding  _et al._ (2020); Lee  _et al._ (2020). However, the physical quantity
determining which path is favored in electric-field switching is not the
height of the energy barrier, but the minimum electric field needed to drive
the system through the path. This is given by the maximum slope of the energy
profile $S=\partial{U}/\partial{P}$, shown in Fig. 2 (b) and (e) for the two
paths being considered here. In path i@ (Fig. 2 (b)), $S$ increases and
reaches its maximum value right before the first-order phase transition, after
which $S$ jumps down and changes its sign. The structures and energies along
path ii@ evolve smoothly with changing polarization, as shown in Fig. 2 (d).
The maximum slope of path ii@ is in fact a little smaller than that in path i@
(18.8 MV/cm vs. 22.9 MV/cm), making path ii@ slightly more favorable. However,
the difference is small so that the two paths should be considered
competitive, with the preferred path depending on the details of the system
(doping, sample preparation, temperature, stress, and bonding condition with
electrodes) Clima  _et al._ (2014); Yang  _et al._ (2020).

The identification of a particular variant as “up-polarized” determines the
sign of the piezoelectric coefficient, since the piezoelectric response of the
up state is, by convention, reported as the piezoelectric response of the
material. The sign of the piezoelectric coefficient thus can reverse if a
different variant is identified as “up-polarized” Dutta  _et al._ (2021). We
first consider the case that structure 3 has been identified as the “up-
polarized” state, as it has been in most discussions in the literature Liu
_et al._ (2020); Yang  _et al._ (2020); Huan  _et al._ (2014); Maeda  _et al._
(2017); Fan  _et al._ (2019); Choe  _et al._ (2021); Chen  _et al._ (2020);
Ding  _et al._ (2020); Lee  _et al._ (2020). First-principles calculations
have shown when a uniaxial stress is applied stretching the system in the $z$
direction, the oxygen atoms move up, decreasing the polarization Dutta  _et
al._ (2021). This decrease in polarization corresponds to the negative
piezoelectric coefficient previously reported in first-principles studies Liu
_et al._ (2020). On the other hand, if the system switches according to path
II and structure 1 is identified as the “up-polarized” structure, a symmetry
transformation of the same calculation shows that the oxygen atoms move down,
increasing the polarization, and the piezoelectric coefficient is positive.
This connection between the sign of the piezoelectric response and the
switching path is manifest in the experimental determination of the
piezoelectric response from the piezoelectric hysteresis loop, as shown in
Fig. 3. If the hysteresis loop is butterfly-shaped, with positive slopes at
electric field above the critical field, the piezoelectric response is
positive You  _et al._ (2019); on the other hand, if the hysteresis loop is
inverted-butterfly-shaped, the piezoelectric response is negative You  _et
al._ (2019).

Figure 3: Schematic illustrations of the electrical and piezoelectric
hysteresis loops of HfO2-based ferroelectrics switching according to path I,
with a negative piezoelectric response ((a) and (c)) and path II, with a
positive piezoelectric response ((b) and (d)). We show path I with a larger
critical field than path II to schematically indicate its slightly larger
maximum slope.

Positive piezoelectric responses in HfO2-based ferroelectrics have been
observed in many experimental works. The samples include but are not limited
to Si-doped HfO2, Y-doped HfO2, and La-doped HfO2 Schenk  _et al._ (2020);
Böscke  _et al._ (2011); Starschich  _et al._ (2014). This discrepancy with
the previous first-principles prediction has attracted considerable interest
and Dutta et al. showed that the computed sign of piezoelectric response of
pure HfO2 can become positive under a 6 % compressive strain. Here, we provide
an alternative explanation which is natural and straightforward; the observed
sign of the piezoelectric response can be understood by identifying the up-
polarized state in the experiment as having the opposite polarization to that
assumed in the previous first-principles studies.

Further, the fact that some samples show negative piezoelectric response Dutta
_et al._ (2021); Chouprik  _et al._ (2019) can also be readily understood.
HfO2 has two competing polarization switching paths with approximately equal
preference. Therefore, it is highly plausible that polarization switching in
different experiments could occur along different paths, leading to both
positive and negative reported piezoelectric responses. As an example, we
consider the effect of substitution of Hf by Zr. In Fig. 5, we plot the slopes
of energy profiles of Hf0.25Zr0.75O2 (75 % Zr doped HfO2) given by our DFT
calculations. Comparing these results to those for pure HfO2, we observe that
the maximum slope of path i@ becomes smaller relative to path ii@, leading to
a theoretical prediction that Zr doping will favor path i@ and a negative
piezoelectric response. In fact, Chouprik et al. has reported an “anomalous”
piezoresponse force microscopy (PFM) switching, which is an indication of
negative piezoelectric response, in Hf0.5Zr0.5O2 thin films Chouprik  _et al._
(2019), consistent with the theoretical prediction.

Figure 4: The slopes of the energy profiles for (a) path i@ and (b) path ii@
of Hf0.25Zr0.75O2 (75 % Zr doped HfO2).

We should note that our analysis so far is based on uniform switching in pure
HfO2, which is typically different from that of real ferroelectrics.
Polarization usually switches through a domain wall nucleation and growth
process, rather than uniformly Merz (1954); Stadler and Zachmanidis (1963);
Tybell  _et al._ (2002); Ahn  _et al._ (2004); Li  _et al._ (2004); Gruverman
_et al._ (2005); Shin  _et al._ (2007); Liu  _et al._ (2016a). Determining the
details for a specific system, which involves domain wall motion Grimley  _et
al._ (2018); Buragohain  _et al._ (2018); Mulaosmanovic  _et al._ (2017);
Fengler  _et al._ (2017); Stolichnov  _et al._ (2018), temperature Zhou  _et
al._ (2015); Park  _et al._ (2018), vacancies Schenk  _et al._ (2014);
Baumgarten  _et al._ (2021), surfaces/interfaces Park  _et al._ (2017a),
doping Pešić  _et al._ (2016); Hoffmann  _et al._ (2015); Schroeder  _et al._
(2014); Park  _et al._ (2017b); Mueller  _et al._ (2012); Müller  _et al._
(2011); Olsen  _et al._ (2012) and so on, is much more complex and beyond the
scope of this work. However, our main interest is in the total displacement of
each ion from the initial state to the final state, which is expected to be
uniform even for inhomogeneous switching. However, if the uniform switching
paths are competitive, it is highly possible that different inhomogenous
switching paths are also competitive and one can be more favorable under
specific experimental conditions, while the other is more favorable under
other conditions.

Figure 5: Structures of CuInP2S6 and the LaVO3-SrVO3 superlattice, which are
other examples of “double-path” ferroelectric. In subfigures (a) and (b), the
Cu atoms in the initial and final structures are represented with blue and
cyan spheres.

We have focused on HfO2 is an example of a double-path ferroelectrics, but
there are other candidate for materials belonging to this family of
ferroelectrics. In Fig. 5 (a) and (b), we show the structure of CuInP2S6
(CIPS), which is a two-dimensional layered ferroelectric Neumayer  _et al._
(2019); You  _et al._ (2019); Brehm  _et al._ (2020); Neumayer  _et al._
(2020). Each molecular layer is composed of sulfur octahedra, and each sulfur
octahedron is filled with a Cu atom, In atom, or a P-P dimer Liu  _et al._
(2016b); Maisonneuve  _et al._ (1995, 1997). The ferroelectricity results from
the displacements of the Cu atoms. It has been shown that the polarization
could switch either by the Cu atoms displacing across the inter-layer gap (as
shown in Fig. 4 (a)) Liu  _et al._ (2016b); Neumayer  _et al._ (2020) or
inside the molecular layer (as shown in Fig. 4 (b)) Neumayer  _et al._ (2020),
leading to the assignment of the variant shown as either up or down,
respectively. The LaVO3-SrVO3 superlattice, which is a theoretically proposed
charge-ordering induced ferroelectric Park  _et al._ (2017c); Qi and Rabe
(2021), is another example. In this system, the V atoms disproportionate into
V3+ and V4+. The superlattice has a polar metastable layered charge ordering
(LCO) structure which can be stabilized over the competing structures by
strain or an applied electric field Park  _et al._ (2017c); Qi and Rabe
(2021). The polarization switching is associated with an electron transfer
from the V3+ ions to V4+ ions. It is possible that the electron could transfer
either though the SrO layer (as shown in Fig. 4 (c)) or through the LaO layer
(as shown in Fig. 4 (d)), leading to the assignment of the variant shown as
either up or down, respectively.

In experiments, the most straightforward way to identify up and down variants
is determining the atomic arrangements in the field-switched up and down
variants by atomic-scale imaging methods. However, this might not be possible,
particularly in thin films. Here, we propose a combined theory-experiment
approach, based on the assumption that first-principles calculations are
generally quite successful in predicting the piezoelectric coefficients of a
crystal Fu and Cohen (2000); Wu and Cohen (2005); Sághi-Szabó  _et al._
(1998); Bellaiche and Vanderbilt (2000); Grinberg  _et al._ (2004); Hamann
_et al._ (2005); Cohen (2008); Wu and Cohen (2005) and that in experiments,
the sign of piezoelectric response can be straightforwardly and unambiguously
measured by the shape of the piezoelectric hysteresis loop You  _et al._
(2019). If the butterfly shape is normal (positive piezoelectric coefficient)
then the variant with positive piezoelectric coefficient as determined by
first-principles calculations is the up-polarized state. If the butterfly
shape is reversed (negative piezoelectric coefficient), then the variant with
negative piezoelectric coefficient is the up polarized state.

For most known materials, piezoelectric responses are positive, indicating
that the polarization in a positively polarized state tends to increase under
a tensile strain. Exceptions with negative piezoelectric responses have
attracted intensive research interest. Known negative piezoelectric materials
include a variety of $ABC$ ferroelectrics Liu and Cohen (2017), several
iii@-v@ zinc blende compounds Bellaiche and Vanderbilt (2000), and most low-
dimensional piezoelectrics Kim  _et al._ (2019); Qi and Rappe (2021), and the
search is still going on. “Double path” ferroelectrics not only have negative
piezoelectric response under suitable conditions, but have the novel feature
that tuning can change the sign of their piezoelectric response from negative
to positive or vice versa.

In this work, we have proposed a specific family of ferroelectrics (referred
to as “double-path” ferroelectric) with two competing polarization switching
paths, leading to different assignments of up- and down- polarized states. We
emphasize that the physical quantity determining the preference of each path
should be the maximum slope of the energy profile, rather than the height of
the energy barrier. Examples of double-path ferroelectrics include but are not
limited to HfO2, CuInP2S6, and theoretically proposed LaVO3-SrVO3
superlattice. Double-path ferroelectrics exhibit novel electromechanical
properties, since their piezoelectric responses can be tuned between positive
and negative under different conditions.

## acknowledgement

Y.Q. and K.M.R. was supported by ONR N00014-21-1-2107. S.E.R.-L. acknowledges
support from ANID FONDECYT Regular grant number 1220986. First-principles
calculations were performed using the computational resources provided by the
Rutgers University Parallel Computing (RUPC) clusters and the High-Performance
Computing Modernization Office of the Department of Defense.

## References

  * Haertling (1999) G. H. Haertling, J. Am. Ceram. Soc. 82, 797 (1999).
  * Setter  _et al._ (2006) N. Setter, D. Damjanovic, L. Eng, G. Fox, S. Gevorgian, S. Hong, A. Kingon, H. Kohlstedt, N. Y. Park, G. B. Stephenson, I. Stolitchnov, A. K. Taganstev, D. V. Taylor, T. Yamada, and S. Streiffer, J. Appl. Phys. 100, 051606 (2006).
  * Ghosez  _et al._ (1999) P. Ghosez, E. Cockayne, U. V. Waghmare, and K. M. Rabe, Phys. Rev. B 60, 836 (1999).
  * Sai  _et al._ (2002) N. Sai, K. M. Rabe, and D. Vanderbilt, Phys. Rev. B 66, 104108 (2002).
  * Diéguez  _et al._ (2005) O. Diéguez, K. M. Rabe, and D. Vanderbilt, Phys. Rev. B 72, 144101 (2005).
  * Qi  _et al._ (2016) Y. Qi, S. Liu, I. Grinberg, and A. M. Rappe, Phys. Rev. B 94, 134308 (2016).
  * Qi  _et al._ (2015) Y. Qi, J. M. P. Martirez, W. A. Saidi, J. J. Urban, W. S. Yun, J. E. Spanier, and A. M. Rappe, Phys. Rev. B 91, 245431 (2015).
  * Bonini  _et al._ (2020) J. Bonini, D. Vanderbilt, and K. M. Rabe, Phys. Rev. B 102, 045141 (2020).
  * King-Smith and Vanderbilt (1993) R. D. King-Smith and D. Vanderbilt, Phys. Rev. B 47, 1651 (1993).
  * Clima  _et al._ (2014) S. Clima, D. J. Wouters, C. Adelmann, T. Schenk, U. Schroeder, M. Jurczak, and G. Pourtois, Appl. Phys. Lett. 104, 092906 (2014).
  * Clima  _et al._ (2020) S. Clima, B. J. O’Sullivan, N. Ronchi, M. G. Bardon, K. Banerjee, G. Van den Bosch, G. Pourtois, and J. Van Houdt (IEEE, 2020) pp. 1–4.
  * Yang  _et al._ (2020) H. Yang, H.-J. Lee, J. Jo, C. H. Kim, and J. H. Lee, Phys. Rev. Appl. 14, 064012 (2020).
  * Huan  _et al._ (2014) T. D. Huan, V. Sharma, G. A. Rossetti Jr, and R. Ramprasad, Phys. Rev. B 90, 064111 (2014).
  * Maeda  _et al._ (2017) T. Maeda, B. Magyari-Kope, and Y. Nishi (IEEE, 2017) pp. 1–4.
  * Fan  _et al._ (2019) P. Fan, Y. K. Zhang, Q. Yang, J. Jiang, L. M. Jiang, M. Liao, and Y. C. Zhou, J. Phys. Chem. C 123, 21743 (2019).
  * Choe  _et al._ (2021) D.-H. Choe, S. Kim, T. Moon, S. Jo, H. Bae, S.-G. Nam, Y. S. Lee, and J. Heo, Mater. Today 50, 8 (2021).
  * Chen  _et al._ (2020) Y.-W. Chen, S.-T. Fan, and C. W. Liu, J. Phys. D: Appl. Phys. 54, 085304 (2020).
  * Ding  _et al._ (2020) W. Ding, Y. Zhang, L. Tao, Q. Yang, and Y. Zhou, Acta Mater. 196, 556 (2020).
  * Lee  _et al._ (2020) H.-J. Lee, M. Lee, K. Lee, J. Jo, H. Yang, Y. Kim, S. C. Chae, U. Waghmare, and J. H. Lee, Science 369, 1343 (2020).
  * Liu  _et al._ (2020) J. Liu, S. Liu, J.-Y. Yang, and L. Liu, Phys. Rev. Lett. 125, 197601 (2020).
  * Dutta  _et al._ (2021) S. Dutta, P. Buragohain, S. Glinsek, C. Richter, H. Aramberri, H. Lu, U. Schroeder, E. Defay, A. Gruverman, and J. Íñiguez, Nat. Commun. 12, 1 (2021).
  * Schenk  _et al._ (2020) T. Schenk, N. Godard, A. Mahjoub, S. Girod, A. Matavz, V. Bobnar, E. Defay, and S. Glinsek, Phys. Status Solidi RRL 14, 1900626 (2020).
  * Böscke  _et al._ (2011) T. Böscke, J. Müller, D. Bräuhaus, U. Schröder, and U. Böttger, Appl. Phys. Lett. 99, 102903 (2011).
  * Starschich  _et al._ (2014) S. Starschich, D. Griesche, T. Schneller, R. Waser, and U. Böttger, Appl. Phys. Lett. 104, 202903 (2014).
  * Chouprik  _et al._ (2019) A. Chouprik, M. Spiridonov, S. Zarubin, R. Kirtaev, V. Mikheev, Y. Lebedinskii, S. Zakharchenko, and D. Negrov, ACS Appl. Electron. Mater. 1, 275 (2019).
  * Sang  _et al._ (2015) X. Sang, E. D. Grimley, T. Schenk, U. Schroeder, and J. M. LeBeau, Appl. Phys. Lett. 106, 162905 (2015).
  * Qi  _et al._ (2020) Y. Qi, S. Singh, C. Lau, F.-T. Huang, X. Xu, F. J. Walker, C. H. Ahn, S.-W. Cheong, and K. M. Rabe, Phys. Rev. Lett. 125, 257603 (2020).
  * Xu  _et al._ (2021) X. Xu, F.-T. Huang, Y. Qi, S. Singh, K. M. Rabe, D. Obeysekera, J. Yang, M.-W. Chu, and S.-W. Cheong, Nat. Mater. 20, 826 (2021).
  * Raeliarijaona and Cohen (2021) A. Raeliarijaona and R. Cohen, arXiv preprint arXiv:2108.09884 (2021).
  * Qi  _et al._ (2021) Y. Qi, S. Singh, and K. M. Rabe, arXiv preprint arXiv:2108.12538 (2021).
  * Giannozzi  _et al._ (2009) P. Giannozzi, S. Baroni, N. Bonini, M. Calandra, R. Car, C. Cavazzoni, D. Ceresoli, G. L. Chiarotti, M. Cococcioni, I. Dabo, A. D. Corso, S. de Gironcoli, S. Fabris, G. Fratesi, R. Gebauer, U. Gerstmann, C. Gougoussis, A. Kokalj, M. Lazzeri, L. Martin-Samos, N. Marzari, F. Mauri, R. Mazzarello, S. Paolini, A. Pasquarello, L. Paulatto, C. Sbraccia, S. Scandolo, G. Sclauzero, A. P. Seitsonen, A. Smogunov, P. Umari, and R. M. Wentzcovitch, J. Phys.: Condens. Matter 21, 395502 (2009).
  * (32) http://opium.sourceforge.net.
  * Monkhorst and Pack (1976) H. J. Monkhorst and J. D. Pack, Phys. Rev. B 13, 5188 (1976).
  * Reyes-Lillo  _et al._ (2014) S. E. Reyes-Lillo, K. F. Garrity, and K. M. Rabe, Phys. Rev. B 90, 140103 (2014).
  * Qi and Rabe (2020) Y. Qi and K. M. Rabe, Phys. Rev. B 102, 214108 (2020).
  * You  _et al._ (2019) L. You, Y. Zhang, S. Zhou, A. Chaturvedi, S. A. Morris, F. Liu, L. Chang, D. Ichinose, H. Funakubo, W. Hu, T. Wu, Z. Liu, S. Dong, and J. Wang, Sci. Adv. 5, 3780 (2019).
  * Merz (1954) W. J. Merz, Phys. Rev. 95, 690 (1954).
  * Stadler and Zachmanidis (1963) H. L. Stadler and P. J. Zachmanidis, J. Appl. Phys. 34, 3255 (1963).
  * Tybell  _et al._ (2002) T. Tybell, P. Paruch, T. Giamarchi, and J.-M. Triscone, Phys. Rev. Lett. 89, 097601 (2002).
  * Ahn  _et al._ (2004) C. H. Ahn, K. M. Rabe, and J.-M. Triscone, Science 303, 488 (2004).
  * Li  _et al._ (2004) J. Li, B. Nagaraj, H. Liang, W. Cao, C. H. Lee, and R. Ramesh, Appl. Rhys. Lett. 84, 1174 (2004).
  * Gruverman  _et al._ (2005) A. Gruverman, B. J. Rodriguez, C. Dehoff, J. D. Waldrep, A. I. Kingon, R. J. Nemanich, and J. S. Cross, Appl. Phys. Lett. 87, 082902 (2005).
  * Shin  _et al._ (2007) Y.-H. Shin, I. Grinberg, I.-W. Chen, and A. M. Rappe, Nature 449, 881 (2007).
  * Liu  _et al._ (2016a) S. Liu, I. Grinberg, and A. M. Rappe, Nature 534, 360 (2016a).
  * Grimley  _et al._ (2018) E. D. Grimley, T. Schenk, T. Mikolajick, U. Schroeder, and J. M. LeBeau, Adv. Mater. Interfaces 5, 1701258 (2018).
  * Buragohain  _et al._ (2018) P. Buragohain, C. Richter, T. Schenk, H. Lu, T. Mikolajick, U. Schroeder, and A. Gruverman, Appl. Phys. Lett. 112, 222901 (2018).
  * Mulaosmanovic  _et al._ (2017) H. Mulaosmanovic, J. Ocker, S. Müller, U. Schroeder, J. Müller, P. Polakowski, S. Flachowsky, R. van Bentum, T. Mikolajick, and S. Slesazeck, ACS Appl. Mater. Interfaces 9, 3792 (2017).
  * Fengler  _et al._ (2017) F. P. G. Fengler, M. Pešić, S. Starschich, T. Schneller, C. Künneth, U. Böttger, H. Mulaosmanovic, T. Schenk, M. H. Park, R. Nigon, P. Muralt, T. Mikolajick, and U. Schroeder, Adv. Electron. Mater. 3, 1600505 (2017).
  * Stolichnov  _et al._ (2018) I. Stolichnov, M. Cavalieri, E. Colla, T. Schenk, T. Mittmann, T. Mikolajick, U. Schroeder, and A. M. Ionescu, ACS Appl. Mater. Interfaces 10, 30514 (2018).
  * Zhou  _et al._ (2015) D. Zhou, Y. Guan, M. M. Vopson, J. Xu, H. Liang, F. Cao, X. Dong, J. Mueller, T. Schenk, and U. Schroeder, Acta Mater. 99, 240 (2015).
  * Park  _et al._ (2018) M. H. Park, C.-C. Chung, T. Schenk, C. Richter, M. Hoffmann, S. Wirth, J. L. Jones, T. Mikolajick, and U. Schroeder, Adv. Electron. Mater. 4, 1700489 (2018).
  * Schenk  _et al._ (2014) T. Schenk, U. Schroeder, M. Pešić, M. Popovici, Y. V. Pershin, and T. Mikolajick, ACS Appl. Mater. Interfaces 6, 19744 (2014).
  * Baumgarten  _et al._ (2021) L. Baumgarten, T. Szyjka, T. Mittmann, M. Materano, Y. Matveyev, C. Schlueter, T. Mikolajick, U. Schroeder, and M. Müller, Appl. Phys. Lett. 118, 032903 (2021).
  * Park  _et al._ (2017a) M. H. Park, Y. H. Lee, H. J. Kim, T. Schenk, W. Lee, K. Do Kim, F. P. G. Fengler, T. Mikolajick, U. Schroeder, and C. S. Hwang, Nanoscale 9, 9973 (2017a).
  * Pešić  _et al._ (2016) M. Pešić, F. P. G. Fengler, L. Larcher, A. Padovani, T. Schenk, E. D. Grimley, X. Sang, J. M. LeBeau, S. Slesazeck, U. Schroeder, and T. Mikolajick, Adv. Funct. Mater. 26, 4601 (2016).
  * Hoffmann  _et al._ (2015) M. Hoffmann, U. Schroeder, T. Schenk, T. Shimizu, H. Funakubo, O. Sakata, D. Pohl, M. Drescher, C. Adelmann, R. Materlik, A. Kersch, and T. Mikolajick, J. Appl. Phys. 118, 072006 (2015).
  * Schroeder  _et al._ (2014) U. Schroeder, E. Yurchuk, J. Müller, D. Martin, T. Schenk, P. Polakowski, C. Adelmann, M. I. Popovici, S. V. Kalinin, and T. Mikolajick, Jpn. J. Appl. Phys. 53, 08LE02 (2014).
  * Park  _et al._ (2017b) M. H. Park, T. Schenk, C. M. Fancher, E. D. Grimley, C. Zhou, C. Richter, J. M. LeBeau, J. L. Jones, T. Mikolajick, and U. Schroeder, J. Mater. Chem. C . 5, 4677 (2017b).
  * Mueller  _et al._ (2012) S. Mueller, J. Mueller, A. Singh, S. Riedel, J. Sundqvist, U. Schroeder, and T. Mikolajick, Adv. Funct. Mater. 22, 2412 (2012).
  * Müller  _et al._ (2011) J. Müller, U. Schröder, T. Böscke, I. Müller, U. Böttger, L. Wilde, J. Sundqvist, M. Lemberger, P. Kücher, T. Mikolajick, and L. Frey, J. Appl. Phys. 110, 114113 (2011).
  * Olsen  _et al._ (2012) T. Olsen, U. Schröder, S. Müller, A. Krause, D. Martin, A. Singh, J. Müller, M. Geidel, and T. Mikolajick, Appl. Phys. Lett. 101, 082905 (2012).
  * Neumayer  _et al._ (2019) S. M. Neumayer, E. A. Eliseev, M. A. Susner, A. Tselev, B. J. Rodriguez, J. A. Brehm, S. T. Pantelides, G. Panchapakesan, S. Jesse, S. V. Kalinin, M. A. McGuire, A. N. Morozovska, P. Maksymovych, and N. Balke, Phys. Rev. Mater. 3, 024401 (2019).
  * Brehm  _et al._ (2020) J. A. Brehm, S. M. Neumayer, L. Tao, A. O’Hara, M. Chyasnavichus, M. A. Susner, M. A. McGuire, S. V. Kalinin, S. Jesse, P. Ganesh, S. T. Pantelides, P. Maksymovych, and N. Balke, Nat. Mater. 19, 43 (2020).
  * Neumayer  _et al._ (2020) S. M. Neumayer, L. Tao, A. O’Hara, J. Brehm, M. Si, P.-Y. Liao, T. Feng, S. V. Kalinin, D. Y. Peide, S. T. Pantelides, P. Maksymovych, and N. Balke, Phys. Rev. Appl. 13, 064063 (2020).
  * Liu  _et al._ (2016b) F. Liu, L. You, K. L. Seyler, X. Li, P. Yu, J. Lin, X. Wang, J. Zhou, H. Wang, H. He, S. T. Pantelides, W. Zhou, P. Sharma, X. Xu, P. M. Ajayan, J. Wang, and Z. Liu, Nat. Commun. 7, 12357 (2016b).
  * Maisonneuve  _et al._ (1995) V. Maisonneuve, M. Evain, C. Payen, V. B. Cajipe, and P. Molinie, J. Alloys Compd. 218, 157 (1995).
  * Maisonneuve  _et al._ (1997) V. Maisonneuve, V. B. Cajipe, A. Simon, R. Von Der Muhll, and J. Ravez, Phys. Rev. B 56, 10860 (1997).
  * Park  _et al._ (2017c) S. Y. Park, A. Kumar, and K. M. Rabe, Phys. Rev. Lett. 118, 087602 (2017c).
  * Qi and Rabe (2021) Y. Qi and K. M. Rabe, arXiv preprint arXiv:2103.16466 (2021).
  * Fu and Cohen (2000) H. Fu and R. E. Cohen, Nature 403, 281 (2000).
  * Wu and Cohen (2005) Z. Wu and R. E. Cohen, Phys. Rev. Lett. 95, 037601 (2005).
  * Sághi-Szabó  _et al._ (1998) G. Sághi-Szabó, R. E. Cohen, and H. Krakauer, Phys. Rev. Lett. 80, 4321 (1998).
  * Bellaiche and Vanderbilt (2000) L. Bellaiche and D. Vanderbilt, Phys. Rev. B 61, 7877 (2000).
  * Grinberg  _et al._ (2004) I. Grinberg, V. R. Cooper, and A. M. Rappe, Phys. Rev. B 69, 144118 (2004).
  * Hamann  _et al._ (2005) D. R. Hamann, X. Wu, K. M. Rabe, and D. Vanderbilt, Phys. Rev. B 71, 035117 (2005).
  * Cohen (2008) R. E. Cohen, in  _Piezoelectricity_ (Springer, 2008) pp. 471–492.
  * Liu and Cohen (2017) S. Liu and R. E. Cohen, Phys. Rev. Lett. 119, 207601 (2017).
  * Kim  _et al._ (2019) J. Kim, K. M. Rabe, and D. Vanderbilt, Phys. Rev. B 100, 104115 (2019).
  * Qi and Rappe (2021) Y. Qi and A. M. Rappe, Phys. Rev. Lett. 126, 217601 (2021).


#  Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence
Scores from Language Models Fine-Tuned with Human Feedback

Katherine Tian${}^{\phantom{*}\dagger}$
<EMAIL_ADDRESS>&Eric Mitchell∗‡
<EMAIL_ADDRESS>&Allan Zhou‡
Archit Sharma‡ &Rafael Rafailov‡ &Huaxiu Yao‡
Chelsea Finn‡ &Christopher D. Manning‡
†Harvard University ‡Stanford University ∗Equal contribution. Katherine
Tian,${}^{\phantom{*}\dagger}$ Eric Mitchell,∗‡ Allan Zhou,‡ Archit Sharma,‡
Rafael Rafailov‡
Huaxiu Yao,‡ Chelsea Finn,‡ Christopher D. Manning‡
†Harvard University ‡Stanford University
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>∗Equal contribution.

###### Abstract

A trustworthy real-world prediction system should produce well-calibrated
confidence scores; that is, its confidence in an answer should be indicative
of the likelihood that the answer is correct, enabling deferral to an expert
in cases of low-confidence predictions. Recent studies have shown that
unsupervised pre-training produces large language models (LMs) whose
conditional probabilities are remarkably well-calibrated. However, the most
widely-used LMs are fine-tuned with reinforcement learning from human feedback
(RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional
probabilities that are very poorly calibrated. In light of this perceived
weakness, we conduct a broad evaluation of methods for extracting confidence
scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find
that verbalized confidences emitted as output tokens are typically better-
calibrated than the model’s conditional probabilities on the TriviaQA, SciQ,
and TruthfulQA benchmarks, often reducing the expected calibration error by a
relative 50%.

Figure 1: Verbalized confidence scores (blue) are better-calibrated than log
probabilities (orange) for gpt-3.5-turbo. Raw model probabilities (top-left)
are consistently over-confident. Verbalized numerical probabilities (bottom)
are better-calibrated. Considering more answer choices (bottom-right) further
improves verbalized calibration (as in ‘Considering the Opposite’ in
psychology; Lord et al. (1985)). Verbalized expressions of likelihood (top-
right) also provide improved calibration. Bar height is average accuracy of
predictions in bin. Darker bars mean more predictions fall in that confidence
range. Results computed on SciQ.

##  1 Introduction

Real-world prediction systems invariably make errors. However, some mitigation
of these errors is possible if the system produces well-calibrated111i.e., the
confidence in a prediction accurately reflects the probability that the
prediction is correct (Guo et al., 2017). confidence estimates. In this case,
the system’s least confident predictions correspond to those that are most
likely to be incorrect, potentially allowing these predictions to be skipped
or overridden by a human. In the context of language models, one consequence
of poor calibration may be hallucination, where a language model confidently
asserts incorrect facts or reasoning. While the ability of very large LMs to
absorb and synthesize knowledge about the outside world has gained significant
attention (Brown et al., 2020; Roberts et al., 2020; Bubeck et al., 2023),
relatively little attention has been given to their well-calibratedness
(Kadavath et al., 2022). Further, most existing analyses of the calibratedness
of LLMs focus on models trained with maximum likelihood, while in practice,
the most widely-used LLMs (such as ChatGPT) are fine-tuned using methods such
as reinforcement learning from human feedback (Christiano et al., 2017). Some
findings suggest that RLHF-LMs may sacrifice well-calibrated predictions for
the sake of closer adherence to user instructions in dialogue (Kadavath et
al., 2022; OpenAI, 2023), as the reinforcement learning objective encourages
the model to allocate probability mass to the most preferred answer(s), rather
than matching the relative frequency of possible answers.

Figure 2: RLHF generally worsens the calibration of Llama-70B’s log
probabilities, as measured by ECE (lower is better) or AUC (higher is better).
However, this paper (Tables 1-5) will show that for several strong RLHF-LMs,
the model’s verbalized confidence is often better-calibrated than its log
probabilities, reversing some of this degradation. This reversal is strongest
for TruthfulQA, an adversarial dataset testing common misconceptions and other
difficult queries.

This paper evaluates several methods for extracting confidences about model
predictions from RLHF-LMs. Due to concerns that RLHF may cause systematic
overconfidence in the model’s probabilities (Figure 2), as well as the general
unavailability of per-token log-probabilities in widely used RLHF-LMs, we pay
particular attention to prompts that elicit verbalized probabilities, i.e.,
the model expresses its confidence in token-space, as either numerical
probabilities or another linguistic expression of uncertainty. We find that,
surprisingly, popular RLHF-LMs are able to directly verbalize confidence
scores that are better-calibrated than the model’s conditional probabilities
(estimated via sampling), without _any_ fine-tuning to learn verbalization. To
further improve calibration, we take inspiration from research in human
psychology showing that overconfidence can be mitigated by considering
alternative answers before responding (Lord et al., 1985; Mussweiler et al.,
2000). We show that prompting a model to produce several answer choices before
giving its confidence scores significantly improves calibration of verbalized
probabilities. Combined with temperature scaling (Guo et al., 2017), this
approach generally provides better calibration than model probabilities for
ChatGPT222gpt-3.5-turbo, accessed in June 2023.,
GPT-4333https://cdn.openai.com/papers/gpt-4-system-card.pdf, and Claude
2444https://www-files.anthropic.com/production/images/Model-Card-Claude-2.pdf
across three datasets, often reducing expected calibration error (ECE) by over
50%.

| TriviaQA |  | SciQ |  | TruthfulQA
---|---|---|---|---|---
Method |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$
Label prob. | 0.140 | 0.097 | 0.142 | 0.869 |  | 0.256 | 0.180 | 0.223 | 0.752 |  | 0.451 | 0.317 | 0.345 | 0.418
‘Is True’ prob. | 0.164 | 0.159 | 0.165 | 0.826 |  | 0.312 | 0.309 | 0.309 | 0.677 |  | 0.470 | 0.471 | 0.476 | 0.384
Entropy | — | — | — | 0.547 |  | — | — | — | 0.483 |  | — | — | — | 0.236
Verb. 1S top-1 | 0.068 | 0.076 | 0.138 | 0.879 |  | 0.234 | 0.084 | 0.214 | 0.744 |  | 0.389 | 0.256 | 0.322 | 0.545
Verb. 1S top-2 | 0.050 | 0.053 | 0.139 | 0.894 |  | 0.132 | 0.050 | 0.201 | 0.766 |  | 0.361 | 0.115 | 0.252 | 0.485
Verb. 1S top-4 | 0.054 | 0.057 | 0.144 | 0.896 |  | 0.065 | 0.051 | 0.209 | 0.763 |  | 0.203 | 0.189 | 0.284 | 0.455
Verb. 2S CoT | 0.110 | 0.123 | 0.168 | 0.830 |  | 0.323 | 0.246 | 0.296 | 0.683 |  | 0.419 | 0.259 | 0.292 | 0.551
Verb. 2S top-1 | 0.131 | 0.099 | 0.148 | 0.855 |  | 0.340 | 0.203 | 0.268 | 0.677 |  | 0.431 | 0.245 | 0.282 | 0.483
Verb. 2S top-2 | 0.047 | 0.045 | 0.147 | 0.887 |  | 0.169 | 0.040 | 0.201 | 0.768 |  | 0.395 | 0.101 | 0.224 | 0.517
Verb. 2S top-4 | 0.050 | 0.051 | 0.156 | 0.861 |  | 0.130 | 0.046 | 0.211 | 0.729 |  | 0.270 | 0.156 | 0.246 | 0.463
Ling. 1S human | 0.062 | 0.069 | 0.137 | 0.884 |  | 0.166 | 0.087 | 0.223 | 0.703 |  | 0.306 | 0.296 | 0.333 | 0.503
Ling. 1S-opt. | 0.058 | 0.066 | 0.135 | 0.878 |  | 0.064 | 0.068 | 0.220 | 0.674 |  | 0.125 | 0.165 | 0.270 | 0.492

Table 1: Measuring calibration of various methods for extracting confidences
from gpt-3.5-turbo (ChatGPT). The model’s conditional probabilities are
relatively poorly calibrated, whether using the model’s conditional
probability of the label given the query (Label prob.) or the probability
assigned to ‘True’ given the query, proposed answer, and a prompt asking if
the answer is correct (‘Is True’ prob.). Surprisingly, directly verbalizing a
probability (Verb. 1S and Verb. 2S) or an expression of confidence such as
‘highly likely’ (Ling. 1S) yields significantly better-calibrated confidence
estimates. 1S refers to one-stage prediction, where the model provides an
answer and confidence probability/expression together. 2S refers to two-stage
prediction, where the model first gives only an answer, and then in a second
stage a confidence. To color the table cells, for each column, we demean and
scale by a constant to obtain a shade in [-1,1], where cyan indicates better
and orange worse performance.

### Related Work.

Several studies have examined the calibration of large LMs (Lin et al., 2022a;
Park and Caragea, 2022; Kadavath et al., 2022; Xiao et al., 2022; Kuhn et al.,
2023), finding that combining large pre-trained LMs with temperature scaling
(Guo et al., 2017) produces very well-calibrated predictions (Kadavath et al.,
2022; Xiao et al., 2022; Kuhn et al., 2023). Other work focuses on the
tendency of language and dialogue models to use linguistic expressions of
uncertainty in a well-calibrated manner (Zhou et al., 2023; Mielke et al.,
2022). However, existing studies focus on LMs trained purely with unsupervised
learning (although Kadavath et al. (2022) briefly examine RLHF-LMs), while
widely used models in practice are fine-tuned with instruction-tuning or RLHF
(Christiano et al., 2017). RLHF has been shown to effectively leverage
annotations of human preferences to control sentiment (Ziegler et al., 2020),
improve summarization or instruction-following quality (Stiennon et al., 2022;
Ouyang et al., 2022), and inject behavioral priors of harmlessness (Bai et
al., 2022b, a). However, recent work has raised the question of whether or not
RLHF harms calibration (OpenAI, 2023). Our work is the first to show that
verbalized probabilities are often better-calibrated than the model’s
conditional probabilities for RLHF-LMs such as ChatGPT, GPT-4, and Claude, and
Llama-2-70B-Chat.

| TriviaQA |  | SciQ |  | TruthfulQA
---|---|---|---|---|---
Method |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$
Label prob. | 0.078 | 0.067 | 0.077 | 0.950 |  | 0.219 | 0.165 | 0.186 | 0.820 |  | 0.445 | 0.334 | 0.362 | 0.462
Verb. 1S top-1 | 0.024 | 0.038 | 0.084 | 0.937 |  | 0.201 | 0.084 | 0.165 | 0.843 |  | 0.350 | 0.156 | 0.227 | 0.622
Verb. 1S top-2 | 0.025 | 0.034 | 0.084 | 0.949 |  | 0.140 | 0.048 | 0.185 | 0.813 |  | 0.315 | 0.112 | 0.228 | 0.623
Verb. 1S top-4 | 0.041 | 0.039 | 0.081 | 0.959 |  | 0.056 | 0.059 | 0.185 | 0.815 |  | 0.198 | 0.144 | 0.245 | 0.619
Ling. 1S-human | 0.051 | 0.041 | 0.086 | 0.931 |  | 0.148 | 0.024 | 0.170 | 0.835 |  | 0.241 | 0.151 | 0.228 | 0.651
Ling. 1S-opt. | 0.056 | 0.051 | 0.088 | 0.927 |  | 0.028 | 0.052 | 0.172 | 0.828 |  | 0.082 | 0.105 | 0.212 | 0.632

Table 2: gpt-4’s verbalized probabilities are substantially better-calibrated
than the model probabilities themselves, even after temperature scaling,
similarly to gpt-3.5-turbo in Table 1.

##  2 Evaluating Calibration in RLHF-LMs

To study the calibration of RLHF-LMs, we conduct experiments with
gpt-3.5-turbo (ChatGPT), gpt-4 (GPT-4), claude-1 (Claude 1), claude-2 (Claude
2), and Llama-2-70b-chat (Llama-2-70B-Chat).

### Metrics.

We measure calibration with multiple metrics. To measure ECE (expected
calibration error; Guo et al. (2017)), we bin model predictions by their
confidence and measure the average accuracy of predictions in each confidence
bin. The ECE is defined as the average (squared) error between the average
accuracy and confidence within each bin, where each error is weighted by the
fraction of samples falling within the bin. We report raw ECE as well as ECE
with temperature scaling (ECE-t). Temperature scaling fits a single
temperature value $\beta$ to the model’s confidences to minimize negative log
likelihood (NLL) on the data, giving scaled probability $\tilde{p}_{i}$ of
class $i$ as $\tilde{p}_{i}\propto p_{i}^{\beta}$. See Figure 1 for a
depiction of ECE binning. Although ECE is a standard and interpretable measure
of calibration error, it completely fails to capture the confidences’
discriminative power.555For binary classification, a system that guesses
randomly and outputs 50% confidence each time has perfect ECE. We therefore
also report Brier Score (BS; Brier (1950)) on temperature-scaled confidences
(BS-t), a proper scoring rule Ovadia et al. (2019) that is the mean squared
error between the confidences and the correctness labels. Finally, we assess
calibration using a metric from the selective classification literature
Geifman and El-Yaniv (2017), specifically, the area under the curve of
selective accuracy and coverage (AUC).

### Datasets.

Our experiments use three question-answering datasets assessing factual
knowledge. TriviaQA (Joshi et al., 2017) contains 650k question-answer pairs
gathered by trivia enthusiasts; SciQ (Welbl et al., 2017) contains
approximately 14k crowdsourced science exam question-answer pairs; TruthfulQA
(Lin et al., 2022b) contains 817 questions designed to test language models’
tendency to mimic human falsehoods. We sample 1000 questions from the
validation split of TriviaQA (rc.web.nocontext) and SciQ and all 817 questions
from the validation split of TruthfulQA (generation) for our experiments.

### Evaluation protocol.

For each dataset, we generate a response and corresponding confidence from
each method on each of the evaluation questions. Because calibration
essentially quantifies the relationship between model confidence and
correctness, computing correctness is crucial to accurate measurements of
calibration. However, we find doing so to be a challenge, especially in
datasets where only a single ground-truth answer (but not aliases or
semantically equivalent rephrases) is provided. To avoid excessive false
negatives in our correctness computation as a result of exact-match
evaluation, we use either GPT-4 or GPT-3.5 to evaluate whether a response is
essentially equivalent to the ground truth answer; see Appendix C for the
complete equivalence-checking procedure.

| TriviaQA |  | SciQ |  | TruthfulQA
---|---|---|---|---|---
Method |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$
Label prob. | 0.074 | 0.079 | 0.117 | 0.915 |  | 0.216 | 0.149 | 0.195 | 0.786 |  | 0.432 | 0.304 | 0.335 | 0.418
Verb. 1S top-1 | 0.049 | 0.059 | 0.160 | 0.839 |  | 0.265 | 0.103 | 0.247 | 0.663 |  | 0.440 | 0.134 | 0.204 | 0.411
Verb. 1S top-2 | 0.046 | 0.047 | 0.158 | 0.875 |  | 0.207 | 0.040 | 0.225 | 0.693 |  | 0.450 | 0.085 | 0.197 | 0.409
Verb. 1S top-4 | 0.075 | 0.079 | 0.176 | 0.814 |  | 0.151 | 0.057 | 0.226 | 0.667 |  | 0.372 | 0.105 | 0.183 | 0.377
Ling. 1S human | 0.053 | 0.050 | 0.151 | 0.867 |  | 0.253 | 0.118 | 0.245 | 0.664 |  | 0.443 | 0.358 | 0.340 | 0.384
Ling. 1S-opt. | 0.074 | 0.060 | 0.149 | 0.863 |  | 0.089 | 0.082 | 0.238 | 0.623 |  | 0.139 | 0.148 | 0.228 | 0.350

Table 3: Claude-1 produces similar- or better-calibrated log probabilities to
gpt-3.5-turbo, but is less able to verbalize well-calibrated confidences,
compared to models in the GPT family of RLHF-LMs. Claude-1 has since been
deprecated.

| TriviaQA |  | SciQ |  | TruthfulQA
---|---|---|---|---|---
Method |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$
Label prob. | 0.089 | 0.089 | 0.137 | 0.882 |  | 0.181 | 0.176 | 0.237 | 0.762 |  | 0.409 | 0.368 | 0.405 | 0.319
Verb. 1S top-1 | 0.072 | 0.071 | 0.141 | 0.903 |  | 0.204 | 0.054 | 0.201 | 0.776 |  | 0.345 | 0.115 | 0.215 | 0.573
Verb. 1S top-2 | 0.049 | 0.054 | 0.133 | 0.918 |  | 0.134 | 0.041 | 0.211 | 0.754 |  | 0.359 | 0.085 | 0.223 | 0.491
Verb. 1S top-4 | 0.072 | 0.063 | 0.158 | 0.890 |  | 0.048 | 0.052 | 0.216 | 0.711 |  | 0.274 | 0.075 | 0.208 | 0.473
Ling. 1S human | 0.085 | 0.061 | 0.151 | 0.878 |  | 0.238 | 0.026 | 0.209 | 0.756 |  | 0.381 | 0.242 | 0.305 | 0.530
Ling. 1S-opt. | 0.060 | 0.070 | 0.151 | 0.874 |  | 0.049 | 0.056 | 0.214 | 0.738 |  | 0.099 | 0.130 | 0.266 | 0.446

Table 4: Claude-2 has weaker conditional probabilities than Claude-1 and
GPT-*, but its verbalized calibration provides consistent improvement over
conditional probabilities at a level comparable to GPT-3.5 and surpassing
GPT-* on TruthfulQA.

| TriviaQA |  | SciQ |  | TruthfulQA
---|---|---|---|---|---
Method |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$ |  |  ECE $\downarrow$ |  ECE-t $\downarrow$ |  BS-t $\downarrow$ |  AUC $\uparrow$
Label prob. | 0.151 | 0.124 | 0.156 | 0.865 |  | 0.266 | 0.189 | 0.243 | 0.707 |  | 0.405 | 0.361 | 0.396 | 0.407
Verb. 1S top-1 | 0.071 | 0.067 | 0.186 | 0.793 |  | 0.196 | 0.053 | 0.239 | 0.648 |  | 0.386 | 0.172 | 0.266 | 0.502
Verb. 1S top-2 | 0.060 | 0.073 | 0.194 | 0.815 |  | 0.153 | 0.032 | 0.230 | 0.667 |  | 0.340 | 0.037 | 0.227 | 0.440
Verb. 1S top-4 | 0.069 | 0.079 | 0.182 | 0.816 |  | 0.105 | 0.043 | 0.229 | 0.648 |  | 0.231 | 0.102 | 0.237 | 0.465
Ling. 1S human | 0.179 | 0.115 | 0.195 | 0.749 |  | 0.071 | 0.101 | 0.252 | 0.603 |  | 0.376 | 0.366 | 0.383 | 0.407
Ling. 1S-opt. | 0.077 | 0.068 | 0.186 | 0.779 |  | 0.019 | 0.042 | 0.236 | 0.590 |  | 0.047 | 0.051 | 0.239 | 0.435

Table 5: With Llama2-70B-Chat, verbalized calibration provides improvement
over conditional probabilities across some metrics, but the improvement is
much less consistent compared to GPT-* and Claude-*.

### Methods.

We compare a wide variety of methods for extracting confidence estimates from
LLMs. For a comprehensive list of the prompts used for each method, see
Appendix Table 6.

First, we consider two methods that leverage the true conditional distribution
of the model to generate confidence scores. The simplest is Label prob., which
uses the conditional probability distribution $p(y|x)$ of the model given a
question $x$, which we estimate using $n=10$ samples, since many RLHF-LMs are
closed-source and do not offer per-token probabilities.666We evaluated
gpt-3.5-turbo on all three datasets using $n=20$ samples, but the calibration
did not meaningfully improve, so we always use $n=10$ to reduce API
costs.777For each closed LM, we use its default sampling parameters (top-p 1.0
for GPT-* and top-p 0.7 for Claude). For Llama-2, we use temperature 1.0 and
top-p 1.0. We return the most common answer, using the LLM-based equivalence
function to determine when two lexically different answers are semantically
equivalent. In a variation of the method described by Kadavath et al. (2022)
(again, we use samples since model probabilities are not available), ‘Is True’
prob. samples a single answer $\hat{y}$ from the model given a question $x$,
and the probability it is true is estimated by the probability the model
assigns to ‘True’ when asked if the given answer is true (where once again the
probabilities are estimated via samples), i.e., $p(\texttt{True}|x,\hat{y})$.

Next, we consider methods that extract confidence scores through verbalization
(Lin et al., 2022a), i.e., where the model expresses its confidence in token
space, either with numerical probabilities or linguistic expressions of
likelihood.888However, note that none of the methods described fine-tune the
model to perform better on verbalization. First, Verb. 1S top-$k$ prompts the
model to produce $k$ guesses and a probability that each is correct all in a
single response (i.e., ‘1 stage’). We take the highest-probability prediction
and its associated probability as the model’s output and confidence. Verb. 2S
top-$k$ similarly uses numerical probabilities, except the model is first
asked to provide only its answers, and afterwards, in a second round of
dialogue, asked to assign probabilities of correctness to each answer (i.e.,
‘2 stages’). Verb. 2S CoT uses a chain-of-thought prompt before giving a
single answer, and in a second round of dialogue, the model is prompted to
assign a probability to that answer (with the chain of thought present in the
model’s context). Ling. 1S-human uses linguistic likelihood expressions,
rather than numerical probabilities, to express uncertainty. The model is
prompted to assign confidences to its guesses by choosing from a set of
linguistic expressions of uncertainty: {Almost certain, Likely, …, Almost no
chance}. Each linguistic likelihood expression is mapped to a probability
using responses from a human survey on social media with 123 respondents
(Fagen-Ulmschneider, 2023). Ling. 1S-opt. uses a held out set of calibration
questions and answers to compute the average accuracy for each likelihood
expression, using these ‘optimized’ values instead. Expressions that are not
used for at least $\frac{1}{N}$ of questions, where $N$ is the number of
calibration questions, simply use the human probability.

##  3 Results

Tables 1–5 show the results of evaluating various methods for extracting
confidence from RLHF-LMs on gpt-3.5-turbo, gpt-4, claude-1, claude-2, and
Llama-2-70b-chat, respectively. We distill several key conclusions from these
experiments. 1. Large RLHF-LMs can often directly verbalize better-calibrated
confidences (either a numerical confidence probability or an expression such
as ‘highly likely’) than the models’ conditional probabilities. 2. Among the
methods for verbalizing probabilities directly, we observe that generating and
evaluating multiple hypotheses improves calibration (see Figure 1), similarly
to humans (Lord et al., 1985), and corroborating a similar finding in LMs
(Kadavath et al., 2022). 3. Language models can express their uncertainty with
numerical probabilities as well or better than with words, which is surprising
in light of long-standing difficulties in representing numbers in language
models (Thawani et al., 2021). 4. Chain-of-thought prompting does not improve
verbalized calibration (see Appendix Figure 5 for additional CoT results). 5.
The calibration of both Claude models’ conditional probabilities roughly falls
between gpt-3.5-turbo and gpt-4; however, while Claude 1 is much weaker at
verbalizing its confidence, Claude 2 is generally a bit stronger than
gpt-3.5-turbo at verbalizing. The verbal calibration of the open source model
Llama-2-70b-chat is generally weaker than that of closed source models but
still demonstrates improvement over its conditional probabilities by some
metrics, and does so most clearly on TruthfulQA.

##  4 Discussion

In summary, we study the calibration of widely used RLHF-LMs. We first
replicate the finding for GPT-4 (OpenAI, 2023) that RLHF can worsen the
calibration of a model’s conditional probabilities using the open-source
Llama-2-70B base and chat models (Figure 2). To mitigate this regression and
ease extraction of calibrated confidence scores for models for which log
probabilities are not available, we propose and study new methods that can
elicit calibrated confidences from RLHF-LMs by prompting the model to
verbalize its confidence in token space. We find verbalized probabilities are
better-calibrated than conditional probabilities across several closed models,
with mixed results for Llama-2-70B-Chat.

Our results raise several questions for future work. Most notably, the
difference between GPT-*, Claude-*, and Llama-2’s ability to verbalize
confidence is significant. What factors are important for learning this skill?
Additionally, the 1-stage and 2-stage verbalized numerical confidence prompts
sometimes differ drastically in the calibration of their confidences. How can
we reduce sensitivity of a model’s calibration to the prompt? Going beyond
question-answering, can we leverage good calibration in short-answer settings
to improve the reliability of long-form generations, perhaps by breaking down
long-form generation into a sequence of short questions? Finally, to what
extent does a language model’s calibration depend on the domain; do our
conclusions in the context of factual recall hold in the context of reasoning
or arithmetic? Answering these questions provides one path toward building
more trustworthy and useful language systems.

Limitations. While our work demonstrates a promising new approach to
generating calibrated confidences through verbalization, there are limitations
that could be addressed in future work. First, our experiments are focused on
factual recall-oriented problems, and the extent to which our observations
would hold for reasoning-heavy settings is an interesting open question.
Additionally, the lack of technical details available for many state-of-the-
art closed RLHF-LMs may limit our ability to understand what factors enable a
model to verbalize well-calibrated confidences and differences in this ability
across different models. Finally, our study is limited to short-form question-
answering; future work should extend this analysis to longer-form generation
settings.

Acknowledgements. CF and CDM are CIFAR Fellows. EM gratefully acknowledges
funding from a Knight-Hennessy Graduate Fellowship. AZ is supported by the NSF
graduate research fellowship program. This research was supported in part by
Juniper Networks, Apple, and ONR grant N00014-20-1-2675. The authors thank
Yoonho Lee and Noah Goodman for helpful feedback on calibration metrics and
experiment design.

## References

  * Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a.  Training a helpful and harmless assistant with reinforcement learning from human feedback.
  * Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b.  Constitutional AI: Harmlessness from ai feedback.
  * Brier (1950) Glenn W. Brier. 1950.  Verification of Forecasts Expressed in Terms of Probability.  _Monthly Weather Review_ , 78(1):1–3.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with GPT-4.  ArXiv preprint arXiv:2303.12712.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Fagen-Ulmschneider (2023) Wade Fagen-Ulmschneider. 2023.  Perception of probability words.  Ms., UIUC, 05-24-2023.
  * Geifman and El-Yaniv (2017) Yonatan Geifman and Ran El-Yaniv. 2017.  Selective classification for deep neural networks.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 4885–4894, Red Hook, NY, USA. Curran Associates Inc.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017.  On calibration of modern neural networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 1321–1330. PMLR.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022.  Language models (mostly) know what they know.  Arxiv arxiv:2207.05221.
  * Kuhn et al. (2023) Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  In _The Eleventh International Conference on Learning Representations_.
  * Lin et al. (2022a) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022a.  Teaching models to express their uncertainty in words.  _Transactions on Machine Learning Research_.
  * Lin et al. (2022b) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022b.  TruthfulQA: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  * Lord et al. (1985) Charles Lord, Mark Lepper, and Elizabeth Preston. 1985.  Considering the opposite: A corrective strategy for social judgment.  _Journal of personality and social psychology_ , 47:1231–43.
  * Mielke et al. (2022) Sabrina J. Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. 2022.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872.
  * Mussweiler et al. (2000) Thomas Mussweiler, Fritz Strack, and Tim Pfeiffer. 2000.  Overcoming the inevitable anchoring effect: Considering the opposite compensates for selective accessibility.  _Personality and Social Psychology Bulletin_ , 26(9):1142–1150.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. 2019.  Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , Red Hook, NY, USA. Curran Associates Inc.
  * Park and Caragea (2022) Seo Yeon Park and Cornelia Caragea. 2022.  On the calibration of pre-trained language models using mixup guided by area under the margin and saliency.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5364–5374, Dublin, Ireland. Association for Computational Linguistics.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Stiennon et al. (2022) Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022.  Learning to summarize from human feedback.
  * Thawani et al. (2021) Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021.  Representing numbers in NLP: a survey and a vision.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 644–656, Online. Association for Computational Linguistics.
  * Welbl et al. (2017) Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017.  Crowdsourcing multiple choice science questions.  _ArXiv_ , abs/1707.06209.
  * Xiao et al. (2022) Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis-Philippe Morency. 2022.  Uncertainty quantification with pre-trained language models: A large-scale empirical analysis.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7273–7284, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhou et al. (2023) Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. 2023.  Navigating the grey area: Expressions of overconfidence and uncertainty in language models.
  * Ziegler et al. (2020) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020.  Fine-tuning language models from human preferences.

##  Appendix A Additional Results

Here, we include the likelihood expression usage distribution for gpt-3.5 and
gpt-4 in Figures 3 and 4, respectively. gpt-3.5 is systematically less
confident for TruthfulQA. The contrast between model confidence for TriviaQA
and SciQ compared with TruthfulQA is even more stark for gpt-4.

We also provide additional calibration results for chain-of-thought methods.
We compare a one-stage verbalized CoT prompt (Verb. 1S CoT), a two-stage
verbalized CoT prompt (Verb. 2S CoT), and a two-stage verbalized method that
uses CoT just before eliciting the numerical confidence (Verb. 2S Cot Prob)
instead of before the guess, as shown for gpt-3.5 on Trivia QA, SciQ, and
Truthful QA in Figure 5. We find that CoT does not noticeably improve
calibration across any setting or dataset.

Figure 3: gpt-3.5-turbo usage rate of each likelihood expression; the model
displays much lower verbalized confidence on TruthfulQA than on standard
factual recall problems. Figure 4: gpt-4 usage rate of each likelihood
expression; the model displays markedly lower verbalized confidence on
TruthfulQA than on standard factual recall problems. Figure 5: Expected
calibration error is not consistently improved for any CoT prompt variant on
gpt-3.5-turbo.

##  Appendix B Fitting Procedure for Temperature and Probabilities for
Linguistic Expressions

To fit the temperature that is used to compute ECE-t and BS-t we split our
total data into 5 folds. For each fold, we use it once to fit a temperature
and evaluate metrics on the remaining folds. We find that fitting the
temperature on 20% of the data yields relatively stable temperatures across
folds. We report the average temperature-scaled ECE and BS as ECE-t and BS-t.

To compute ECE and AUC for Ling. 1S-opt., we similarly split our total data
into 5 folds, using 4 folds to fit the probabilities behind each linguistic
expression of confidence, then evaluating on the remaining fold. To compute
ECE-t and BS-t for Ling. 1S-opt, we hold out one of the 5 folds to fit
temperature. We use 3 folds to fit probabilities for linguistic expressions,
compute the temperature based on these probabilities on the temperature set,
and evaluate metrics on the last fold. We then average metrics across all 20
rotations of folds.

##  Appendix C Prompt Templates

Method |  Template
---|---
Label prob. |  Provide your best guess for the following question. Give ONLY the guess, no other words or explanation.\n\nFor example:\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nThe question is:${THE_QUESTION}
‘Is True’ prob. |  Question: ${QUESTION}\nProposed Answer: ${ANSWER}\nIs the proposed answer:\n\t(A) True or\n\t(B) False?\n The proposed answer is:
Verb. 1S top-1 |  Provide your best guess and the probability that it is correct (0.0 to 1.0) for the following question. Give ONLY the guess and probability, no other words or explanation. For example:\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\n Probability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!>\n\nThe question is: ${THE_QUESTION}
Verb. 1S top-$k$ |  Provide your ${k} best guesses and the probability that each is correct (0.0 to 1.0) for the following question. Give ONLY the guesses and probabilities, no other words or explanation. For example:\n\nG1: <first most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nP1: <the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!> ... G${k}: <${k}-th most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nP${k}: <the probability between 0.0 and 1.0 that G${k} is correct, without any extra commentary whatsoever; just the probability!> \n\nThe question is: ${THE_QUESTION}
Verb. 2S CoT |  Provide your best guess for the following question. Before giving your answer, provide a step-by-step explanation of your thought process. Then on a new line give the guess with no other words or explanation.\n\nFor example:\n\nExplanation: <one sentence step-by-step explanation of your thought process>\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nThe question is: ${THE_QUESTION} Provide the probability that your guess is correct. Give ONLY the probability, no other words or explanation.\n\nFor example:\n\nProbability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!>\n
Verb. 2S top-1 |  Provide your best guess for the following question. Give ONLY the guess, no other words or explanation.\n\nFor example:\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nThe question is:${THE_QUESTION} Provide the probability that your guess is correct. Give ONLY the probability, no other words or explanation.\n\nFor example:\n\nProbability: <the probability between 0.0 and 1.0 that your guess is correct, without any extra commentary whatsoever; just the probability!>\n
Verb. 2S top-$k$ |  Provide your ${k} best guesses for the following question. Give ONLY the guesses, no other words or explanation. For example:\n\nG1: <first most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nP1: <the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!> ... G${k}: <${k}-th most likely guess, as short as possible; not a complete sentence, just the guess!>\n\nThe question is:${THE_QUESTION} Provide the probability that each of your guesses is correct. Give ONLY the probabilities, no other words or explanation.\n\nFor example:\n\nP1: <the probability between 0.0 and 1.0 that G1 is correct, without any extra commentary whatsoever; just the probability!>\n... P${k}: <the probability between 0.0 and 1.0 that G${k} is correct, without any extra commentary whatsoever; just the probability!>
Ling. 1S |  Provide your best guess for the following question, and describe how likely it is that your guess is correct as one of the following expressions: ${EXPRESSION_LIST}. Give ONLY the guess and your confidence, no other words or explanation. For example:\n\nGuess: <most likely guess, as short as possible; not a complete sentence, just the guess!>\nConfidence: <description of confidence, without any extra commentary whatsoever; just a short phrase!>\n\nThe question is: ${THE_QUESTION}
Table 6: Prompt templates for each method evaluated. Methods above the double
line use multiple samples in order to estimate confidence scores; methods
below the double line use the verbalized confidences directly, requiring only
a single sample.

The prompt template for each sampling method is provided in Table 6. The
question is substituted for the variable ${THE_QUESTION} in each prompt. To
evaluate answer correctness, we use gpt-3.5-turbo for SciQ and TruthfulQA and
gpt-4 for TriviaQA due to gpt-3.5-turbo’s high disagreement with a human
evaluator on TriviaQA. Using the ground truth answer as ${GOLD_ANSWER} and the
model-generated answer as ${PRED_ANSWER}, we use the following prompt
template:

Are the following two answers to my question Q semantically equivalent?\n\nQ:
${THE_QUESTION}\nA1: ${GOLD_ANSWER}\nA2: ${PRED_ANSWER}\n\nPlease answer with
a single word, either ‘‘Yes." or ‘‘No.", and explain your reasoning.