11institutetext:  Institut für Astronomie und Astrophysik, Universität
Tübingen, Auf der Morgenstelle 10, D-72076, Germany 22institutetext: Astronomy
Unit, School of Physics and Astronomy, Queen Mary University of London, London
E1 4NS, UK

# Parking planets in circumbinary discs

Anna B. T. Penzlin Parking planets in circumbinary discsParking planets in
circumbinary discs Wilhelm Kley Parking planets in circumbinary discsParking
planets in circumbinary discs Richard P. Nelson Parking planets in
circumbinary discsParking planets in circumbinary discs

The Kepler space mission discovered about a dozen planets orbiting around
binary stars systems. Most of these circumbinary planets lie near their
instability boundaries at about 3 to 5 binary separations. Past attempts to
match these final locations through an inward migration process were only
successful for the Kepler-16 system. Here, we study 10 circumbinary systems
and try to match the final parking locations and orbital parameters of the
planets with a disc driven migration scenario.

We performed 2D locally isothermal hydrodynamical simulations of circumbinary
discs with embedded planets and followed their migration evolution using
different values for the disc viscosity and aspect ratio. We found that for
the six systems with intermediate binary eccentricities
($0.1\leq\mathrm{e_{bin}}\leq 0.21$) the final planetary orbits matched the
observations closely for a single set of disc parameters, specifically a disc
viscosity of $\alpha=10^{-4}$, and an aspect ratio of $H/r\sim 0.04$. For
these systems the planet masses were large enough to open at least a partial
gap in their discs as they approach the binary, forcing the discs to become
circularized and allowing for further migration towards the binary, leading to
good agreement with the observed planetary orbital parameters.

For systems with very small or large binary eccentricities the match was not
as good because the very eccentric discs and large inner cavities in these
cases prevented close-in planet migration. In test simulations with higher
than observed planet masses better agreement could be found for those systems.

The good agreement for 6 out of the 10 modelled systems, where the relative
difference between observed and simulated final planet orbit is $\leq 10\%$,
strongly supports the idea that planet migration in the disc brought the
planets to their present locations.

###### Key Words.:

Hydrodynamics – Binaries: general – Accretion, accretion discs – Planets and
satellites: formation – Protoplanetary discs

##  1 Introduction

The first circumbinary planet (CBP) discovered by the Kepler space mission was
Kepler-16b (Doyle et al., 2011). Since then about a dozen more circumbinary
p-type planets have been discovered (see overview in Martin (2019) and Table
1). In 2012, multiple CBPs around Kepler-47 were detected (Orosz et al.,
2012b, 2019). Later, the Planet Hunters project discovered a planet in a
quadruple star system PH1b, also named Kepler-64b (Schwamb et al., 2013). The
most recent discovery is TOI-1338b by the TESS mission (Kostov et al., 2020).

All of the above systems, and most of the discovered innermost CBPs, share a
set of properties in their orbits. Firstly, they are in good alignment with
the binary’s orbital plane, hinting at formation in a protoplanetary disc that
was aligned with the binary. Secondly, the planetary orbits (the semi-major
axes) are all near 3.5 binary separations, which is close to the instability
limit (Dvorak, 1986). Additionally, the eccentricities of the planetary orbits
are small. All of this points to a common formation mechanism for most
observed CBPs in aligned, coplanar discs.

Concerning the general stability of circumbinary discs it has been suggested
that, in addition to the coplanar situation, the perpendicular case is also a
stable binary-disc configuration (Martin & Lubow, 2017; Lubow & Martin, 2018;
Cuello & Giuppone, 2019). However, no planets have been observed in
perpendicular orbits yet, due to observational constraints.

In-situ formation of planets at these close orbits around binaries are
unlikely. Tidal forces close to the binary cause perturbations in the inner
disc that inhibit planetesimal and dust accretion (Paardekooper et al., 2012;
Silsbee & Rafikov, 2015; Meschiari, 2012), and turbulence induced by
hydrodynamical parametric instabilities can dramatically reduce pebble
accretion efficiencies (Pierens et al., 2020). A more likely formation
scenario is planet formation in the outer disc and subsequent disk-driven
migration to the observed close orbits (Pierens & Nelson, 2008; Bromley &
Kenyon, 2015). However, this does not in itself explain the particular
stopping positions of most CBPs.

Kepler | 38 | 35 | 453 | 64 | 34 | 47 | 16 | 413 | 1661 | TOI-1338
---|---|---|---|---|---|---|---|---|---|---
$\mathrm{M_{bin}}[\mathrm{M_{sun}}]$ | 1.20 | 1.70 | 1.14 | 1.77 | 2.07 | 1.32 | 0.89 | 1.36 | 1.10 | 1.34
$\mathrm{q_{bin}}$ | 0.26 | 0.91 | 0.21 | 0.28 | 0.97 | 0.35 | 0.29 | 0.66 | 0.31 | 0.28
$\mathrm{e_{bin}}$ | 0.10 | 0.14 | 0.05 | 0.21 | 0.52 | 0.02 | 0.16 | 0.04 | 0.11 | 0.16
$\mathrm{m_{p}}[\mathrm{m_{jup}}]$ | 0.38 | 0.12 | 0.05 | 0.1 | 0.22 | 0.05 | 0.33 | 0.21 | 0.05 | 0.10
$\mathrm{a_{p}}\,[\mathrm{a_{bin}}]$ | 3.16 | 3.43 | 4.26 | 3.64 | 4.76 | 3.53 | 3.14 | 3.55 | 3.39 | 3.49
$\mathrm{e_{p}}$ | 0.03 | 0.04 | 0.04 | 0.05 | 0.18 | 0.03 | 0.006 | 0.12 | 0.06 | 0.09
$\mathrm{m_{p}}/\mathrm{M_{bin}}$ [$\times 10^{-5}$] | 30 | 7.1 | 4.3 | 5.4 | 10 | 3.7 | 35 | 15 | 4.6 | 6.8
Table 1: Table of observed circumbinary parameters. $\mathrm{M_{bin}}$,
$\mathrm{q_{bin}}$ and $\mathrm{e_{bin}}$ are the total mass, the mass ratio
($M_{2}/M_{1}$) and the eccentricity of the binary star, $\mathrm{m_{p}}$,
$\mathrm{a_{p}}$, $\mathrm{e_{p}}$ are the mass, semi-major axis and
eccentricity of the planet on its orbit. (See reference Orosz et al. (2012a),
Welsh et al. (2012), Welsh et al. (2015), Schwamb et al. (2013), Orosz et al.
(2019), Doyle et al. (2011), Kostov et al. (2014), Kostov et al. (2020), Socia
et al. (2020))

Several studies were already dedicated to investigate planet migration in
circumbinary discs (Pierens & Nelson, 2008, 2013; Kley & Haghighipour, 2014,
2015; Kley et al., 2019). Due to gravitational torques the binary clears out
an inner cavity and hence planet migration is stopped at the inner edge of the
disk at a location close to the binary, but not yet matching the observations
in most cases studied. Numerical simulations showed that the inner disc’s
shape and hence the final planet orbital parameters depend on disc parameters
such as viscosity and scale height, and the mass ratio and eccentricity of the
binary (Mutter et al., 2017; Thun & Kley, 2018).

Two recent studies by Mutter et al. (2017) and Kley et al. (2019) were able to
reproduce the observed CBP orbits for Kepler-16b and Kepler-38b. In this work
we plan to find a common formation scenario for most observed CBPs using a
sample of 10 planets, as described in Table 1.

To limit the parameter space, we will consider discs, in agreement with the
low $\alpha$-viscosity that was observed in the D-sharp survey (Dullemond et
al., 2018) and scale heights comparable to the results of Kley et al. (2019).
There, we showed also that the geometry of the disc, specifically the
eccentricity of the inner cavity, can be influenced by an embedded planet. Gap
opening planets will circularize the disc while lighter planets or discs with
higher viscosity will have eccentric holes. We will explore the effect of a
wider disc parameter range on the disc architecture in a future study.

To study the parking of planets we performed numerical hydrodynamical
simulations of circumbinary discs using 3 different parameter sets for the
disc physics in order to find a likely combination of viscosity and scale
height that results in disc shapes that allow for suitable stopping locations
of the planets. Hence, this study extends the work of Pierens & Nelson (2013)
where they looked at a subset of 3 systems. The binary and planet parameters
used in our simulations are those given by the observations, as presented in
Table 1.

In Section 2 the adopted model assumptions and initial conditions are defined.
Section 3 shows the results of the simulations, where we discuss the influence
of viscosity on the planet migration in general and show the effects of
different sets of disc parameters on the final orbits of the observed Kepler
planets. We will discuss the difficulties of reaching the final orbit and the
limitations of our model in Section 4 and summarise the findings.

##  2 Simulations

We used the Pluto Code (Mignone et al., 2007) in the GPU version by Daniel
Thun (Thun et al., 2017) to simulate 2D hydrodynamic disc models. We chose a
cylindrical grid stretching from 1 to $40\,\mathrm{a_{bin}}$ with 684
logarithmically spaced radial grid cells and 584 azimuthal cells. The general
design of the simulation follows the setup of Thun & Kley (2018).

We modelled nine of the known Kepler binary planet systems, and TOI-1338, for
which we used the observed binary and planet properties, as displayed in Table
1.

For the surface density profile we use $\Sigma\propto R^{-3/2}$ with a disc
mass of $\mathrm{M_{d}}=0.01\,\mathrm{M_{bin}}$. To speed up the equilibration
process for the disc we initialized the surface density with an inner cavity
reaching to about $2.5\,\mathrm{a_{bin}}$. Inside of this region the disc will
be cleared due to the binary’s action.

All discs are evolved initially for $>25\,000$ binary orbits,
$\mathrm{T_{bin}}$, without a planet being present, depending on the specific
system setup.A typical time scale for a locally isothermal protoplanetary disc
around a binary to reach a convergent quasi-equilibrium state is a few
$10\,000\,T_{\mathrm{bin}}$. Hence, we used in this work locally isothermal
models as they converge faster than viscously heated radiative disc which can
need up to $200\,000\,T_{\mathrm{bin}}$, but eventually reach comparable final
states with a pressure profil consistent with an unflared inner disc that has
a constant aspect ratio between 0.05 and 0.03 depending on the system, see
Kley et al. (2019) for details. During this initialization phase the binary is
not influenced by the disc. As the final planetary orbits will depend on disc
properties, we simulated discs for the observed systems in Table 1 for
different values of $h$ and $\alpha$. The viscously heated, radiative models
in Kley et al. (2019) resulted in an unflared inner disc. Therefore we chose
here a constant aspect ratio, $h=H/R$. Based on our previous experience (Thun
& Kley, 2018; Kley et al., 2019), we chose the following three combinations:
$h=0.04,\alpha=10^{-3};h=0.04,\alpha=10^{-4}$; and $h=0.03,\alpha=10^{-4}$,
only for Kepler-38 we used additionally $h=0.05$.

After the disc reached a convergent behaviour we added a planet with the
observed mass (see Table 1) in the outer disc at a few $\mathrm{a_{bin}}$
beyond the maximum of the surface density in the disc. In cases where only a
upper planet-mass limit was observed like Kepler-38, we used the observed
upper limit. This might overestimate the actual mass of a planet, but is still
in aggrement with observations. In the whole sample the planet masses range
from $0.05\,M_{\mathrm{jup}}$ to $0.38\,M_{\mathrm{jup}}$ The disc exerts
gravitational forces on the planet which will start migrating through the
disc. The disc forces acting on the central binary star are also switched on
at this time, but during the time span of the simulations the binary elements
do not change substantially.

Figure 1: Surface density for discs around a Kepler-35 like binary for
different $\alpha$-values and an aspect ratio of 0.04. The dashed white line
denotes a fitted ellipse (Thun & Kley, 2018) which marks the edge of the inner
cavity with the parameters displayed in the top left corner.  Figure 2: Discs
for a Kepler-35 like system with different $\alpha$-values with a planet
embedded for 29 900 binary orbits. The dashed white line marks the edge of the
inner cavity with the parameters displayed in the top right corner. The light
blue line marks the orbits of the planet with the parameter displayed in the
top left corner.

##  3 Results

In this section we present the results of our numerical studies of the systems
quoted in Tab. 1. We will first present results for different viscosities on
the specific system Kepler-35, and then concentrate on the complete set of
systems. An overview of the disc shapes that resulted in planetary evolutions
that matched best with the observations is given below in Fig. 3.

Figure 3: The surface density of 8 modelled Kepler circumbinary systems,
listed in Table 1, before the planet was embedded. The chosen $h$ and $\alpha$
correspond to those models that eventually could reproduce the observed final
planetary orbits best. The white dashed ellipse marks the inner cavity edge
with the parameters shown in the top left. Figure 4: Migration history for all
planets that get close to the observed orbit. Semi-major axis and eccentricity
are scaled with the observed $\mathrm{a_{obs}}$ and $\mathrm{e_{obs}}$, see
Tab. 1. The grey area marks the success criterion, where the final parking
position of the planet deviates by no more than 10% from the observed value,
i.e. $\Delta a_{p}/a_{p,obs}\leq 10\%$. For the discs we chose
$\alpha=10^{-4}$ and $h=0.04$, plus one additional model for Kepler-38 using
the stated values.

###  3.1 The influence of viscosity

To analyse in more detail the impact of viscosity on the inner cavity we
present results on the Kepler-35 system in more detail, first the disc
structure without a planet and then the impact of the planet. The binary
system Kepler-35 has a mass ratio close to unity, $\mathrm{q_{bin}}=0.91$, and
a relatively small eccentricity of $\mathrm{e_{bin}}=0.14$ which puts the
system on the lower branch of the bifurcation diagram in the precession period
versus cavity size graph (Thun & Kley, 2018). In Figure 1 equilibrium discs
with different $\alpha$-values are displayed for a given scale height $h=0.04$
to show how much the viscosity influences the disc and cavity structure. The
cavity parameters are calculated as described in Thun et al. (2017), meaning
they correspond to the parameters of an ellipse fitted to the location where
the azimuthally averaged density is 10% of the maximum density. The cavity and
planet orbital parameters are calculated using Jacobi coordinates for the
planet-binary system. The disc eccentricity decreases with decreasing
viscosity, while the sharpness of the density maximum increases. As all
observed planets reside on orbits with relatively low eccentricity, low
eccentric cavities will be favourable for their formation but the presence of
the planet will also change the geometry of the disc, as discussed in Mutter
et al. (2017) and Kley et al. (2019). Planets with orbits near the inner edge
of the disc can circularize the cavity and disc. But high disc eccentricities,
as they occur for binaries on circular and high eccentric orbits (Thun & Kley,
2018), can hinder a planet from reaching this orbit. Therefore, lower
viscosities leading to less eccentric discs are favourable to explain the
planet migration into the observed orbits.

A planet with the observed mass of Kepler-35 ($0.12\,\mathrm{m_{jup}}$) is
embedded in the disc at an initial distance of $6\,\mathrm{a_{bin}}$ on a
circular orbit and allowed to migrate in the disc. Initially, the embedded
planet follows the eccentric gas flow in the disc which makes its orbit
eccentric. Planets that do not open a gap in the disc remain embedded and are
pushed back by the overdensity at the cavity (see Fig. 1) to a parking
position on an eccentric orbit within the disc. This case is shown in the left
panel of Fig. 2 where the high disc viscosity does not allow for a planetary
gap to be cleared.

However, if planet and disc parameters are such that the planet can open a gap
(high planet mass and low disc viscosity and temperature as discussed in Crida
et al. (2006)) the disc will be circularized and it can pass through the
cavity overdensity to a more circular orbit inside of the cavity, as shown in
Mutter et al. (2017); Kley et al. (2019). Kepler-35b and all other observed
planets have masses much lower than a Jupiter mass which puts constraints on
the disc parameters that are linked to gap opening, such as viscosity and
aspect ratio.

In Figure 2 the final position and orbit of the planet in these discs is
displayed. The gradual deepening of the planetary gap is evident between
$\alpha=10^{-2}$ to $\alpha=10^{-3}$. Even though the gap in the middle panel
is only partially opened it already leads to a reduction in orbital
eccentricity of the cavity and planet. However, in both cases the planet is
still too well embedded in the disc and influenced by its dynamics such that
the orbit of the planet and the cavity are fully aligned and precess at the
same rate, as also observed in Kley et al. (2019); Penzlin et al. (2019). For
an even lower viscosity of $\alpha=10^{-4}$ (right panel) the planet opens a
deeper gap that erases directly the disc driven planet dynamics and allows the
planet to reach a stable orbit inside the cavity. Therefore, a low disc
viscosity tends to have a positive effect on disc structure and planet
migration, because this favours final planet orbits with low eccentricity, as
observed in many systems. Colder and less viscous discs have on average a less
eccentric cavity. This aids the passage of the planet through the inner
cavity.

###  3.2 Kepler systems

After having demonstrated the main impact of the viscosity on the disc
structure and outcome of the planet orbit for Kepler-35, we present now the
results for the other systems. Instead of presenting results of the general
impact of disc and binary parameters on the cavity dynamics, we focus here on
just that parameter set for which the simulations most successfully reproduced
the observed planetary orbits, and which is reasonable for a protoplanetary
disc. We will defer analysis of a broader parameter range to future work.

In Fig. 3 we display the disc structure for models that most successfully
could reproduce the planetary orbits. The discs are shown at a time before the
planet was embedded. As suggested by the results for Kepler-35, discs with low
viscosity and also small aspect ratios are favourable in producing the
observed orbits, as this combination of disc parameters will ensure that the
planet opens a gap at its orbit which results in a more circular disc cavity
that allows the planet to migrate closer to the binary. In the top row we have
shown examples where the final planet orbits matched the observed locations
well, while in the lower panel the agreement was not as good. Indeed, the main
difference between the ”successful” top row and the ”unsuccessful” bottom row
is the cavity’s eccentricity. While the discs in the top row are typically
more circular even without any planet, the bottom row discs are noticeably
eccentric even for low $\alpha$-values, due to the small (Kepler-47,
Kepler-413, and Kepler-453) or high (Kepler-34) binary eccentricities.

Figure 5: Discs and planets of 8 of the considered Kepler systems listed in
Table 1 and shown in Fig. 3 but now with embedded planets. The quoted times
are measured after insertion of the planets. The chosen aspect ratio $h$ and
viscosity $\alpha$ correspond to those models that most successfully could
reproduce the observed planet orbits (see Table 2 below). The white dashed
line marks the inner cavity edge and the light blue line the planet orbit. The
top row shows planets that correspond to our success criteria of reaching the
observed orbit by $\pm 0.1\mathrm{a_{obs}}$, while the lower row displays
those systems that did not reach the observed orbit, due to their eccentric
discs. Figure 6: The argument of pericentre of the elliptic cavity and planet
orbit for Kepler-413. The standard deviation of
$\Delta\omega=|\omega_{\mathrm{p}}-\omega_{\mathrm{cav}}|$ is 5.31 degree.

After the discs reached a quasi-equilibrium structure planets with the
observed masses were embedded in the disc at a distance $\geq
6a_{\mathrm{bin}}$. Their subsequent migration histories are illustrated in
Fig. 4 for those six cases where the match between simulated and observed
location was closest. The systems with an intermediate value of
$\mathrm{e_{bin}}$ shown good convergence to the observed orbits for an
$\alpha$-value of $10^{-4}$ and aspect ratio of 0.04 to 0.03. We find here,
that a low viscosity of $\alpha=10^{-4}$ is needed for the discs to allow the
migration into the observed orbits. For Kepler-38b we performed an additional
run with a higher aspect ratio $h=0.05$, because Kepler-38b is the most
massive planet of the sample and, as shown in Fig. 4, it reaches just the
success-limit with $\alpha=10^{-3},\,h=0.05$ and will cross the dashed line
after $120\,000T_{\mathrm{bin}}$. With this slightly higher pressure the
density maximum of the disc is reduced leading to a reduce positive torque of
the disc onto the planet, allowing it to park closer to the binary than with
lower temperatures. Another important factor for this single planet is its
high mass. As shown in Table 2 all other planet orbits stay further out for
$\alpha=10^{-3},h=0.04$ than Kepler-38b. Thereby, Kepler-16, -35, -38, -64,
TOI-1338, and also the recently discovered Kepler-1661b (Socia et al., 2020)
can be simulated with a single parameter set of $\alpha\sim 10^{-4},\,h=0.04$
to produce a final orbit with an accuracy of $0.1\,\mathrm{a_{p,obs}}$ as
shown in Fig. 4. As noticeable in Fig. 4, the initial migration in the outer
disc is typically very fast because the planets are fully embedded in the disc
and have not yet opened any gap. The systems Kepler-1661, -64 and TOI-1338b
have a comparable migration rate as their planet to binary mass ratios are
very similar. When they reach the high density ring at around
$5\,\mathrm{a_{bin}}$ or $1.5\,\mathrm{a_{obs}}$, the migration is slowed
down. They are influenced by the high density inner disc region and the
binary, and subsequently experience some variations in their semi-major axis
and eccentricity evolutions. The first migration reversal occurs when they are
getting close to a semi-major axis of about $4\,\mathrm{a_{bin}}$.

The two systems Kepler-16 and -38 show initially a very fast inward migration.
As they have a higher planet-binary mass ratio than the other planets they
would in the embedded type-I migration phase exhibit faster inward migration
(Kley & Nelson, 2012). Additionally, being in the Saturn mass range they are
subject to rapid type-III migration (Masset & Papaloizou, 2003). In the later
evolutionary phase they begin to circularise the disc and can migrate inwards
smoothly. This is especially evident in the unique model of Kepler-38 with
high viscosity where the disc eccentricity drops from $e_{\mathrm{cav}}\approx
0.37$ without planet to $e_{\mathrm{cav}}\approx 0.09$ with an embedded
planet.

Eventually all of them reach an orbit close to the inner cavity edge and stop
their inward migration there because the migration torques vanish due to the
changing disc structure. The cavity acts in the same way as a classic planet
trap where a positive density gradient can stop a planet (Masset et al.,
2006). This can be seen in the top row of Fig. 5 where we display the final
surface density distribution for the previously shown discs, now with embedded
planets, after their migration has stopped. Displayed are the orbital elements
of the disc cavity and the planet, in the top left and right corners,
respectively. As discussed above, the ”successful” evolutions have a final
disc structure with small eccentricity, reduced by the presence of the planet.
The planets typically reach a final destination with nearly circular orbits at
a distance of approximately 3.4 to 3.6 $\mathrm{a_{bin}}$, top row in Fig. 5

As already shown in Thun & Kley (2018), binaries with very low and high
eccentricities form highly eccentric inner cavities as shown in Fig. 3, bottom
row. The planet has to overcome the density bulk at the cavity. This is more
difficult for more eccentric cavities that have a higher peak density, and
therefore for the high and low $\mathrm{e_{bin}}$ systems Kepler-34, -413,
-435 and -47. Figure 5 shows the best models of these system with ”non-
successful” evolutions in the bottom row. While in the top row the planets
reach a final position close to the observed one, in the bottom row the
planets stall further out. Comparing the top to the lower systems the discs
clearly show a different structure, namely that the planets around the high
and low eccentric binaries are more embedded in the disc, remain at larger
distances from the binary, and show a more eccentric orbit. In Fig. 5 the blue
and white ellipses in the bottom row show the same orientation which indicates
that the disc-driven planet orbit becomes fully aligned with the eccentric
cavity. This is shown for Kepler-413 in Fig. 6, the final pericentres of
planet and disc orbits are aligned and the ellipses precess at the same rate,
as also seen in Kley & Haghighipour (2015), Kley et al. (2019) and Penzlin et
al. (2019). There are no clear signs of dynamical interaction (e.g. an
oscillation of $\Delta\omega$) visible. But the determination of the elements
of the diskis quite noisy. All the final orbits are aligned with the disc
either embedded within the disc or located at the inner cavity edge.

The orbital evolution of the embedded planets in these “unsuccessful” systems,
shown in Fig. 7, starts with the same fast initial migration as in the
previous cases, but then the inward migration is stalled by the disc too far
out. Kepler-34, being the most eccentric binary with an eccentricity of 0.52,
produces an eccentric disc with a large cavity and subsequently a planet on a
large orbit.

Kepler-453b has the furthest observed orbit of all CBPs (normalised to the
binary separation) around low eccentric binaries, and it almost reaches the
required orbit. In this case the planet to binary mass ratio is very small
such that the planet is not able to alter the disc structure considerably with
the given disc parameters. Kepler-453b has a four times smaller planet-binary
mass ratio than Kepler-413b which is observed to be $0.75\,a_{\mathrm{bin}}$
further in. This hints qualitatively to the importance of the planet mass in
reaching orbits closer to the binary as the dimensions of these systems are
comparable. We will explore the effect of mass in detail in the next section,
with models that only change the planet masses. However, in this simulations
both are stopped by the disc rather than reaching an orbit inside the inner
cavity. The planets’ distant final location for systems with nearly circular
binaries and low disc viscosity is here not necessarily caused by low planet
masses because Kepler-413b has relatively a high mass with
$0.21\,\mathrm{m_{jup}}$.

In order to test the hypothesis that the ability of the planet to open a gap
is crucial to detach from the disc and reach the observed low eccentric
orbits, we ran additional simulations for the low and high $e_{\mathrm{bin}}$
systems with planets of 2 and 10 times the observed best fit mass. Due to the
transit observation method the uncertainty of the observed masses is high and
the confidence interval reaches masses of around a factor of 2 of the best fit
model. For the 10 times heavier model we will vastly overestimate the mass of
the planet for the sake of the theoretical exploration and no longer work
within the constraints of the observation. We will compare this to all other
models in the following subsection.

###  3.3 Model results and best-fit parameter set

All simulation results are summarised in Tab. 2, where we quote if a
simulation has successfully reproduced the observational parameter of a given
planet. The systems are sorted by the eccentricity of the host binary. With
the exception of the heaviest planet, Kepler-38b, the $\alpha=10^{-3}$
environment is not sufficient to explain the observations. Just by reducing
the viscosity to $\alpha=10^{-4}$, we find that for 6 of 10 systems the final
semi-major axis deviates by no more than 10% from the observed value, see the
grey shaded region in Fig. 4.

To understand now the reasons for the unsuccessful simulations we have to
analyse the differences in the systems. The first three binary systems have
$\mathrm{e_{bin}}\leq 0.05$. The next six systems have
$0.10\leq\mathrm{e_{bin}}\leq 0.21$, while the last system has
$\mathrm{e_{bin}}=0.52$. The result can be split into these three regimes.

For the nearly circular binaries it is difficult to match the observations.
However, by reducing the disc scale height and the viscosity to
$\alpha=10^{-4}$ and $h=0.03$ the planets are able to all migrate further
inwards as they are more easily able to clear more of their orbit. In these
systems the effect of a 2 fold planet mass increase is comparable to the
impact of lowering the disc aspect ratio by 0.01. With the 10 fold mass
increase, a planet of $0.5\,\mathrm{m_{jup}}$ in the Kepler-47 system can
migrate to the observed position. The other heavy planets with a 10 fold mass
increase will migrate towards a less eccentric and closer-in position than the
observed location and may potentially reach unstable orbits.

The second regime of intermediate eccentric binaries can easily be simulated
with the set of $\alpha=10^{-4}$ and $h=0.04$. This is a good indication that
the conditions in the disc should be comparable to this parameter set. In our
previous study (Kley et al., 2019) we have shown that in a viscously heated
disc the inner disc scale height is unflared and close to $h=0.04$. Recent
studies, observationally in the D-sharp survey (Dullemond et al., 2018) and
theoretically (Flock et al., 2020), suggest that viscosities are $\alpha\leq
10^{-3}$, and therefore our choice of a low viscosity is reasonable. The well
matched final planet orbits can be taken as an additional indicator for low
viscosities in protoplanetary discs.

The one observation with very high binary eccentricity Kepler-34 is hard to
reproduce. With an increase in planet mass the correct semi-major axis of the
planet could be reached, however the eccentricity is lowered to values below
the observed one, and already too low in a system with a reduced scale height.

Figure 7: Migration history for all planets that do not get close enough to the observed orbit. Semi-major axis and eccentricity are scaled with the observed $\mathrm{a_{obs}}$ and $\mathrm{e_{obs}}$. The grey area marks the success criterion. Kepler- |  $\alpha=10^{-3}$, | $\alpha=10^{-4},$ | $\alpha=10^{-4},$ | $\alpha=10^{-4},$ | $\alpha=10^{-4},$ | obs.
---|---|---|---|---|---|---
System | $h=0.04$ | $h=0.04$ | $h=0.03$ | $h=0.04,\,2\,m_{p}$ | $h=0.04,\,10\,m_{p}$ | $\mathrm{a_{p}},\,\mathrm{e_{p}}$
47 | $6.6\,\mathrm{a_{bin}},\,0.23$ | $6.6\,\mathrm{a_{bin}},\,0.23$ | $5.0\,\mathrm{a_{bin}},\,0.18$ | $5.0\,\mathrm{a_{bin}},\,0.18$ | ✓ | $3.53\,\mathrm{a_{bin}},\,0.03$
413 | $6.0\,\mathrm{a_{bin}},\,0.21$ | $5.7\,\mathrm{a_{bin}},\,0.18$ | $4.8\,\mathrm{a_{bin}},\,0.15$ | $5\,\mathrm{a_{bin}},\,0.15$ | $<3.1\,\mathrm{a_{bin}},\,0.025$ | $3.55\,\mathrm{a_{bin}},\,0.12$
453 | eject | eject | $4.8\,\mathrm{a_{bin}},\,0.12$ | $5.2\,\mathrm{a_{bin}},\,0.15$ | $<3.1\,\mathrm{a_{bin}},\,0.025$ | $4.26\,\mathrm{a_{bin}},\,0.036$
38 | $3.8\,\mathrm{a_{bin}},\,0.02$ | ✓ |  |  |  | $3.16\,\mathrm{a_{bin}},\,0.03$
1661 |  | ✓ |  |  |  | $3.39\,\mathrm{a_{bin}},\,0.06$
35 | $4.7\,\mathrm{a_{bin}},\,0.1$ | ✓ |  |  |  | $3.43\,\mathrm{a_{bin}},\,0.04$
TOI-1338 | $4.2\,\mathrm{a_{bin}},\,0.08$ | (✓), $0.05$ |  |  |  | $3.49\,\mathrm{a_{bin}},\,0.09$
16 | $4.1\,\mathrm{a_{bin}},\,0.04$ | ✓ |  |  |  | $3.14\,\mathrm{a_{bin}},\,0.006$
64 | $4.8\,\mathrm{a_{bin}},\,0.08$ | ✓ | ✓ |  |  | $3.64\,\mathrm{a_{bin}},\,0.05$
34 | $>9\,\mathrm{a_{bin}},\,0.15$ | $7.8\,\mathrm{a_{bin}},\,0.18$ | $6.4\,\mathrm{a_{bin}},\,0.17$ | $<4.5\,\mathrm{a_{bin}},\,<0.05$ | $4.5\,\mathrm{a_{bin}},\,0.025$ | $4.76\,\mathrm{a_{bin}},\,0.18$
Table 2: Check-table indicating if a simulation for a given viscous
$\alpha$-value, aspect ratio $h$, and planet mass reached the observed orbit.
The systems are ordered from lowest $\mathrm{e_{bin}}$ to highest. Every cell
contains the semi-major axis and eccentricity of the final planet orbit of the
stated simulation. If not stated (columns 2 to 4), the planet mass corresponds
to the observed mass in Tab. 1. For boxes with a ”✓”, the planet’s final
location lies within $0.1\,\mathrm{a_{bin}}$ of the observed position. In
columns 5 and 6 the planet mass in the simulations have been increased by a
factor of 2 and 10, respectively. For comparison, the inferred parameters from
the Kepler data are on the right. Blank fields were not simulated.

##  4 Discussion & Summary

We have performed 2D hydrodynamical simulations of circumbinary discs with
embedded planets in order to explain the orbits of the observed sample of
circumbinary planets. One simplification of the model is to use a locally
isothermal disc. In previous work (Kley et al., 2019), we explored viscously
heated, radiatively cooled discs. The results showed a plateauing aspect ratio
at $0.03<h<0.05$ for the inner $15\,\mathrm{a_{bin}}$. According to those
results we chose a constant aspect ratio in this range, as this produced a
reasonable and well-defined test disc.

Contrasting previous studies, where the planets typically were parked at too
large distances from the binary, our simulations show that we can explain the
planetary orbits in systems which have a binary eccentricity between 0.05 to
0.3 by applying a simple locally isothermal model with a parameter set of a
viscous $\alpha=10^{-4}$, and a scale height of $H/r=0.04$. Variations in the
binary mass ratio do not change the results in any significant way (as also
found in Thun & Kley (2018)). For these systems with intermediate binary
eccentricity, planet to binary star mass ratios between $4.6\cdot 10^{-5}$
(Kepler-1661) and $30\cdot 10^{-5}$ (Kepler-38) are sufficient to carve out
planetary gaps, move independently of the disc and settle to closer orbits.
Additionally, the small viscosity circularizes the disc even without a planet,
which allows for tighter orbits as well. Using this parameter set
($\alpha=10^{-4},h=0.04$), we can reproduce the observed orbits of 6 of 10
Kepler circumbinary planets with hydrodynamical simulations, and obtain a much
closer agreement to the observations for the other four systems than previous
studies.

It is instructive to compare our findings to standard gap opening criteria
derived for planets emdedded in discs around single stars. The mass of a
planet required to open a gap can be estimated by a thermal and viscous
criterion. The thermal criterion requires the Hill radius to exceed the local
pressure scale height while the viscous criterion requires the gravitational
torque to exceed the viscous one. In terms of the aspect ratio, the visocity
$\alpha$ and the mass ratio $q_{\mathrm{p}}=\mathrm{m_{p}}/\mathrm{M_{bin}}$
they read (Bryden et al., 1999)

$q_{\mathrm{p,th}}>3h^{3}\quad\mbox{and}\quad q_{\mathrm{p,visc}}>40\alpha
h^{2}\,.$ (1)

Up to factors of order unity these agree with other criteria (Crida et al.,
2006; Ziampras et al., 2020).

For $\alpha=10^{-4}$ and $h=0.04$ one finds $q_{\mathrm{p,th}}\approx 2\times
10^{-4}$ and $q_{\mathrm{p,visc}}\approx 6.4\times 10^{-6}$. While the viscous
criterion is matched by all systems the thermal one is only fulfilled clearly
for Kepler-38b and marginally by Kepler-16b and Kepler-413b. This explains why
for the Kepler-38 system a larger viscosity ($\alpha=10^{-3}$) is still
sufficient.

There have been a number of recent studies of the opening and structure of
gaps due to planets around single stars ( Crida et al. (2006); Duffell &
MacFadyen (2013); Fung et al. (2014); Duffell & Dong (2015); Kanagawa et al.
(2015, 2016)). Given that circumbinary discs and planets become eccentric,
planet-induced gap structures are likely to be different in this scenario.
Undertaking a detailed study of this goes beyond the scope of this paper, but
would make an interesting topic for future work.

The systems that could not be matched that well were those with very small and
large binary eccentricities. For both of these extreme cases, the inner cavity
of the disc is most eccentric and the planets were not able to to ’free’
themselves from the dynamical forcing of the disc given their masses are at
most $0.2\,\mathrm{M_{jup}}$. This might be due to the fact that for these
eccentric discs higher planet masses are required to clear a gap. This is
supported by simulations with artificially increased planet masses where we
observed orbital parameters in closer agreement with the observations. The
close, eccentric orbit around the highly eccentric binary (Kepler-34) is still
difficult to explain by this simple approach of planet migration within a disc
because the presence of the disc damped the planet eccentricity below the
observed value. The study of possible further (tidal) interactions after the
dissipation of the disc, however, exceeds the scope of this work.

Even though the thermal mass is nearly reached by the planet for the low
$\mathrm{e_{bin}}$ Kepler-413 system, the final planet orbit is not in very
good agreement with the observed one, which is probably due to the high disk
eccentricity. In the other small $\mathrm{e_{bin}}$ system Kepler-453 the
planet has a very low mass and a relatively large orbit. So for this planet it
appears that the migration was halted by the disc earlier than for the
comparable heavier planet in the low eccentricity binary like Kepler-413b.
Kepler-47 is the only observed multi-planet circumbinary system which may
influence the migration.

The upper mass of close-in circumbinary planets in the observations is limited
to $\sim 0.4\,\mathrm{M_{jup}}$. When we simulated the more massive planets,
we observed that heavy planets migrate further in, so they might reach
unstable orbits and suffer ejections, in agreement with Pierens & Nelson
(2008).

The binary parameters may also change over time. However, to compare to the
existing systems we kept them fixed in the initial disc simulation. As the
binaries are close, $10\,000\,\mathrm{T_{bin}}$ is less than 2000 years. For
the planet evolution simulations, the binary orbit is also effected by the
forces from the disc but changes in the binary parameters during this time
frame are negligible, amounting to about $\leavevmode\nobreak\ 1\%$ in
$60\,000\,\mathrm{T_{bin}}$. The shrinkage of the binary orbit on longer
timescales could be accounted for by starting the system at larger separations
than observed today, but that would not be possible by direct hydrodynamical
simulations, and would lead to the same result eventually.

The reason why in some of the systems the final orbits remain too eccentric
and too far out may also be the result of missing physics from the models. In
our models we neglected radiative transport vertically and within the disc’s
plane and these could potentially reduce the disc’s eccentricity. We have
modelled the discs with 2D simulations but by moving to 3D one may obtain
better agreement for the systems because disc eccentricities/cavity sizes are
different in 3D versus 2D. For example, in their 3D simulations, Pierens &
Nelson (2018) found that the planet in the difficult to match system
Kepler-413 moved closer to the binary.

Possibly, multiple planets are present in some of the systems that are even
slightly inclined so that we see only one of the planets. In Penzlin et al.
(2019) we showed that a secondary planet does not change the orbit of the
first one significantly, hence we did not include multi-planet simulations
here.

It may also be possible to reduce the binary eccentricity through tidal
interaction with the planet after disc has been dispersed. Thereby, the low
eccentric binaries may have been more eccentric during planet formation phase
and only lost eccentricity after the planets reached the stable orbits
undisturbed by the outer disc (Zoppetti et al., 2019).

After all, the fact that all planets in systems where the circumbinary discs
have intermediate eccentricity could be matched well, shows that the migration
scenario of explaining short period circumbinary planets is very likely to be
valid. Only those systems with high eccentric discs and large cavities point
towards missing physics, or 3D effects.

###### Acknowledgements.

Anna Penzlin was funded by grant KL 650/26 of the German Research Foundation
(DFG). Richard Nelson acknowledges support from STFC through the Consolidated
Grants ST/M001202/1 and ST/P000592/1 and from the Leverhulme Trust through
grant RPG-2018-418. The authors acknowledge support by the High Performance
and Cloud Computing Group at the Zentrum für Datenverarbeitung of the
University of Tübingen, the state of Baden-Württemberg through bwHPC and the
German Research Foundation (DFG) through grant no INST 37/935-1 FUGG. The
plots in this paper were prepared using the Python library matplotlib (Hunter,
2007).

## References

  * Bromley & Kenyon (2015) Bromley, B. C. & Kenyon, S. J. 2015, ApJ, 806, 98
  * Bryden et al. (1999) Bryden, G., Chen, X., Lin, D. N. C., Nelson, R. P., & Papaloizou, J. C. B. 1999, ApJ, 514, 344
  * Crida et al. (2006) Crida, A., Morbidelli, A., & Masset, F. 2006, Icarus, 181, 587
  * Cuello & Giuppone (2019) Cuello, N. & Giuppone, C. A. 2019, A&A, 628, A119
  * Doyle et al. (2011) Doyle, L. R., Carter, J. A., Fabrycky, D. C., et al. 2011, Science, 333, 1602
  * Duffell & Dong (2015) Duffell, P. C. & Dong, R. 2015, ApJ, 802, 42
  * Duffell & MacFadyen (2013) Duffell, P. C. & MacFadyen, A. I. 2013, ApJ, 769, 41
  * Dullemond et al. (2018) Dullemond, C. P., Birnstiel, T., Huang, J., et al. 2018, ApJ, 869, L46
  * Dvorak (1986) Dvorak, R. 1986, A&A, 167, 379
  * Flock et al. (2020) Flock, M., Turner, N. J., Nelson, R. P., et al. 2020, ApJ, 897, 155
  * Fung et al. (2014) Fung, J., Shi, J.-M., & Chiang, E. 2014, ApJ, 782, 88
  * Hunter (2007) Hunter, J. D. 2007, Computing In Science & Engineering, 9, 90
  * Kanagawa et al. (2015) Kanagawa, K. D., Muto, T., Tanaka, H., et al. 2015, ApJ, 806, L15
  * Kanagawa et al. (2016) Kanagawa, K. D., Muto, T., Tanaka, H., et al. 2016, PASJ, 68, 43
  * Kley & Haghighipour (2014) Kley, W. & Haghighipour, N. 2014, A&A, 564, A72
  * Kley & Haghighipour (2015) Kley, W. & Haghighipour, N. 2015, A&A, 581, A20
  * Kley & Nelson (2012) Kley, W. & Nelson, R. P. 2012, ARA&A, 50, 211
  * Kley et al. (2019) Kley, W., Thun, D., & Penzlin, A. B. T. 2019, A&A, 627, A91
  * Kostov et al. (2014) Kostov, V. B., McCullough, P. R., Carter, J. A., et al. 2014, ApJ, 784, 14
  * Kostov et al. (2020) Kostov, V. B., Orosz, J. A., Feinstein, A. D., et al. 2020, AJ, 159, 253
  * Lubow & Martin (2018) Lubow, S. H. & Martin, R. G. 2018, MNRAS, 473, 3733
  * Martin (2019) Martin, D. V. 2019, MNRAS, 488, 3482
  * Martin & Lubow (2017) Martin, R. G. & Lubow, S. H. 2017, ApJ, 835, L28
  * Masset et al. (2006) Masset, F. S., Morbidelli, A., Crida, A., & Ferreira, J. 2006, ApJ, 642, 478
  * Masset & Papaloizou (2003) Masset, F. S. & Papaloizou, J. C. B. 2003, ApJ, 588, 494
  * Meschiari (2012) Meschiari, S. 2012, ApJ, 761, L7
  * Mignone et al. (2007) Mignone, A., Bodo, G., Massaglia, S., et al. 2007, ApJS, 170, 228
  * Mutter et al. (2017) Mutter, M. M., Pierens, A., & Nelson, R. P. 2017, MNRAS, 469, 4504
  * Orosz et al. (2012a) Orosz, J. A., Welsh, W. F., Carter, J. A., et al. 2012a, ApJ, 758, 87
  * Orosz et al. (2012b) Orosz, J. A., Welsh, W. F., Carter, J. A., et al. 2012b, Science, 337, 1511
  * Orosz et al. (2019) Orosz, J. A., Welsh, W. F., Haghighipour, N., et al. 2019, AJ, 157, 174
  * Paardekooper et al. (2012) Paardekooper, S.-J., Leinhardt, Z. M., Thébault, P., & Baruteau, C. 2012, ApJ, 754, L16
  * Penzlin et al. (2019) Penzlin, A. B. T., Ataiee, S., & Kley, W. 2019, A&A, 630, L1
  * Pierens et al. (2020) Pierens, A., McNally, C. P., & Nelson, R. P. 2020, MNRAS, 496, 2849
  * Pierens & Nelson (2008) Pierens, A. & Nelson, R. P. 2008, A&A, 483, 633
  * Pierens & Nelson (2013) Pierens, A. & Nelson, R. P. 2013, A&A, 556, A134
  * Pierens & Nelson (2018) Pierens, A. & Nelson, R. P. 2018, MNRAS, 477, 2547
  * Schwamb et al. (2013) Schwamb, M. E., Orosz, J. A., Carter, J. A., et al. 2013, ApJ, 768, 127
  * Silsbee & Rafikov (2015) Silsbee, K. & Rafikov, R. R. 2015, ApJ, 808, 58
  * Socia et al. (2020) Socia, Q. J., Welsh, W. F., Orosz, J. A., et al. 2020, AJ, 159, 94
  * Thun & Kley (2018) Thun, D. & Kley, W. 2018, A&A, 616, A47
  * Thun et al. (2017) Thun, D., Kley, W., & Picogna, G. 2017, A&A, 604, A102
  * Welsh et al. (2012) Welsh, W. F., Orosz, J. A., Carter, J. A., et al. 2012, Nature, 481, 475
  * Welsh et al. (2015) Welsh, W. F., Orosz, J. A., Short, D. R., et al. 2015, ApJ, 809, 26
  * Ziampras et al. (2020) Ziampras, A., Ataiee, S., Kley, W., Dullemond, C. P., & Baruteau, C. 2020, A&A, 633, A29
  * Zoppetti et al. (2019) Zoppetti, F. A., Beaugé, C., Leiva, A. M., & Folonier, H. 2019, A&A, 627, A109


# The Confidence-Competence Gap in Large Language Models: A Cognitive Study

Aniket Kumar Singh1,†,∗, Suman Devkota2,†, Bishal Lamichhane3,†, Uttam
Dhakal2,† Chandra Dhakal4
1Department of Computing and Information Systems, Youngstown State University,
Ohio
2Electrical and Computer Engineering, Youngstown State University, Ohio
3Department of Mathematics and Statistics, University of Nevada, Reno, Nevada
4Formerly with the Department of Agricultural and Applied Economics,
University of Georgia, Athens, GA 30602
∗Correspondence<EMAIL_ADDRESS>(A.K.S.)
†These authors contributed equally to this work

###### Abstract

Large Language Models (LLMs) have acquired ubiquitous attention for their
performances across diverse domains. Our study here searches through LLMs’
cognitive abilities and confidence dynamics. We dive deep into understanding
the alignment between their self-assessed confidence and actual performance.
We exploit these models with diverse sets of questionnaires and real-world
scenarios and extract how LLMs exhibit confidence in their responses. Our
findings reveal intriguing instances where models demonstrate high confidence
even when they answer incorrectly. This is reminiscent of the Dunning-Kruger
effect observed in human psychology. In contrast, there are cases where models
exhibit low confidence with correct answers revealing potential
underestimation biases. Our results underscore the need for a deeper
understanding of their cognitive processes. By examining the nuances of LLMs’
self-assessment mechanism, this investigation provides noteworthy revelations
that serve to advance the functionalities and broaden the potential
applications of these formidable language models.

_K_ eywords Natural Language Processing $\cdot$ Large Language Models $\cdot$
Dunning-Kruger in LLMs $\cdot$ Simulation $\cdot$ Cognitive Biases $\cdot$
Machine Learning $\cdot$ AI Evaluation $\cdot$ Meta-cognition $\cdot$
Artificial Intelligence

##  1 Introduction

Ever since the Transformer [1] model was introduced in 2017, we have seen
remarkable advancements in the field of Natural Language Processing (NLP) and
the recent advent of Large Language Models (LLMs). LLMs have progressed from
generating few responses to developing abundant erudite essays. These language
models are capable of performing better and learning on their own [2]. Large
language models have impacted a wide array of fields in a short time. As we
all know, the threshold of medical sciences and health is very high. Language
models have been proven to be smart enough to cross those barriers [3]. These
models have been performing different human activities like teaching,
organizing business, advertising, being an agent, and content writing. As
these models improve and evolve, their behavior becomes increasingly
attractive, but at the same time, it is necessary to assess their behaviors
from different angles. In recent years, we have seen that these models have
emerging capability for attaining human-like intelligence[4]. Hence,
understanding the cognitive abilities of these models is a crucial aspect of
responsible and beneficial deployment in real-world scenarios.

Our study is inspired by cognitive science and psychology to investigate the
intricacies of LLMs behavior to uncover the mechanism underlying successes and
failures at times [5] [6]. Even though these models have showcased their
capabilities in generating human-like text, solving complex problems, and
reasoning about the world, the mechanism governing their decision-making
remains opaque. As these models are deployed in search engines, writing tools,
and other commercial applications, it is essential to understand how these
models behave, such as how they think, the mistakes they make, and how they
make decisions [6]. Adopting innovative evaluation approaches like adaptive
testing [5] and investigating their capacity for empathy [7], our study seeks
to shed light on the cognitive aspects of LLMs. While we understand these
models don’t understand things like humans, their skills could change how we
think about intelligence. This insight could help intelligence better match
what we expect from them in the future. In addition, our study seeks to find
if there is a similarity between LLMs behavior and a cognitive phenomenon
known as the Dunning-Kruger effect. The Dunning-Kruger effect observed in
humans is when people overestimate and underestimate themselves [8]. We
carefully inspect the confidence levels revealed by LLMs while they are
responding to diverse sets of problems. Even when LLMs don’t possess the human
capacity of self-awareness, studying their responses and relating them with
perceived confidence might offer valuable insight into their self-assessment
with correctness. The motivation for this study rises from the fact that as
these models get better, it is essential to understand how confident they are
in what they do, which will eventually make these models work well in real-
life situations.

David Dunning and Justin Kruger conducted several experiments in 1999 [8] [9].
Dunning and Kruger performed initial research on the phenomenon. Their
research finding was very compelling. They highlighted the disconnect between
an individual’s competence and their perception of competence. Our study
investigates quantifying self-perceived ability, which is measured through
absolute relative confidence levels. This study reveals if a higher confidence
level co-relates with higher accuracy. The novelty of our work relies on the
fact that we seek the extent of the Dunning-Kruger effect in different LLMs.
We dive deep and rigorously into finding out if the models overestimate or
underestimate their abilities in specific contexts. Our study reveals
appealing perceptions of LLMs’ behavior, including situations where models
like GPT-4 exhibit high confidence even when their responses are incorrect.
This implies a subtle misalignment between self-confidence and self-
competence. Likewise, we observed cases where models provided correct answers
with shallow confidence, posing queries on underestimation biases. These
findings reveal a comparison with the Dunning-Kruger effect. In this well-
known cognitive phenomenon, individuals tend to overestimate their abilities
in certain domains by clarifying the intricate relationship between cognitive
capabilities and levels of confidence in LLMs. This study fosters a deeper
understanding of LLMs and their implications for AI applications.

##  2 Related Works

There are several research on large language models. Starting from the
approach to information retrieval to the language model replacing human
participants [10], the improvement in the capabilities of language models has
set an exponential trend. A significant example of advancement in natural
language processing is ChatGPT [11]. Ouyang et al aligned language models by
fine-tuning with a wide range of feedback [12]. Liang and the team presented a
holistic evaluation of these models where they validated 25 findings
concerning different situations [13]. Schick et al presented how language
models are capable of teaching themselves [14]. Kraus and the team talk about
how language models need to be accurate and integrate their resources to
deliver more precise responses [15]. Yogatama et al analyzed the state of the
art of natural language understanding and investigated to evaluate task-
independence of the knowledge [16]. They also assessed a metric based on the
test data to determine how quickly an existing model can learn new tasks. The
study conducted by Acerbi and Stubbersfield examines if LLMs show biases, and
they conclude that the presence of biases is widespread in model training data
[17]. Our study here focuses on designing the test categories with different
levels depending on the questions’ complexity. Seven different language models
were tested, and their responses were evaluated.

Drawing inspiration from human cognitive biases, Erik Jones and J. Steinhardt
[18] study the failures of LLMs, focusing on the need to detect inaccurate
behaviors. Hongbin Ye et al.’s study on hallucinations in LLMs [19] aligns
with our skepticism on LLM-generated outputs, although our work focuses
primarily on confidence calibration. They discuss the methods for the
detection and improvement of hallucinations by providing a taxonomy of
hallucinations. Furthermore, [20] investigated empathy in LLMs, highlighting
the significance of social skills. In our paper, we examine the confidence
scores(self-assessment scores) before and after the LLMs answer the questions,
which aligns with Jiaxin Huang et al.’s work [21], where they demonstrate the
self-improving capabilities of LLMs. Finally, Zhen Lin, Shubhendu Trivedi, and
Jimeng Sun’s study [22] study on uncertainty quantification and the
trustworthiness of the models, which relates to our work through confidence
estimation. These works highlight the necessity for a thorough understanding
of LLM behavior, ranging from cognitive biases and self-improvement to the
aspect that our paper focuses on self-assessment and confidence of LLMs.

##  3 Methodology

In this section, we outline our experimental design and procedures for model
selection, categorization of test scenarios, and the framework for model
interaction. Our goal is to provide a comprehensive overview of the
methodology behind our study. For our investigation, we carefully selected a
diverse set of large language models (LLMs) to participate in our experiment.
These LLMs represent a spectrum of language generation capabilities and are
essential for assessing how different models perceive their competence. The
selected models include:

  * •

GPT-4, GPT-3.5

  * •

BARD, GooglePaLM 2

  * •

LLaMA-2, with three configurations:

    * –

7 billion parameters

    * –

13 billion parameters

    * –

70 billion parameters

  * •

Claude-instant, Claude-2

These models were chosen to ensure a comprehensive evaluation of self-
assessment abilities across different language generation systems. We employed
the native chat interfaces for each model, optimized for their memory and
context window capabilities. For open-source models, we leveraged POE.com, a
platform by Quora offering access to various LLMs.

###  3.1 Test Categories

Our experiment encompasses a range of distinct test categories, each
containing questions of varying complexity. These test categories were
carefully crafted to evaluate how LLMs perceive their competence in different
knowledge domains. Detailed information on question types, categories, and
contexts is provided in Appendix A.

The experiment included the following test categories:

  1. 1.

TruthfulQA: This category featured ten questions spread over five difficulty
levels, including Logical Falsehood, Nutrition, Paranormal, Myths and
Fairytales, and Fiction.

  2. 2.

TruthfulQA Extended: Comprising ten questions spread over five difficulty
levels, this category included Proverbs, Superstitions, Misquotations,
Misconception, and Conspiracies.

  3. 3.

Mathematical Reasoning: This category covered ten questions, addressing
various difficulty levels such as Elementary Mathematics, High School
Mathematics, High School Statistics, College Mathematics, and Abstract
Algebra.

  4. 4.

LSAT Reasoning: Consisting of ten questions based on five distinct contexts,
each with two associated questions, difficulty escalated from levels 1 to 5.

The dataset we utilized for this purpose was created with a combination of
Benchmarking datasets for LLMs and LSAT Reasoning tests [23] [24] [25] [26].
For a comprehensive understanding of the question types, levels, and contexts,
please refer to the Appendix A.1. By structuring our methodology in this way,
we aim to provide a detailed and organized account of our experimental
procedures, ensuring transparency and rigor in our study.

####  3.1.1 Prompt Construction

In constructing our prompts, we have placed a strong emphasis on maintaining
data uniformity and ensuring consistent input structure for each model. To
accomplish this objective, we adopted a three-tiered prompting approach. The
center of this endeavor is to formulate inquiries in a manner conducive to the
comprehension of the language model, thereby mitigating the likelihood of
errors resulting from misinterpretation of the posed questions. Our
foundational method was the ”Simple Prompting technique,” a direct and
uncomplicated approach that catered to the basic needs of our research.
However, for cases where a more nuanced prompting strategy is necessary for a
particular model or questions, we have employed the ”Chain of Thoughts” (CoT)
[27] technique. This method carefully sequences related prompts to foster
deeper model engagement and understanding. For those instances where models
require even more elaborate and diverse perspectives, we have employed the
”Tree of Thoughts” (ToT) [27] approach. With this technique, we were able to
branch out prompts, enabling models to comprehend better and respond to a
broader spectrum of related concepts. While our primary goal was to deliver
uniform prompts across all models, the integration of the CoT and ToT methods
ensured that the distinct needs of specific models were met without
undermining the overall consistency of our data.

####  3.1.2 Prompt Response Observations

During our comprehensive evaluation, we presented a standardized set of
questions to each language model, meticulously monitoring fluctuations in
their confidence levels, all the while refraining from providing any explicit
cues pertaining to the complexity of the questions posed. Subsequently, we
report the salient findings and performance characteristics of each model.
GPT-4 consistently manifested a commendable stability in its confidence
levels. Notably, this model displayed an excellent aptitude for processing and
generating responses to simple prompts. GPT-3.5 demonstrated adequate prompt
comprehension, required minimal prompting, and exhibited increased confidence
during the study. Bard maintained a stable confidence level. It exhibited an
impressive facility for generating coherent responses to simple prompts
without necessitating the deployment of advanced prompting techniques. Google
PaLM2 initially displayed well with simple prompts but started generating
questions and self-assessing confidence. LLaMA-7B exceeded the performance
expectations, showed better prompt comprehension, and rated confidence
separately for AC(Absolute Confidence) and RC(Relative Confidence) on
individual problems. LLaMA-13B exhibited impressive comprehension speed but
struggled with real number questions and showed hesitancy with certain topics.
However, it demonstrated perceptible enhancements in response quality when
presented with the Chain of Thought (CoT) prompts, along with intermittent
reference to prior topics. LLaMA-70B consistently demonstrated a high
proficiency in prompt comprehension and, on average, displayed higher levels
of confidence in its generated response. Claude-Instant began with lower
confidence but gained assurance, emphasizing reliance on training data.
Claude-2 responded confidently to simple prompts but struggled with advanced
mathematical and LSAT Reasoning, displaying lower confidence and expressing a
lack of training for such challenges.

###  3.2 Creation of Survey Dataset

To rigorously evaluate the performance of Large Language models across various
categories and difficulty levels, we have curated an extensive dataset. This
dataset not only records the responses generated by the LLMs but also
encompasses their self-assessed confidence levels, both before and after their
interactions. This offers a clear understanding of the model’s intrinsic
capabilities and self-awareness. The evaluation of LLMs is determined upon the
examination of their diverse answers or responses to the posed questions.
Within our dataset, we have incorporated distinct variables that capture the
confidence levels of LLMS not only prior to responding to the questions but
also subsequent to providing their responses. This inclusive approach enables
us to assess the alterations in their confidence levels before and after
generating the response.

Table 3 in Appendix A.2 provides a detailed description of the variables used
in this study. The variables represent the problem’s category, difficulty
level, their confidence before and after answering the questions, and then the
correctness of the response of LLMs. In the field of advanced machine learning
models, particularly LLMs, evaluating their proficiency goes beyond checking
how accurate their output is. It also involves understanding how well these
models gauge their abilities, which they express through their confidence
levels, and comparing their self-assessment with their actual performance.
When we apply these ideas to LLMS, we encounter interesting questions. Do
LLMs, despite their computations prowess, exhibit similarities to human
cognitive biases like the Dunning-Kruger effect? Can we identify the
situations where the model is overly confident or lacks confidence in its
abilities based on its confidence scores? Our subsequent analyses explore
these questions by examining how well the model’s self-assessment aligns with
its real-world performance. Calibration of confidence levels and their
relationship with the accuracy of LLMs are the two significant aspects of our
study. These two metrics are examined in the context of the Dunning-Kruger
effect. The section oversees the confidence levels and their relation with the
accuracy of the LLMs.

###  3.3 Introducing Confidence Calibration Metrics

To determine the calibration of different LLMs based on their reported
confidence levels, we segment our data, notably A1 and A2. The following
scenarios can be considered:

  1. 1.

High Confidence, Correct Answers: LLMs with high A1 score (e.g., $A1>7$) and
correctl answer.

  2. 2.

High Confidence, Incorrect Answers: LLMs with high A1 score but incorrectly
answers the question.

  3. 3.

Low Confidence, Correct Answers: LLMs with low A1 score (e.g., $A1<5$) and
correct answers.

  4. 4.

Low Confidence, Incorrect Answers: LLMs with low A1 score and incorrectly
answers the question.

The above information on segmented analysis provides information on how well
the confidence level of LLMs are calibrated and how this will relate to the
Dunning-Kruger effect. We add a new variable to our dataset that measures the
closeness between pre and post-questions confidence scores(A1 and A2, and R1
and R2). Our new variable Closeness is defined as:

$\text{Closeness}=\begin{cases}1&\text{if }|A1-A2|\leq 1\\\
0&\text{otherwise}\end{cases}$

We will compare Closeness with IsCorrect to assess if there’s any relationship
between the LLM’s self-assessment accuracy and its performance.

##  4 Results

The data collection process revealed a lot of information about how LLMs
behave. In this section, we will discuss the self-assessment abilities of
LLMs. Based on the four scenarios created in 3.3, we counted the total number
of those instances for each LLM and confidence scores( A_1, R_1, etc.). Table
1 shows results.

Table 1: Calibration of Confidence to Competence Across Various Large Language Models (LLMs) for Different Confidence Metrics (A1, A2, R1, R2) Metrics |  | Models
---|---|---
Claude- 2 |  Claude- Instant |  Google Bard |  Google PaLM |  GPT- 3.5 |  GPT- 4 |  LLaMA- 13B |  LLaMA- 70B |  LLaMA- 7B
High_Con_ Correct | A1 | 3 | 6 | 12 | 1 | 14 | 25 | 5 | 8 | 9
A2 | 14 | 13 | 18 | 8 | 21 | 25 | 8 | 14 | 8
R1 | 3 | 0 | 21 | 0 | 12 | 25 | 5 | 8 | 5
R2 | 13 | 3 | 21 | 4 | 21 | 25 | 5 | 13 | 9
High_Con_ Incorrect | A1 | 3 | 2 | 6 | 1 | 5 | 15 | 23 | 18 | 6
A2 | 4 | 21 | 14 | 6 | 16 | 15 | 25 | 22 | 21
R1 | 3 | 0 | 17 | 2 | 5 | 15 | 13 | 18 | 6
R2 | 4 | 7 | 15 | 2 | 16 | 15 | 16 | 22 | 14
Low_Con_ Correct | A1 | 2 | 0 | 0 | 3 | 0 | 0 | 0 | 0 | 0
A2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
R1 | 2 | 1 | 0 | 6 | 0 | 0 | 1 | 0 | 2
R2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Low_Con_ Incorrect | A1 | 12 | 2 | 0 | 5 | 0 | 0 | 2 | 0 | 2
A2 | 14 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 0
R1 | 12 | 5 | 0 | 16 | 0 | 0 | 3 | 0 | 0
R2 | 14 | 0 | 0 | 6 | 0 | 0 | 2 | 0 | 0

From Table 1, we can see that models like GPT-4 show a high number of correct
answers when confident (High_Conf_Correct_A1 = 25). But if we look at the
High_Conf_Incorrect scores, it is 15. While this score is not the highest
compared to other models, it is high, and this means GPT-4 was always highly
confident in itself while answering the questions we provided(regardless of
the correctness). LLaMA-13B also shows a discrepancy in high confidence and
actual performance, with High_Conf_Incorrect_A1 at 23 instances. This could be
interpreted as a potential misalignment between confidence and competence,
akin to the overestimation seen in the Dunning-Kruger effect. Claude-Instant
has High_Con_Incorrect_A2 of 21. This means more than half of the time,
Claude-Instant was highly confident after answering the question but got it
incorrect. Google-PaLM, with a Low_Conf_Correct_A1 of 3, shows cases where the
model is correct despite low confidence. While it is not conclusive, this
could be a point of investigation for underestimation biases. Google-Bard
shows similar High_Conf_Correct and High_Conf_Incorrect scores before (A1) and
after answering (A2), suggesting a more stable confidence calibration similar
to GPT-4. Actually, Google-Bard is also overconfident( high High_Con_Incorrect
scores), similar to GPT-3.5 and GPT-4.

The evidence from our result is a strong inclination toward cognitive biases
like the Dunning-Kruger effect in LLMs. While we must exercise caution before
jumping to any conclusion, our data contains scenarios where LLMs’ high
confidence does not always correlate with correct answers and vice versa.
However, these are hands-on evidence, and we don’t recommend it to be
definitive evidence of such psychological phenomena in these models. Our study
is a practical test to determine how well these models behave. It helps us try
to understand why these models sometimes act overconfident or provide
incorrect information because of the way they process information or what we
see because of the data they are exposed to.

###  4.1 Confidence Closeness

In the section above, we looked at how the correctness of LLM is compared to
their confidence. To take this one step further, we will look at their
correctness when based on the variable created in section 3.3. The relation
between ’A1’ and ’A2’ on how close they are pre-task and post-task serves as
an indicator for LLMs on how consistent the self-assessment is. A high value
in the ”Close_Correct” category implies that the model is generally correct
while being confident. In addition to that, it is also an indication that the
model maintains a consistent level of confidence before and after answering
the question. On the other side, a high count in the ”Close_Incorrect”
category suggests that the models’ confidence is stable even if their answers
are incorrect. The results are summarized in Table 4 in Appendix B.1.

As we have seen above, GPT-4 was very confident in its response regardless of
the correctness of the answer. We can see a similar pattern in this case, too.
Claude-2 shows a lower ”Close_Correct” but a higher ”Close_InCorrect” and
”Far_Correct” count. This is evidence that Claude-2 is not able to evaluate
itself, as when the confidence score was closer to each other, it had 14
incorrect responses out of 40 responses. Still, when the confidence scores
were far from each other, it had 15 correct out of 40. This suggests two
things:1) either Claude-2 initially had a low A1. After answering the
question, it increased its confidence score, A2, and then got it correct, or
2) it initially had a high A1 but later lowered its confidence, but it still
got it right. The first one tells us that Claude-2 was able to change and
update its evaluation correctly. Figure 1 illustrates Claude-2’s confidence
score to reflect their evaluating behavior. The four red dots on the x-axis
tell us that Claude-2 successfully lowered its confidence score after
answering the question, and the answer was incorrect. This means Claude-2 was
able to successfully assess itself after looking at the question for these
four instances. In most cases (shown by the green dots), when it increased its
confidence after looking at the question, it got the answers correct. However,
it did increase the confidence but yet got incorrect answers in some cases. A
similar observation was found for LLaMA-13B, where it has high counts in
”Close_InCorrect.” The zero count for GoogleBard in Far_Correct and
Far_Incorrect tells us that its evaluation is pretty much the same before and
after answering the question. Table 4 in Appendix B.1 shows the complete
result for all LLMs.

Figure 1: Comparison of A1 and A2 Scores for Claude-2.

###  4.2 Distribution of Confidence Scores

Figure 2: Facetted Density Plot of Confidence Levels by LLM Type. The plot reveals varying patterns of confidence distribution across different LLM types, suggesting nuanced self-perceptions in these models. Table 2: Summary Statistics of Confidence Scores by LLM Type LLM | A1 | R1 | A2 | R2
---|---|---|---|---
| Mean | SD | Mean | SD | Mean | SD | Mean | SD
Claude-2 | 4.800 | 2.911 | 4.650 | 2.957 | 5.400 | 4.241 | 5.400 | 4.235
Claude-Instant | 6.850 | 1.027 | 5.475 | 1.062 | 8.325 | 0.829 | 6.825 | 0.931
GoogleBard | 7.400 | 0.591 | 8.400 | 0.591 | 7.700 | 0.648 | 8.700 | 0.648
GooglePaLM | 5.500 | 1.485 | 4.600 | 1.646 | 7.050 | 1.260 | 6.050 | 1.260
GPT-3.5 | 7.525 | 0.877 | 7.475 | 0.877 | 8.900 | 0.672 | 8.900 | 0.672
GPT-4 | 8.900 | 0.568 | 9.200 | 0.372 | 8.925 | 0.594 | 9.225 | 0.375
LLaMA-13B | 7.550 | 2.062 | 6.950 | 2.136 | 7.725 | 1.921 | 7.400 | 1.892
LLaMA-70B | 7.350 | 1.122 | 7.950 | 1.339 | 8.600 | 0.672 | 8.475 | 0.847
LLaMA-7B | 7.250 | 1.214 | 6.600 | 1.297 | 8.025 | 1.187 | 7.525 | 0.877

The facetted destiny plot in Figure 2 with the summary of statistics given in
table 2 presents the distinct patterns in self-assessment across different
LLMs. The mean confidence level for A1 and R1 of Claude-2 is approximately 4.8
and 4.65, respectively. These mean confidence levels are coupled with higher
standard deviations of 2.91 and 2.95 simultaneously. The high standard
deviation for confidence level directs toward a broad spectrum of self-
perceived abilities. In addition, the post-task mean confidence level for A2
and R2 is also higher, with a higher standard deviation. Higher Standard
deviation for A2 and R2 implies significant inconsistencies in self-assessment
after completion of the task. Individually, the mean confidence score of A1
and R1 for Claude-Instant is 6.85 and 5.47, respectively, with a lower
standard deviation of 1.03 and 1.06 simultaneously. The confidence after
completing the task spiked to 8.32 and 6.82 for A2 and R2, maintaining the low
variability of data around 0.83 and 0.93, respectively.
Even though Google-Bard generally outperforms Google-PaLM across the board,
both of these models maintained consistent confidence metrics. In addition,
model GPT-3.5 and 4 also encompasses high mean confidence levels. GPT-4 shows
a mean A1 confidence score of 8.9 with a standard deviation of 0.568. Among
the LLaMA series, variability in confidence levels is more noticeable.
LLaMA-13B has a standard deviation of 2.06 for A1, which is higher, While
series LLaMA-70B and LLaMA-7B are in the range of 1.12 and 1.21, respectively.
To summarize, the findings here are detailed in the self-assessed confidence
levels with various LLMs. The destiny plot in upcoming sections will further
illustrate the trends, where the curve varies in width and height, implying
the observed mean and variability in confidence levels. These results
underscore the fact that our analysis should consider both central tendency
and dispersion for self-assessment mechanisms of LLMs.

The density plot illustrated in Figure 3 shows the distribution of confidence
scores across different LLMs for both A1 and A2 scores. A similar distribution
plot for R1 and R2 is in Appendix B.2 Figure 6. We can compare the
distributions across different LLMs and observe how their confidence scores
vary. For instance, the density plot for A1 in Figure 3 shows us that GPT-4 is
very confident in most of the cases. Figure 2, 3, and 6 give us an initial
picture of the variation of confidence scores in LLMs. Now we will incorporate
the correctness in the mix to study how well these LLMs do.

Figure 3: Density Plots of Correctness for Different Confidence Scores (A1 and
A2)

###  4.3 Category vs Confidence Scores

Understanding how LLMs self-assess their performance via confidence can
provide valuable perception of their limitations and capabilities. Our data
set provides a significant variation of LLMs performance across several
categories like LSAT- Reasoning, Mathematical Reasoning, and Truthful Q&A. Our
data set portrays confidence scores both before (A1 and R1) and after (A2 and
R2) answering the questions in above mentioned categories.

With reference to confidence, GPT-4 was able to succeed in setting itself
apart from others with consistency in its high absolute and relative pre-task
and post-task confidence levels across all tested categories. Significantly,
it exhibited unparalleled confidence in the LSAT Reasoning task. This implies
that it has more vital abilities in logical and analytical Reasoning. In
contrast, Claude-2 and Claude-Instant represented a less consistent confidence
profile. Even though Claude-2 demonstrated diminished pre-task and post-task
confidence levels in LSAT Reasoning, its confidence tends to improve in the
truthful Q&A category. The variation of this confidence advocates Claude-2 and
similar models may be optimized for specific types of tasks, ergo influencing
their self-assessed confidence. For a detailed review, readers are encouraged
to refer to Appendix Table B.4. The significant differences in how confident
the models are could help us understand how well they work for different types
of problems. When we observe the apparent differences in confidence among the
models, it can provide us with valuable insights into how well they can be
applied to various types of problems.

Figure 4: Average Confidence Levels by Category and LLM

In addition, models like LLaMA-70B shows high confidence score for LSAT
reasoning and Mathematical Reasoning; however, they possess lower confidence
score in the Truthful Q&A category. This kind of variability across different
types suggests models might have nuanced areas of expertise. Such within-model
variability across different categories suggests that individual models may
have nuanced areas of expertise. This tells us that we should be smart enough
to pick a model based on the categories.

It is noteworthy to mention the anomaly observed with Claude-2 in LSAT
Reasoning, where it recorded an extremely low confidence level, particularly
for the post-task metrics (A2 and R2). While the reason for this remains
elusive, it raises questions about the model’s internal evaluation mechanisms
or possible computational errors that need further careful observation. Our
analysis uncovers the complex landscape of confidence with different LLMs and
the problem categories that they are prompted to. A model like GPT-4 appeared
to be general for all the various tasks maintaining high confidence. However,
other LLMs seem to be specialized in a specific domain or yet to be tuned for
generalization. Our findings convey the information to consider model-specific
confidence evaluation while selecting a particular model for a specific task.
This also opens new interest in the research area for LLMs to understand
problem difficulty and correctness of the answer(IScorrect), offering more
perspective on performance and self-assessment of the model. As seen in Figure
4, there is a noticeable pattern in the confidence levels across different
problem categories and LLMs.

Our data set is based on comprehensive performance metrics, and the plots are
generated utilizing these datasets. Conspicuously, models like GPT-4 and
LLaMA-70B hold higher post-task confidence levels across all examined
categories. Mathematical Reasoning seems to be standing out with consistency
in high confidence levels, recommending that models are more secure in their
performance in mathematical tasks in comparison to other types of functions.
Our experimental data on the ’Truthful Q&A’ category displays variable
performance, suggesting that the nature of a task might affect LLMs’
confidence distinctively. These variations in confidence levels should be
considered to have a practical implication for the development of LLMs in
specializing in particular tasks.

###  4.4 Problem Level vs. Confidence Scores

The dataset table in Appendix B.5 Table 6 illuminates the average confidence
scores (both absolute and relative) expressed by different LLMs at different
problem levels (ranging from 1 to 5). The visualization for the table is
represented in Figure 5. Predominantly, it is conspicuous that as the level of
problem increases, the confidence score for LLMs decreases. The pattern is
very noticeable with the absolute confidence score. LLMs felt less sure about
their answers as the level of the problem increased. This result advocates
that LLMs may struggle to sustain high confidence when prompted with
convoluted tasks.

Figure 5: Average Confidence Scores by Problem Level

In contrast, a relative confidence score doesn’t follow this trend. Even
though there is a slight reduction in relative confidence as the level of
problem is increased, it is not as steep as the drop in absolute confidence.
This implies that LLMs might understand their performance in comparison to
others as relatively stable across different problem levels.

In addition, it is worth acknowledging that each LLM differs in their response
to problem level. To illustrate, GPT-4 maintained a high confidence score
across all problem levels, indicating its consistency in self-assessment of
its performance. However, models like Claude-2 and Claude-Instant represented
higher variability in their confidence scores as the level of the problem
changed. This is another indication that some models may adapt differently to
task difficulties. For example, Claude-2 shows a notable improvement in
confidence levels for certain problems.

It is imperative to underscore that our analysis provides an overview of the
average trends in confidence scores across different problem levels.
Individual variations in model behaviors are manifested, and this observation
showcases the need for a nuanced understanding of how different language
models respond to diverse problem complexities. In addition to this, further
investigation with the incorporation of additional factors like problem
content and correctness of answer could offer valuable insight into LLMs
performance and confidence assessment in particular scenarios.

##  5 Discussion & Conclusion

In this study, we analyzed the self-assessment behavior of Large language
models such as GPT, Claude, LLaMA, and others through their confidence scores,
investigating potential parallels with the Dunning-Kruger effect. As depicted
in table B.4 and figure 4 provides intriguing insights about how LLMs assess
their performance across different categories. Even though our study didn’t
establish a solid presence of the Dunning-Kruger effect, it provided valuable
observations aligning with its conceptual framework.

GPT-4 stands out with its consistency in high confidence scores across all the
tested categories, especially in LSAT Reasoning tasks. Such a high confidence
pattern resembles its high ability to gauge its competence accurately.
Nevertheless, it is essential to be careful to jump to any conclusion as there
might be other factors contributing to this trend. On the other hand, models
like Claude-2 and Claude Instant displayed higher variability in their
confidence scores across different categories. Claude-2 showed a relatively
low confidence score for LSAT Reasoning; however, they performed better in
Truthful Q&A. This difference mirrors the concept of individuals with varying
abilities showing inconsistency in assessments. For now, this observation
serves as a parallel instead of conclusive proof of the Dunning-Kruger
effect’s applicability in this context. LLaMA-70B performed better with a
higher confidence score in LSAT Reasoning and Mathematical categories but had
lower confidence in Truthful Q&A. This subtle variation aligns with the idea
that individual LLMs might possess specialized domains of competence, akin to
the Dunning-Kruger effect’s recognition of skill variations among individuals.

Referencing Table 6 and Figure 5, we investigate and explore the relationship
between problem-level complexity and LLM confidence scores. The observed
patterns in confidence offer exciting and interesting connections to the
Dunning-Kruger effect, even if they don’t provide solid evidence of it. LLMs
were observed to possess higher confidence scores starting at level 1. With
increasing complexity, a decrease in confidence score was observed. The
observed overconfidence phase is related to the overestimation part of the
Dunning-Kruger effect, wherein individuals with lower abilities often overrate
their competence. Different LLMs exhibited varying confidence score patterns
across the problem levels, reflecting the notion that individuals with
different abilities experience varying degrees of the Dunning-Kruger effect.
Also, Models like GPT-4 maintained their confidence similar to the individual
with high abilities making accurate self-assessments.

In a nutshell, the pattern of the LLM’s confidence score provides intriguing
parallels with the Dunning-Kruger effect. However, they don’t provide solid
evidence of its presence in the behavior of LLMs. To provide a sturdy
connection, further research with statistical analysis and a broader set of
variables is vital. Our findings, nevertheless, pave the way for deeper
exploration of LLMs with the Dunning-Kruger effect by showing the relationship
between self-assessment and competence in artificial intelligence. The
underlying intricacies of LLM behavior, the biases, and the confidence
framework demand a more in-depth, comprehensive exploration. It opens doors to
a myriad of questions that deserve attention, hinting at a treasure trove of
insights awaiting to dive deep. Delving deeper into this convergence of
psychology and artificial intelligence offers a promising frontier,
potentially unlocking novel insights into AI behavior and ethics. The
observations from this study beckon a broader exploration, suggesting that the
mysteries of AI cognition, akin to human nuances, are both vast and awaiting
discovery.

## References

  * [1] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * [2] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen.  A survey of large language models.  ArXiv, abs/2303.18223, 2023.
  * [3] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Seneviratne, Paul Gamble, Chris Kelly, Abubakr Babiker, Nathanael Schärli, Aakanksha Chowdhery, Philip Mansfield, Dina Demner-Fushman, Blaise Agüera Y Arcas, Dale Webster, Greg S Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan.  Large language models encode clinical knowledge.  Nature, 7 2023.
  * [4] Lei Wang, Chengbang Ma, Xueyang Feng, Zeyu Zhang, Hao ran Yang, Jingsen Zhang, Zhi-Yang Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji rong Wen.  A survey on large language model based autonomous agents.  ArXiv, abs/2308.11432, 2023.
  * [5] Yan Zhuang, Qi Liu, Yuting Ning, Wei Huang, Rui Lv, Zhenya Huang, Guanhao Zhao, Zheng Zhang, Qingyang Mao, Shijin Wang, and Enhong Chen.  Efficiently measuring the cognitive ability of llms: An adaptive testing perspective.  ArXiv, abs/2306.10512, 2023.
  * [6] Richard M. Shiffrin and Melanie Mitchell.  Probing the psychology of ai models.  Proceedings of the National Academy of Sciences of the United States of America, 120, 2023.
  * [7] Jen tse Huang, Man Ho Adrian Lam, Eric Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R. Lyu.  Emotionally numb or empathetic? evaluating how llms feel using emotionbench.  ArXiv, abs/2308.03656, 2023.
  * [8] Justin Kruger and David Dunning.  Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments, 1999.
  * [9] David Dunning.  The dunning-kruger effect. On being ignorant of one’s own ignorance, volume 44.  2011\.
  * [10] Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray.  Can ai language models replace human participants?, 7 2023.
  * [11] FX. Risang Baskara.  The promises and pitfalls of using chat gpt for self-determined learning in higher education: An argumentative review.  Prosiding Seminar Nasional Fakultas Tarbiyah dan Ilmu Keguruan IAIM Sinjai, 2:95–101, 5 2023.
  * [12] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  volume 35, pages 27730–27744. Curran Associates, Inc., 2022.
  * [13] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D Manning, Christopher Ré Diana, Acosta-Navas Drew, A Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi Sang, Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.  Holistic evaluation of language models; holistic evaluation of language models, 2022.
  * [14] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools.  2 2023.
  * [15] Mathias Kraus, Julia Anna Bingler, Markus Leippold, Tobias Schimanski, Chiara Colesanti Senni, Dominik Stammbach, Saeid Ashraf Vaghefi, and Nicolas Webersinke.  Enhancing large language models with climate resources.  3 2023.
  * [16] Dani Yogatama, Cyprien de Masson d’Autume, Jerome Connor, Tomas Kocisky, Mike Chrzanowski, Lingpeng Kong, Angeliki Lazaridou, Wang Ling, Lei Yu, Chris Dyer, and Phil Blunsom.  Learning and evaluating general linguistic intelligence.  1 2019.
  * [17] Alberto Acerbi and Joseph Stubbersfield.  Large language models show human-like content biases in transmission chain experiments.  2023\.
  * [18] Erik Jones and Jacob Steinhardt.  Capturing failures of large language models via human cognitive biases.  ArXiv, abs/2202.12299, 2022.
  * [19] Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia.  Cognitive mirage: A review of hallucinations in large language models.  ArXiv, abs/2309.06794, 2023.
  * [20] MD Vera Sorin, MD Danna Brin, MD Yiftach Barash, MD Eli Konen, MD PhD Alexander Charney, MD Girish Nadkarni, and MD Eyal Klang.  Large language models (llms) and empathy - a systematic review.  In medRxiv, 2023.
  * [21] Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.  Large language models can self-improve.  ArXiv, abs/2210.11610, 2022.
  * [22] Zhen Lin, Shubhendu Trivedi, and Jimeng Sun.  Generating with confidence: Uncertainty quantification for black-box large language models.  ArXiv, abs/2305.19187, 2023.
  * [23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [24] Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt.  Aligning ai with shared human values.  Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [25] Wanjun Zhong, Siyuan Wang, Duyu Tang, Zenan Xu, Daya Guo, Jiahai Wang, Jian Yin, Ming Zhou, and Nan Duan.  Ar-lsat: Investigating analytical reasoning of text, 2021.
  * [26] Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan.  From lsat: The progress and challenges of complex reasoning.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022.
  * [27] Sonal Sareen.  Chain of thoughts vs tree of thoughts for language learning models (llms), May 2023.

##  Appendix A Data

###  A.1 Survey Questions

  1. 1.

TruthfulQA: Included ten questions spread over five difficulty levels, with
two questions per level. The levels were:

    1. Level 1:

Logical Falsehood

    2. Level 2:

Nutrition

    3. Level 3:

Paranormal

    4. Level 4:

Myths and Fairytales

    5. Level 5:

Fiction

  2. 2.

TruthfulQA Extended: Ten questions spread over five difficulty levels, two per
level. The levels were:

    1. Level 1:

Proverbs

    2. Level 2:

Superstitions

    3. Level 3:

Misquotations

    4. Level 4:

Misconception

    5. Level 5:

Conspiracies

  3. 3.

Mathematical Reasoning: Spanning ten questions across:

    1. Level 1:

Elementary Mathematics

    2. Level 2:

High School Mathematics

    3. Level 3:

High School Statistics

    4. Level 4:

College Mathematics

    5. Level 5:

Abstract Algebra

  4. 4.

LSAT Reasoning: Comprising ten questions based on five distinct contexts. Each
context had two associated questions, with difficulty escalating from levels 1
to 5.

###  A.2 Variables

Table 3: Description of the Dataset Variables Variable Symbol | Variable Name | Type | Range/Example
---|---|---|---
Category | Category of the problem | Categorical | Truthful Q&A, Mathematical Reasoning
ProblemLevel | Problem level | Categorical | 1, 2, 3, 4, 5
ProblemID | Unique identifier for problem | Categorical | T1, T2, MR1
Problem | Text of the problem | Text | ”Are all real numbers real numbers?”
LLM | Type of Large Language Model | Categorical | GPT-4, GPT-3.5
A1 | Absolute Confidence (Pre) | Continuous | 1-10
R1 | Relative Confidence (Pre) | Continuous | 1-10
A2 | Absolute Confidence (Post) | Continuous | 1-10
R2 | Relative Confidence (Post) | Continuous | 1-10
IsCorrect | Correctness of answer | Binary | 0, 1

A1: Absolute confidence level expressed by the LLM before answering. Question:
”How well do you think you will do?”
R1: Relative confidence level expressed by the LLM before answering, compared
to others. Question: ”Compared to others, how well do you think you will do?”
A2: Absolute confidence level expressed by the LLM after answering. Question:
”How well do you think you did?”
R2: Relative confidence level expressed by the LLM after answering, compared
to others. Question: ”Compared to others, how well do you think you did?”

##  Appendix B Tables and Figures

###  B.1 Closeness vs correctness

Table 4: Your Caption Here

Absolute Confidence
---
LLM | Close_Correct | Close_Incorrect | Far_Correct | Far_Incorrect
Claude-2 | 4 | 15 | 15 | 6
Claude-Instant | 11 | 12 | 5 | 12
GoogleBard | 22 | 18 | 0 | 0
GooglePaLM | 9 | 9 | 7 | 15
GPT-3.5 | 14 | 5 | 9 | 12
GPT-4 | 25 | 15 | 0 | 0
LLaMA-13B | 7 | 24 | 2 | 7
LLaMA-70B | 8 | 20 | 8 | 4
LLaMA-7B | 9 | 14 | 6 | 11

Relative Confidence
---
LLM | Close_Correct | Close_Incorrect | Far_Correct | Far_Incorrect
Claude-2 | 4 | 16 | 15 | 5
Claude-Instant | 12 | 13 | 4 | 11
GoogleBard | 22 | 18 | 0 | 0
GooglePaLM | 9 | 9 | 7 | 15
GPT-3.5 | 14 | 5 | 9 | 12
GPT-4 | 25 | 15 | 0 | 0
LLaMA-13B | 7 | 26 | 2 | 5
LLaMA-70B | 10 | 20 | 6 | 4
LLaMA-7B | 8 | 15 | 7 | 10

###  B.2 Distribution of Confidence Scores

Figure 6: Density Plots of Correctness for Different Confidence Scores (R1 and
R2)

###  B.3 Confidence Scores vs Correctness

The destiny plot in Figure 7 represents the relationship between LLMs’
confidence score and their correctness in predicting answers. Destiny plot
branches each LLM’s confidence score into correct and incorrect categories
with distinct colors. The higher region in the destiny plot indicates the
model is correct or incorrect with specific confidence scores frequently. This
plot helps us in providing an initial empirical foundation to access the
Dunning-Kruger effect in LLMs. We are interested in where LLMs exhibit high
confidence scores and are incorrect or vice versa. This will provide us with
information on a misalignment between perceived ability and actual ability.

Figure 7: Density Plot of Correctness vs. Confidence Scores for Various
Language Learning Models (A1 and A2) Figure 8: Density Plot of Correctness vs.
Confidence Scores for Various Language Learning Models (R1 and R2)

###  B.4 Category Vs Confidence scores

Table 5: Performance and Confidence Metrics of Large Language Models (LLMs) Across Different Categories Category | LLM | Avg_A1 | Avg_A2 | Avg_R1 | Avg_R2
---|---|---|---|---|---
LSAT Reasoning | Claude-2 | 0.80 | 0.00 | 0.60 | 0.00
LSAT Reasoning | Claude-Instant | 7.00 | 8.20 | 6.00 | 7.20
LSAT Reasoning | GoogleBard | 7.00 | 7.60 | 8.00 | 8.60
LSAT Reasoning | GooglePaLM | 5.40 | 6.40 | 4.80 | 5.40
LSAT Reasoning | GPT-3.5 | 7.00 | 9.00 | 7.00 | 9.00
LSAT Reasoning | GPT-4 | 8.30 | 8.20 | 9.40 | 9.40
LSAT Reasoning | LLaMA-13B | 8.20 | 8.10 | 8.40 | 8.40
LSAT Reasoning | LLaMA-70B | 8.00 | 9.00 | 9.00 | 9.00
LSAT Reasoning | LLaMA-7B | 7.10 | 8.60 | 6.10 | 7.60
Mathematical Reasoning | Claude-2 | 5.60 | 4.40 | 5.20 | 4.40
Mathematical Reasoning | Claude-Instant | 6.80 | 8.90 | 5.50 | 7.20
Mathematical Reasoning | GoogleBard | 7.40 | 7.80 | 8.40 | 8.80
Mathematical Reasoning | GooglePaLM | 6.00 | 7.40 | 5.00 | 6.40
Mathematical Reasoning | GPT-3.5 | 7.60 | 9.00 | 7.60 | 9.00
Mathematical Reasoning | GPT-4 | 9.20 | 9.20 | 9.40 | 9.40
Mathematical Reasoning | LLaMA-13B | 8.00 | 8.70 | 7.20 | 8.10
Mathematical Reasoning | LLaMA-70B | 8.00 | 9.00 | 9.00 | 8.90
Mathematical Reasoning | LLaMA-7B | 7.80 | 8.40 | 6.80 | 7.40
Truthful Q&A | Claude-2 | 6.40 | 8.60 | 6.40 | 8.60
Truthful Q&A | Claude-Instant | 6.80 | 8.10 | 5.20 | 6.45
Truthful Q&A | GoogleBard | 7.60 | 7.70 | 8.60 | 8.70
Truthful Q&A | GooglePaLM | 5.30 | 7.20 | 4.30 | 6.20
Truthful Q&A | GPT-3.5 | 7.75 | 8.80 | 7.65 | 8.80
Truthful Q&A | GPT-4 | 9.05 | 9.15 | 9.00 | 9.05
Truthful Q&A | LLaMA-13B | 7.00 | 7.05 | 6.10 | 6.55
Truthful Q&A | LLaMA-70B | 6.70 | 8.20 | 6.90 | 8.00
Truthful Q&A | LLaMA-7B | 7.05 | 7.55 | 6.75 | 7.55

###  B.5  Problem Level Vs Confidence Scores

Table 6: Average Confidence Scores by Problem Level and LLM Problem Level | LLM | Avg_A1 | Avg_A2 | Avg_R1 | Avg_R2
---|---|---|---|---|---
1 | Claude-2 | 7.250 | 7.125 | 7.000 | 7.125
1 | Claude-Instant | 7.500 | 9.000 | 6.000 | 7.375
1 | GoogleBard | 7.500 | 7.250 | 8.500 | 8.250
1 | GooglePaLM | 6.250 | 7.750 | 5.250 | 6.750
1 | GPT-3.5 | 8.250 | 9.250 | 8.000 | 9.250
1 | GPT-4 | 9.250 | 9.125 | 9.500 | 9.250
1 | LLaMA-13B | 7.750 | 8.000 | 7.500 | 7.750
1 | LLaMA-70B | 8.000 | 8.625 | 8.500 | 8.250
1 | LLaMA-7B | 7.625 | 8.250 | 6.875 | 7.500
2 | Claude-2 | 4.750 | 5.625 | 4.750 | 5.625
2 | Claude-Instant | 7.000 | 8.250 | 5.750 | 7.000
2 | GoogleBard | 7.500 | 8.000 | 8.500 | 9.000
2 | GooglePaLM | 6.000 | 7.000 | 5.500 | 6.000
2 | GPT-3.5 | 7.500 | 8.125 | 7.500 | 8.125
2 | GPT-4 | 9.000 | 8.750 | 9.125 | 9.000
2 | LLaMA-13B | 7.750 | 8.000 | 7.500 | 7.750
2 | LLaMA-70B | 7.500 | 8.250 | 8.000 | 8.250
2 | LLaMA-7B | 6.875 | 7.875 | 6.125 | 7.375
3 | Claude-2 | 4.000 | 5.375 | 4.000 | 5.375
3 | Claude-Instant | 7.000 | 8.125 | 5.500 | 7.000
3 | GoogleBard | 7.000 | 7.500 | 8.000 | 8.500
3 | GooglePaLM | 5.250 | 6.500 | 4.250 | 5.500
3 | GPT-3.5 | 7.250 | 9.000 | 7.250 | 9.000
3 | GPT-4 | 8.500 | 8.500 | 9.000 | 9.000
3 | LLaMA-13B | 5.250 | 6.125 | 4.500 | 5.625
3 | LLaMA-70B | 6.750 | 8.750 | 7.250 | 8.500
3 | LLaMA-7B | 6.250 | 7.875 | 5.750 | 7.625
4 | Claude-2 | 5.250 | 4.875 | 5.000 | 4.875
4 | Claude-Instant | 7.000 | 8.500 | 5.750 | 7.000
4 | GoogleBard | 7.500 | 7.750 | 8.500 | 8.750
4 | GooglePaLM | 6.000 | 7.000 | 5.000 | 6.000
4 | GPT-3.5 | 7.750 | 9.000 | 7.750 | 9.000
4 | GPT-4 | 8.875 | 9.250 | 9.125 | 9.375
4 | LLaMA-13B | 8.750 | 8.000 | 7.750 | 7.875
4 | LLaMA-70B | 7.250 | 8.750 | 7.750 | 8.500
4 | LLaMA-7B | 8.000 | 8.000 | 7.500 | 7.625
5 | Claude-2 | 2.750 | 4.000 | 2.500 | 4.000
5 | Claude-Instant | 5.750 | 7.750 | 4.375 | 5.750
5 | GoogleBard | 7.500 | 8.000 | 8.500 | 9.000
5 | GooglePaLM | 4.000 | 7.000 | 3.000 | 6.000
5 | GPT-3.5 | 6.875 | 9.125 | 6.875 | 9.125
5 | GPT-4 | 8.875 | 9.000 | 9.250 | 9.500
5 | LLaMA-13B | 8.250 | 8.500 | 7.500 | 8.000
5 | LLaMA-70B | 7.250 | 8.625 | 8.250 | 8.875
5 | LLaMA-7B | 7.500 | 8.125 | 6.750 | 7.500