# Helium as a signature of the double detonation in Type Ia supernovae

Christine E. Collins,1,2 Stuart A. Sim,2 Luke. J. Shingles,1,2 Sabrina
Gronow,3,4 Friedrich K. Röpke,4,5 Rüdiger Pakmor,6 Ivo R. Seitenzahl7 and
Markus Kromer4
1GSI Helmholtzzentrum für Schwerionenforschung, Planckstraße 1, 64291
Darmstadt, Germany
2Astrophysics Research Center, School of Mathematics and Physics, Queen’s
University Belfast, Belfast BT7 1NN, Northern Ireland, UK
3Zentrum für Astronomie der Universität Heidelberg, Astronomisches Rechen-
Institut, Mönchhofstr. 12-14, 69120 Heidelberg, Germany
4Heidelberger Institut für Theoretische Studien, Schloss-Wolfsbrunnenweg 35,
69118 Heidelberg, Germany
5Zentrum für Astronomie der Universität Heidelberg, Institut für theoretische
Astrophysik, Philosophenweg 12, 69120 Heidelberg, Germany
6Max-Planck-Institut für Astrophysik, Karl-Schwarzschild-Str. 1, D-85748,
Garching, Germany
7School of Science, University of New South Wales, Australian Defence Force
Academy, Canberra, ACT 2600, Australia
E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

The double detonation is a widely discussed mechanism to explain Type Ia
supernovae from explosions of sub-Chandrasekhar mass white dwarfs. In this
scenario, a helium detonation is ignited in a surface helium shell on a
carbon/oxygen white dwarf, which leads to a secondary carbon detonation.
Explosion simulations predict high abundances of unburnt helium in the ejecta,
however, radiative transfer simulations have not been able to fully address
whether helium spectral features would form. This is because helium can not be
sufficiently excited to form spectral features by thermal processes, but can
be excited by collisions with non-thermal electrons, which most studies have
neglected. We carry out a full non-local thermodynamic equilibrium (non-LTE)
radiative transfer simulation for an instance of a double detonation explosion
model, and include a non-thermal treatment of fast electrons. We find a clear
He i $\lambda 10830$ feature which is strongest in the first few days after
explosion and becomes weaker with time. Initially this feature is blended with
the Mg ii $\lambda 10927$ feature but over time separates to form a secondary
feature to the blue wing of the Mg ii $\lambda 10927$ feature. We compare our
simulation to observations of iPTF13ebh, which showed a similar feature to the
blue wing of the Mg ii $\lambda 10927$ feature, previously identified as C i.
Our simulation shows a good match to the evolution of this feature and we
identify it as high velocity He i $\lambda 10830$. This suggests that He i
$\lambda 10830$ could be a signature of the double detonation scenario.

###### keywords:

radiative transfer – line: formation – methods: numerical – transients:
supernovae – white dwarfs

††pubyear: 2023††pagerange: Helium as a signature of the double detonation in
Type Ia supernovae–Helium as a signature of the double detonation in Type Ia
supernovae

##  1 Introduction

Type Ia supernovae (SNe Ia) are the thermonuclear explosion of a white dwarf,
but the exact mechanism by which the white dwarf explodes is not yet fully
understood. The double detonation is a promising scenario to explain SNe Ia
for a range of luminosities. In this scenario, a helium detonation is ignited
in a helium shell on a carbon-oxygen white dwarf. The helium detonation then
ignites a secondary carbon detonation in the core. Early double detonation
models (see eg. Taam, 1980; Nomoto, 1980, 1982; Livne, 1990; Woosley & Weaver,
1994; Höflich & Khokhlov, 1996; Nugent et al., 1997) invoked relatively
massive helium shells ($\sim 0.2$ ${\rm M}_{\sun}$), and predicted light
curves and spectra that were not consistent with those of normal SNe Ia, due
to over production of iron group elements in the outer ejecta. There has been
renewed interest in the double detonation following suggestions that
considerably lower mass helium shells ($<0.1$${\rm M}_{\sun}$) may be able to
ignite a secondary core detonation (Bildsten et al., 2007; Shen & Bildsten,
2009; Fink et al., 2010; Shen et al., 2010), and significantly reduce the
discrepancies with observations caused by the over-production of iron group
elements at high velocities, produced in the helium shell detonation (Kromer
et al., 2010; Woosley & Kasen, 2011). Simulations by Townsley et al. (2019)
and Shen et al. (2021) have suggested that by considering minimal helium shell
masses, double detonations may be able to account for normal SNe Ia, although
Collins et al. (2022) show that a minimal He shell mass does not necessarily
lead to good agreement with normal SNe Ia. Additionally, Glasner et al. (2018)
have shown for the first time conditions leading to the ignition of a helium
detonation in an accreted helium envelope on top of a carbon-oxygen white
dwarf.

A number of objects have recently been observed that have specifically been
suggested to be produced by helium shell detonations, for example SN 2016jhr
(Jiang et al., 2017) and SN 2018byg (De et al., 2019). These objects showed
early peaks in their light curves, suggested to be the result of radioactive
material produced in the outer layers of the ejecta. This has been predicted
as a signature of a double detonation by Noebauer et al. (2017). Here,
however, we consider a different potential signature of a double detonation.
We investigate whether He spectral features could be directly detected. Double
detonation simulations predict fairly large amounts of unburnt helium at high
velocities in the outer layers of the SN ejecta (Fink et al., 2010; Shen &
Moore, 2014; Gronow et al., 2020; Gronow et al., 2021). In each of the double
detonation models presented by Fink et al. (2010), approximately half of the
initial helium shell mass was ejected as unburnt helium. Radiative transfer
simulations have not been able to predict whether helium features should be
able to form for the recent generation of double detonation simulations
considering low mass helium shells (e.g. Kromer et al., 2010; Woosley & Kasen,
2011; Townsley et al., 2019; Polin et al., 2019; Gronow et al., 2020; Shen et
al., 2021; Collins et al., 2022; Pakmor et al., 2022), as these studies did
not carry out full non-local thermodynamic equilibrium (non-LTE) simulations
including non-thermal processes.

The ionisation and excitation rates of helium are strongly affected by
collisions with non-thermal electrons, and therefore to make predictions of
helium spectral features, non-LTE radiative transfer simulations are required,
and must account for non-thermal processes. Non-thermal electrons are produced
by $\gamma$-rays from the decays of 56Ni. The importance of collisions with
non-thermal electrons on the formation of helium features has been
demonstrated for Type Ib/Ic supernovae (Chugai, 1987; Lucy, 1991; Hachinger et
al., 2012).

Non-LTE simulations by Dessart & Hillier (2015) for a helium shell detonation
model by Waldman et al. (2011) clearly showed that He spectral features form,
where radiative transfer calculations are non-LTE and include non-thermal
effects. This model, however, did not consider a secondary core detonation,
and therefore the composition of the ejecta is different to the composition
predicted for the double detonation scenario. In particular, this model
predicted large amounts of helium at all velocities, whereas the double
detonation predicts helium predominantly at high velocities (see e.g. Fink et
al., 2010). The predicted spectral properties of the helium detonation model
were classified as a hybrid Type Ia/Ib due to the clear detection of helium
features.

Boyle et al. (2017) investigated whether helium spectral features could form
for the recent generation of double detonation models with relatively low mass
helium shells (e.g. the models of Fink et al., 2010). In this study, Boyle et
al. (2017) used an analytic approximation to estimate the excited He i level
populations, and found that the He i 10830 Å and 2 $\mu$m lines may be
produced by high velocity He ($\sim$19 000 km s-1). They note that since the
He i 10830 Å is substantially blueshifted, it may be blended with the blue
wing of the Mg ii 10927 Å feature. Further, more detailed simulations are
required to confirm this result, but this suggests that He i spectral features
may be observable for this scenario, and indeed recently Dong et al. (2022)
showed that SN 2016dsg had an absorption line around 9700-10500 Å which they
find is consistent with the He i 10830 Å feature predicted by Boyle et al.
(2017).

Liu et al. (2023) have pointed out that SN 2022jgb and other candidate thick
helium-shell detonation SN 2018byg-like SNe (De et al., 2019) show a prominent
absorption feature at $\sim 1\mu$m, which Liu et al. (2023) suggest could be
high-velocity ($\sim 26000$ km s-1) He $\lambda 10830$. Previous works have
also attempted to identify helium spectral features in the near-IR spectra of
SNe Ia (Meikle et al., 1996; Höflich & Khokhlov, 1996; Mazzali & Lucy, 1998;
Marion et al., 2003, 2009). Marion et al. (2009) present a large sample of
near-IR spectra of normal SNe Ia and do not find any clear detections of the
strongest lines expected for He i, at 10830 Å and 2 $\mu$m. However, they also
do not detect C i in any of the spectra, which has since possibly been
identified in near-IR spectra (Hsiao et al., 2013, 2015b; Marion et al., 2015;
Wyatt et al., 2021). Wyatt et al. (2021) consider whether their C i detections
could be misidentified He i. Following discussion of simulations by Boyle et
al. (2017), Wyatt et al. (2021) conclude that it is unlikely the feature is He
i, given that in the simulations of Boyle et al. (2017), helium absorption is
visible at maximum light and grows stronger with time, whereas the observed
feature becomes weaker toward maximum light. Additionally, the observed
spectra show optical C ii 6580 Å, confirming the presence of carbon in the SN
ejecta. While the lower mass model considered by Boyle et al. (2017) does
clearly show a trend of increasing He i 10830 Å strength, the higher mass
model predicts that the He i 10830 Å feature remains a similar strength to the
Mg ii 10927 Å feature. This indicates that the evolution of potential He
features is likely model dependent, and requires further study. The
approximate non-LTE treatment used by Boyle et al. (2017) assumes that the He
ii population is dominant. One limiting factor for this approximation is
whether the non-thermal ionisation rate remains sufficiently high that the
helium remains ionised. Therefore, the strengths of the predicted features
provide upper limits on the potential strength of helium features. In this
work, the ionisation state is calculated self-consistently, and can therefore
better constrain the predicted strength of potential helium features.

In this paper, we investigate the potential for He spectral features to form
in the double detonation explosion scenario, and whether these features could
be detected in observations. The presence of He in the ejecta is a fundamental
prediction of the double detonation scenario, but it has not yet been fully
addressed whether the He would produce an observable spectral feature for
contemporary double detonation explosion simulations. If an observable He
spectral feature is produced, then this could be direct observational evidence
for the double detonation explosion scenario, allowing us to identify the
class of explosion as well as the progenitor system. We consider an ejecta
profile from a hydrodynamical simulation of the double detonation, which we
describe in Section 2.2. In Section 3.1, we present the simulated spectra,
focusing specifically on predictions of He spectral features. We discuss how
this simulation compares to observations of a SN Ia that showed similar
behaviour to our simulation in Section 3.2.2, suggesting that He may have
already been detected in SNe Ia, but misidentified.

##  2 Methods

###  2.1 Non-LTE and non-thermal radiative transfer

We use the time-dependent multi-dimensional Monte Carlo radiative transfer
code artis (Sim 2007; Kromer & Sim 2009; Bulla et al. 2015, based on the
methods of Lucy 2002, 2003, 2005). Artis has been extended by Shingles et al.
(2020) to solve the non-LTE population and ionisation balance, including
ionising collisions with fast non-thermal electrons. artis solves the Spencer-
Fano equation (as framed by Kozma & Fransson 1992) to trace the energy
distribution of high-energy leptons seeded by high-energy particles emitted in
radioactive decays. Similarly to Shingles et al. (2020), we allow Auger
electrons to contribute to heating, ionisation, and excitation. We do not
include excitation of bound electrons by non-thermal collisions, since this is
particularly computationally demanding, however, Hachinger et al. (2012) show
that their results for He are rather insensitive to the excitation rates.

For photoionisation, artis uses the full Monte Carlo photon-packet
trajectories (following Lucy 2003) to obtain a rate estimator for each level
pair. For all other atomic processes, a parameterised radiation field is used
to estimate transition rates. We use an atomic dataset based on the
compilation of cmfgen (Hillier, 1990; Hillier & Miller, 1998, see Shingles et
al. 2020 for details).

The radiative transfer simulations are carried out between 2 and 32 days after
explosion, using 111 logarithmically spaced time steps. The initial time steps
are in LTE and at time step 12 (2.7 days) the non-LTE solver is switched on.

###  2.2 Double detonation ejecta model

Figure 1: Model densities at 100 seconds after explosion.

The ejecta model we consider in this work is based on the 3D double detonation
explosion model M2a, described by Gronow et al. (2020). The light curves
predicted for this model were of normal SNe Ia brightness, similar to SN
2011fe (Nugent et al., 2011). We therefore select this model to investigate
the potential for He spectral features to form in normal brightness SNe Ia. We
construct a 1D model from a slice of model M2a in an equatorial line of sight.
Although artis can perform multi-dimensional simulations, the performance cost
is substantial and requires sacrifices to the number of detailed
photoionisation estimators. We find that at early times, while the ejecta are
optically thick, this 1D model produces light curves and spectra similar to
those expected from a 3D simulation in this line of sight. However, after
maximum light, as the ejecta become more optically thin, we find that this
model is no longer such a good representation of the 3D line of sight.
Therefore, in this work we limit our calculations to the first few weeks after
explosion. Since the He is predominantly found in the outer ejecta, we expect
that He features are more likely to form at early times. We show the
composition of this model in Figure 1. We note that low density, outer ejecta
($v>23200$ km s-1) are ignored since these quickly become optically thin and
are not expected to affect the spectrum at the times we consider here. This
allows for better spatial resolution of the ejecta at velocities important in
forming the spectra. The ejecta in the equatorial line of sight of Model M2a
that we have based our ejecta on extend to $\sim 25000$ km s-1, however, in
the polar lines of sight the ejecta extend to higher velocities of $>30000$ km
s-1 (see Gronow et al. 2020).

The initial WD mass of Model M2a was 1.05 M⊙, with a He shell of 0.073 M⊙ (see
Gronow et al. 2020 for model details). After explosion, the total mass of He
in the ejecta of Model M2a was 0.028 M⊙, and the mass of 56Ni was 0.58 M⊙. In
the 1D model we consider here the He mass is 0.018 M⊙, and the 56Ni mass is
0.49 M⊙.

The radiative transfer simulations carried out by Gronow et al. (2020) did not
include non-thermal ionisation, and made approximate non-LTE calculations.
Therefore they could not address whether helium spectral features would be
expected to form for these models. In this work we carry out a full non-LTE
simulation and include non-thermal ionisation (as implemented by Shingles et
al. 2020), with the aim of investigating the effect of the more detailed
treatment of ionisation on the formation of He features for the double
detonation scenario. In this work, we investigate only one model due to the
high computational expense of these simulations.

The double detonation scenario leads to strong asymmetries, and therefore, in
future, 3D simulations should be carried out to investigate observer angle
dependencies. Additionally, the ejecta structure and synthesised abundances
show strong variations with different masses of the initial He shell and C/O
core (e.g. Fink et al., 2010; Gronow et al., 2021; Boos et al., 2021). It
should also be investigated how the predicted He spectral features would vary
in response to such differences in the ejecta structure.

Figure 2: Spectrum at 5 days after explosion calculated using artis including
a non-thermal treatment. We show the contributions of He i and Mg ii to the
spectrum, and indicate the absorption by these ions beneath the axis. The
wavelength of the minimum of the He i 10830 Å feature is indicated by the
dashed line. We note that no C i feature is formed.

(a) 8 days after explosion.

(b) 10 days after explosion.

(c) 14 days after explosion.

Figure 3:  Model spectra sequence showing the evolution of the He i 10830 Å
feature with time. The simulated time since explosion is noted below each
model spectrum. The wavelengths of the minima of the He i 10830 Å and Mg ii
10927 Å features are indicated by the dashed lines. We compare our model
spectra to iPTF13ebh (Hsiao et al., 2015b). Since the time of explosion for
this object is uncertain we list the observed times relative to B-band maximum
of iPTF13ebh. The observed spectra have been scaled to be of similar
brightness to our simulated spectra. We note that our model does not predict a
C i feature.

##  3 Results

###  3.1 Helium spectral features

In future work we will present the full results of this non-LTE simulation,
but in this paper we limit discussion to the ability of this simulation to
produce He spectral features. We note that a level of Monte Carlo noise is
present in our model spectra.

We identify the presence of He features in our simulated spectra by tagging
escaping Monte Carlo radiation packets with the ion with which they last
interacted. By examining packets whose last interaction was with He, we can
quantify the contribution of He to the synthetic spectra and identify the
specific spectral features for which He is responsible.

(a) 5 days after explosion. The dashed lines indicate the wavelengths of the
He i absorption.

(b) 9 days after explosion.

Figure 4: Model optical spectra where the contributions of key species are
indicated. We compare the spectrum at 9 days after explosion to the spectrum
of iPTF13ebh (at 11.1 days before the observed B-band maximum light). The
observed spectrum has been scaled to be of similar brightness to the simulated
spectrum.

####  3.1.1 NIR He i $\lambda$10830

We predominantly focus on the wavelength region around where the He i 10830 Å
(23S - 23P) feature would form. We expect this to be the strongest He feature
with the least blending from other spectral lines. In Figure 2 we show the
spectrum at these wavelengths at 5 days after explosion, and indicate the
contributions of He i to the spectrum. In the radiative transfer simulation,
we record details of the last interaction each Monte Carlo packet underwent
before escaping the ejecta. For each wavelength bin in the synthetic spectrum
the area under the spectrum is colour coded in proportion to the energy
carried by packets in that wavelength bin whose last interaction was with each
of the ions considered. We also construct an equivalent histogram based on
where the wavelength bin packets were prior to their last interaction (i.e.
indicating where packets last underwent absorption/scattering/fluorescence)
and plot this on the negative axis under the spectrum to indicate the key
absorption processes. We find that the He i 10830 Å feature is expected to
form for this model, and that a relatively strong He absorption feature forms
in the first few days after explosion. Over time the strength of this feature
weakens, and at times later than $\sim$2 weeks after explosion this feature is
no longer visible (see the spectral time series shown by Figures 3(a), 3(b),
and 3(c)).

In Figure 2, showing the spectrum at 5 days after explosion, the He i 10830 Å
feature is initially completely blended with the Mg ii 10927 Å feature. In
observations, this could be mistaken for a broad Mg ii feature. In the region
around the 10830 Å feature, the emission from other lines is relatively weak,
hence we find the 10830 Å feature to be relatively strong. The ‘Other’
elements in Figure 2 are predominantly intermediate mass elements, with the
strongest contributions being from Si and S.

By 8 days after explosion, the He i feature has separated from the Mg ii
feature and is clearly distinguishable at the blue wing of the Mg ii feature.
The wavelength bin sizes have been chosen to minimise Monte Carlo noise in the
model spectra. We note that the He i feature separating from the Mg ii feature
can be seen when using smaller wavelength bins. At 10 days after explosion, a
distinct secondary feature to the blue wing of the Mg ii feature can also be
seen in Figure 3(b), although this is weaker than at 8 days after explosion.
At 5 days after explosion, the He i 10830 Å feature is formed at $\sim$19500
km/s, while the Mg ii 10927 Å feature is formed at $\sim$16000 km/s. Over
time, the velocity of the He i 10830 Å feature decreases. At 2 weeks after
explosion the feature is formed at $\sim$18000 km/s.

Such features to the blue wing of the Mg ii feature have previously been
detected (Höflich et al., 2002; Hsiao et al., 2013, 2015b; Marion et al.,
2015), but have been identified as C i 10693 Å. This will be discussed further
in Section 3.2.1. By 2 weeks after explosion, a clear He i 10830 Å feature can
no longer be seen, however, the blue wing of the Mg ii feature is flattened
due to He i 10830 Å.

This demonstrates that NIR He i 10830 Å could be a signature of the double
detonation explosion scenario, and that early NIR spectra are key to detecting
the presence of this feature.

####  3.1.2 Optical He

We now discuss the presence of He features in the model optical spectra. At 5
days we find the He i 5876 Å feature to be the strongest optical He feature
(see Figure 4(a)), although, it is blended with other strong lines,
predominantly from S ii and Si ii. A weak He i absorption feature is found at
$\sim 5555$ Å, however, this is blended with S ii absorption. Similarly to the
10830 Å feature, this is also strongest at the earliest times, and becomes
weaker over time, however the 5876 Å feature fades more rapidly than the 10830
Å, and is no longer formed by $\sim$ 9 days after explosion. Figure 4(b) shows
the key species contributing to the optical spectrum at 9 days after
explosion, where contributions from He i can no longer be seen around 5876 Å.
This shows that the He i 10830 Å feature can be relatively strong without the
expectation that the 5876 Å feature should also be strong. This was also noted
by Mazzali & Lucy (1998), and Nugent et al. (1997) found no strong evidence
for optical He lines in their calculations for a helium detonation model. We
note that we find no He ii features in our simulation.

####  3.1.3 He i 2 $\mu$m

Our model spectra show a high level of Monte Carlo noise in the wavelength
region around 2 $\mu$m, where we would expect the He i 2 $\mu$m feature to
form. Increasing the number of Monte Carlo packets in our simulation to
improve the signal to noise at such red wavelengths is prohibitively
computationally expensive at this time. We therefore cannot predict whether
this model would produce a clear He i 2 $\mu$m feature. However,
qualitatively, we find that there are contributions to the spectrum from He i
in this wavelength region at very early times which decrease in strength with
time and have faded by $\sim 1$ week after explosion. We therefore suggest
that it is possible that a He i 2 $\mu$m feature could form within the first
week after explosion, however we cannot confirm this with our current
simulation.

###  3.2 Discussion

We have shown for a particular instance of a double detonation model that the
formation of a clear He i 10830 Å feature is predicted by our radiative
transfer calculations. Future simulations should investigate whether this is a
prediction of all double detonation models. Importantly, we do not expect a
clear He feature in the optical spectra, implying that the non-detection of an
optical He spectral feature can not be used to rule out the presence of He
when identifying a He i 10830 Å feature.

The He shell in this model before ignition was relatively massive (compared to
that considered by Townsley et al. 2019, or the minimal He shell models
presented by Gronow et al. 2021 and Boos et al. 2021), and so has more He in
the ejecta at a more extended range of velocities than would be synthesised
for a model with a thinner He shell. Additionally, the 3D model M2a (Gronow et
al., 2020) on which our 1D model is based showed strong asymmetries. For
example, He extends to higher velocities along the negative polar axis (see
Gronow et al. 2020, figure 13), in which case a He feature may form at higher
velocities. Investigating the dependence of the strength and velocity of He
features on the choice of model and observer orientation is beyond the scope
of this work, however, it is likely that both the mass of He, and the mass of
56Ni mixed with the He will influence the predicted He spectral features. In
our model, 56Ni is synthesised in the He shell detonation, and the presence of
56Ni in the ejecta in the same regions as the He likely increases the number
of non-thermal collisions with He. Future simulations should investigate the
sensitivity of the line formation to the degree of mixing of 56Ni with the He.

It is interesting that we find the He features become weaker with time, given
that in simulations of SNe Ib this feature is found to grow in strength with
time (Hachinger et al., 2012; Teffs et al., 2020), and also grows in strength
in the He shell detonation model of Dessart & Hillier (2015). The He in the
SNe Ib simulations, and in the He shell detonation model, extends over a wide
range of velocities. In our simulation, however, the He is limited to a
smaller range of velocities (see Figure 1) in the outer ejecta. In the model
considered by Dessart & Hillier (2015) He is the most abundant element at all
velocities (see their figure 1) and they have more He by mass than in our
model. Dessart & Hillier (2015) find that He i 10830 Å is strong at all times
beyond 5 days. They also find He i 5875 Å is clearly visible between 5–15 days
until line blanketing by Ti ii causes the feature to fade. It is possible that
the He i 10830 Å feature remained visible at all times due to the high
abundance of He present at all velocities, and that the strengths of the He i
features were found to be so strong due to the higher abundance of He in the
model. Future work should investigate how model structure, and masses of He
affect predictions of He spectral features.

Boyle et al. (2017) consider both a low-mass model (0.58 ${\rm M}_{\sun}$ CO
core and 0.21 ${\rm M}_{\sun}$ He shell) and a high-mass model (1.025 ${\rm
M}_{\sun}$ CO core and 0.055 ${\rm M}_{\sun}$ He shell). Like our model, the
high-mass model shows a deficit of He at intermediate velocities, however,
their low-mass model does not show this (see figure 3 of Boyle et al. 2017).
Their low-mass model predicts that the He i 10830 Å feature clearly grows in
strength, and becomes much stronger than the Mg ii 10927 Å feature. In the
high-mass model, however, the He i 10830 Å feature becomes slightly stronger
with time, but remains similar in strength to the Mg ii 10927 Å feature
(figure 11 of Boyle et al. 2017). The approximate non-LTE treatment used by
Boyle et al. (2017) assumed that the He ii population is dominant. In our
simulation we find that indeed He ii is the dominant ionisation state in the
outer ejecta throughout our simulation. Generally, He i is the next most
highly populated in the outer ejecta.

####  3.2.1 NIR C i

A number of detections of a feature to the blue wing of the Mg ii feature
(similar to our simulated He i 10830 Å feature) have been reported, which have
previously been suggested to be C i. Specifically, clear features were
identified in SN 1999by (Höflich et al., 2002), iPTF13ebh (Hsiao et al.,
2015b), SN 2015bp (Wyatt et al., 2021) and SN 2012ij (Li et al., 2022), and an
additional feature affecting the blue wing of the Mg ii feature has been
reported in SN 2011fe (Hsiao et al., 2013) and SN 2014J (Marion et al., 2015).

Hsiao et al. (2015b) claim the feature in the blue wing of the Mg ii feature
in iPTF13ebh is C i (see Section 3.2.2 below). Wyatt et al. (2021) and Liu et
al. (2023) have detected similar features in SN 2015bp and SN 2012ij
respectively, for which they also propose identification with C i. These
objects were classified as transitional SNe Ia, showing similar spectral
properties to 91bg-like SNe, but also showing secondary NIR maxima, which also
includes objects such as SN 1986G (Phillips et al., 1987), SN 2003gs
(Krisciunas et al., 2009) and SN 2004eo (Pastorello et al., 2007; Mazzali et
al., 2008). These have been suggested to link 91bg-like SNe to normal SNe Ia.
Höflich et al. (2002) also identify the feature in SN 1999by as C i. SN 1999by
is a 91bg-like, subluminous object, and the spectral evolution of this feature
is similar to that in iPTF13ebh.

C i has tentatively been identified in the NIR spectra of the normal SN 2011fe
(Hsiao et al., 2013), which showed a flattened wing to the Mg ii feature. SN
2014J also showed a flattened wing similar to that in SN 2011fe, and was shown
by Marion et al. (2015) to be a possible detection of C i. The feature in SN
2011fe showed an increase in strength towards maximum, however in iPTF13ebh it
rapidly decreased in strength.

In our simulation we do not find any C i features. We do find contributions to
the optical spectra from C ii 6580 Å and 7235 Å, but these are weak and would
not be easily detectable as they are heavily blended with stronger lines (see
Figure 4(a)). At the times when our He i feature forms, even in the outer
layers of our simulation ejecta where temperatures are coolest and the ejecta
are the least highly ionised, C ii is the dominant ionisation stage, with C
iii being the next most abundant, indicating that in this region C iii is
recombining to C ii. C i is significantly depopulated relative to both C ii
and C iii (on the order of 5 magnitudes lower ionisation fraction). As such,
we would not expect C i features to form. This is consistent with e.g. Tanaka
et al. (2008) and Heringer et al. (2019) who also found C ii to be the
dominant ionisation state, rather than C i. However, note that some
simulations have found that C i features can form (e.g. Höflich et al., 2002;
Hsiao et al., 2015b).

####  3.2.2 Comparison of simulated spectra to observations

Table 1: Peak light curve brightnesses for iPTF13ebh from Hsiao et al. (2015b), compared to the values calculated for our model. | iPTF13ebh | Model
---|---|---
Mu,max | -18.34 ± 0.19 | -18.2
MB,max | -18.95 ± 0.19 | -18.9
MV,max | -19.01 ± 0.18 | -19.1
Mg,max | -19.03 ± 0.18 | -19.1
Mr,max | -18.99 ± 0.18 | -18.8
Mi,max | -18.52 ± 0.18 | -18.3
MJ,max | -18.75 ± 0.18 | -18.4
MH,max | -18.59 ± 0.18 | -18.7

We compare our simulated spectra to those of iPTF13ebh (Hsiao et al., 2015b),
which showed a clear second component to the blue wing of the Mg ii feature in
the early NIR spectra, and was identified as C i 10693 Å. We chose this object
given the high quality NIR spectra obtained at early times. It also showed
similar peak brightnesses to our model (see Table 1). We note that we have not
tuned our model in any way to try to match the spectra of iPTF13ebh. We are
comparing our prediction based on an existing double detonation ejecta model
to the observations of iPTF13ebh given the similarities between our simulated
He i 10830 Å feature and the observed feature (previously identified as C i)
in iPTF13ebh.

iPTF13ebh has a $\Delta$m15(B) value of 1.79 and is therefore in the fast-
declining SN Ia category, but is classified as a transitional event: the
photometric properties of iPTF13ebh place it in the category of normal SNe Ia,
however its spectral properties resemble those of SN 1999by which is a member
of the 91bg-like class. iPTF13ebh however, showed no apparent Ti ii features
and is therefore not considered a 91bg-like object.

Hsiao et al. (2015b) found no optical C i features in iPTF13ebh. They did,
however, find evidence for a weak optical C ii 6580 Å feature at approximately
the same epoch, which they also found to weaken rapidly. They propose
identification of C i 9093 Å as weak notches in the two earliest spectra, and
suggest that C i 9406 Å and 11754 Å features could be present in the spectra,
but blending makes these detections uncertain.

Since the observed feature at the blue wing of the Mg ii 10927 Å feature in
iPTF13ebh shows similar behaviour to the simulated high velocity He i 10830 Å
feature in our model and the feature, we compare our model to the early NIR
spectra of iPTF13ebh (Hsiao et al., 2015a). All spectra have been dereddened
and redshift corrected.

We plot the spectra of iPTF13ebh in Figures 3(a), 3(b) and 3(c), along with
our model spectra. We use times relative to maximum light for the observed
spectra since the inferred time since explosion is uncertain (see Hsiao et al.
2015b). In the spectra of iPTF13ebh the feature can be seen decreasing in
strength over time until in Figure 3(c) at 6.8 days before maximum it is
barely visible. Maximum light in B-band occurs in our simulation at $\sim$18.6
days after explosion. The He i feature predicted in our simulation is able to
reproduce the blue wing of the Mg ii feature in all three spectra, although at
$\sim$ 2.2 days earlier than the times of the observed spectra. The velocities
of the He i and Mg ii features appear to be well matched, as well as the trend
in strength of the He i feature relative to the Mg ii feature.

Given this match between the evolution of our He i 10830 Å feature and the
feature in iPTF13ebh, as well as the lack of C i in our model spectra, we
identify this feature as He i $\lambda 10830$, and suggest that this could be
a signature of the double detonation scenario.

We show a comparison between the optical spectrum of our model at 9 days after
explosion to the observed spectrum of iPTF13ebh in Figure 4(b). Our model
shows a Ti ii trough at $\sim$ 4000–4500 Å which is not observed in iPTF13ebh,
but apart from this feature the spectra show reasonable agreement. At this
time our model does not predict any optical He i features, which is consistent
with the observations of iPTF13ebh. We note that the spectra of iPTF13ebh have
strong telluric absorption in the region around where the He i 2 $\mu$m
feature could form (Hsiao et al., 2015b), and therefore we can not identify
whether this feature is present in the observed spectra.

##  4 Conclusions

We have shown for an instance of a double detonation explosion model that He i
10830 Å is a predicted spectral feature. By including a non-thermal solution
in our calculations, He i features are able to form for the double detonation
scenario, and could be a signature of this explosion mechanism.

In our simulation, we find a clear absorption feature due to high velocity He
i 10830 Å. We find the He i 10830 Å feature is strongest in the first few days
after explosion, and becomes weaker over time. After 2 weeks, this feature is
no longer visible. The He i 10830 Å feature is initially blended with Mg ii,
and therefore could potentially be concealed in observed spectra. We find that
over time the He i feature separates from the Mg ii, such that we see two
distinct features. We note, however, it is likely that the strength and
evolution of the He i $\lambda 10830$ feature is model dependent, and future
studies should be carried out investigating different mass models.

We find that the He i 5876 Å feature is able to form in our simulation at very
early times, however, this is blended with other strong lines and quickly
fades. Therefore, the non-detection of this feature in the optical spectra
should not be used to rule out the presence of the He i 10830 Å feature,
particularly later than the first week after explosion.

Due to Monte Carlo noise in our simulation around 2 $\mu$m we cannot confirm
whether a clear He i 2 $\mu$m feature would form for this model. However, we
do find He i contributions to the model spectra in this wavelength range
indicating that a He i 2 $\mu$m feature could potentially form that would be
strongest at early times and decrease in strength until $\sim 1$ week after
explosion.

In addition to our result that He i 10830 Å is a predicted spectral feature
for the double detonation scenario, we compare the simulated evolution of this
feature to observations of iPTF13ebh, which we suggest showed evidence of a He
i 10830 Å feature. A secondary component at the blue wing of the Mg ii
$\lambda 10927$ feature has previously been observed in transitional SNe Ia,
such as iPTF13ebh (Hsiao et al., 2015b), which showed a similar evolution to
that predicted by the He i 10830 Å feature in our model. However, this has
previously been identified as C i 10693 Å. We have compared our model spectra
to the observations of iPTF13ebh and found that He i 10830 Å can reproduce
this feature, and we suggest that this is evidence for He i in SNe Ia and for
the double detonation explosion scenario. This shows that He i may not always
be concealed in the spectra of SNe Ia. In order to potentially detect the He i
10830 Å in observations, and to identify this feature, early NIR observations
are required.

## Acknowledgements

CEC is grateful for support from the Department for the Economy (DfE) and
acknowledges support by the European Research Council (ERC) under the European
Union’s Horizon 2020 research and innovation program under grant agreement No.
759253. CEC and LJS acknowledge support by Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) - Project-ID 279384907 - SFB 1245 and MA
4248/3-1. The work of SAS was supported by the Science and Technology
Facilities Council [grant numbers ST/P000312/1, ST/T000198/1, ST/X00094X/1].
LJS acknowledges support by the European Research Council (ERC) under the
European Union’s Horizon 2020 research and innovation program (ERC Advanced
Grant KILONOVA No. 885281). The work of SG and FKR was supported by the
Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under
Project-ID 138713538, SFB 881 (“The Milky Way System”, Subproject A10), by the
ChETEC COST Action (CA16117), by the National Science Foundation under Grant
No. OISE-1927130 (IReNA), and by the Klaus Tschira Foundation. The authors
gratefully acknowledge the Gauss Centre for Supercomputing e.V. (www.gauss-
centre.eu) for funding this project by providing computing time on the GCS
Supercomputer JUWELS at Jülich Supercomputing Centre (JSC). This work was
performed using the Cambridge Service for Data Driven Discovery (CSD3), part
of which is operated by the University of Cambridge Research Computing on
behalf of the STFC DiRAC HPC Facility (www.dirac.ac.uk). The DiRAC component
of CSD3 was funded by BEIS capital funding via STFC capital grants
ST/P002307/1 and ST/R002452/1 and STFC operations grant ST/R00689X/1. DiRAC is
part of the National e-Infrastructure. CEC is grateful for computational
support by the VIRGO cluster at GSI. NumPy and SciPy (Oliphant, 2007),
Matplotlib (Hunter, 2007) and artistools111https://github.com/artis-
mcrt/artistools/(Shingles et al., 2023) were used for data processing and
plotting.

## Data Availability

The spectra presented here will be made available on the Heidelberg supernova
model archive hesma222https://hesma.h-its.org (Kromer et al., 2017).

## References

  * Bildsten et al. (2007) Bildsten L., Shen K. J., Weinberg N. N., Nelemans G., 2007, ApJ, 662, L95
  * Boos et al. (2021) Boos S. J., Townsley D. M., Shen K. J., Caldwell S., Miles B. J., 2021, ApJ, 919, 126
  * Boyle et al. (2017) Boyle A., Sim S. A., Hachinger S., Kerzendorf W., 2017, A&A, 599, A46
  * Bulla et al. (2015) Bulla M., Sim S. A., Kromer M., 2015, MNRAS, 450, 967
  * Chugai (1987) Chugai N. N., 1987, Soviet Astronomy Letters, 13, 282
  * Collins et al. (2022) Collins C. E., Gronow S., Sim S. A., Röpke F. K., 2022, MNRAS, 517, 5289
  * De et al. (2019) De K., et al., 2019, ApJ, 873, L18
  * Dessart & Hillier (2015) Dessart L., Hillier D. J., 2015, MNRAS, 447, 1370
  * Dong et al. (2022) Dong Y., et al., 2022, ApJ, 934, 102
  * Fink et al. (2010) Fink M., Röpke F. K., Hillebrandt W., Seitenzahl I. R., Sim S. A., Kromer M., 2010, A&A, 514, A53
  * Glasner et al. (2018) Glasner S. A., Livne E., Steinberg E., Yalinewich A., Truran J. W., 2018, Monthly Notices of the Royal Astronomical Society, 476, 2238
  * Gronow et al. (2020) Gronow S., Collins C., Ohlmann S. T., Pakmor R., Kromer M., Seitenzahl I. R., Sim S. A., Röpke F. K., 2020, A&A, 635, A169
  * Gronow et al. (2021) Gronow S., Collins C. E., Sim S. A., Röpke F. K., 2021, A&A, 649, A155
  * Hachinger et al. (2012) Hachinger S., Mazzali P. A., Taubenberger S., Hillebrand t W., Nomoto K., Sauer D. N., 2012, MNRAS, 422, 70
  * Heringer et al. (2019) Heringer E., van Kerkwijk M. H., Sim S. A., Kerzendorf W. E., Graham M. L., 2019, ApJ, 871, 250
  * Hillier (1990) Hillier D. J., 1990, A&A, 231, 111
  * Hillier & Miller (1998) Hillier D. J., Miller D. L., 1998, ApJ, 496, 407
  * Höflich & Khokhlov (1996) Höflich P., Khokhlov A., 1996, ApJ, 457, 500
  * Höflich et al. (2002) Höflich P., Gerardy C. L., Fesen R. A., Sakai S., 2002, ApJ, 568, 791
  * Hsiao et al. (2013) Hsiao E. Y., et al., 2013, ApJ, 766, 72
  * Hsiao et al. (2015a) Hsiao E. Y., et al., 2015a, VizieR Online Data Catalog: Optical and NIR spectra of SN iPTF13ebh, doi:10.26093/cds/vizier.35780009
  * Hsiao et al. (2015b) Hsiao E. Y., et al., 2015b, A&A, 578, A9
  * Hunter (2007) Hunter J. D., 2007, Computing in Science & Engineering, 9, 90
  * Jiang et al. (2017) Jiang J.-A., et al., 2017, Nature, 550, 80
  * Kozma & Fransson (1992) Kozma C., Fransson C., 1992, ApJ, 390, 602
  * Krisciunas et al. (2009) Krisciunas K., et al., 2009, AJ, 138, 1584
  * Kromer & Sim (2009) Kromer M., Sim S. A., 2009, MNRAS, 398, 1809
  * Kromer et al. (2010) Kromer M., Sim S. A., Fink M., Röpke F. K., Seitenzahl I. R., Hillebrandt W., 2010, ApJ, 719, 1067
  * Kromer et al. (2017) Kromer M., Ohlmann S., Röpke F. K., 2017, Mem. Soc. Astron. Italiana, 88, 312
  * Li et al. (2022) Li Z., et al., 2022, ApJ, 927, 142
  * Liu et al. (2023) Liu C., et al., 2023, ApJ, 946, 83
  * Livne (1990) Livne E., 1990, ApJ, 354, L53
  * Lucy (1991) Lucy L. B., 1991, ApJ, 383, 308
  * Lucy (2002) Lucy L. B., 2002, A&A, 384, 725
  * Lucy (2003) Lucy L. B., 2003, A&A, 403, 261
  * Lucy (2005) Lucy L. B., 2005, A&A, 429, 19
  * Marion et al. (2003) Marion G. H., Höflich P., Vacca W. D., Wheeler J. C., 2003, ApJ, 591, 316
  * Marion et al. (2009) Marion G. H., Höflich P., Gerardy C. L., Vacca W. D., Wheeler J. C., Robinson E. L., 2009, AJ, 138, 727
  * Marion et al. (2015) Marion G. H., et al., 2015, ApJ, 798, 39
  * Mazzali & Lucy (1998) Mazzali P. A., Lucy L. B., 1998, MNRAS, 295, 428
  * Mazzali et al. (2008) Mazzali P. A., Sauer D. N., Pastorello A., Benetti S., Hillebrandt W., 2008, MNRAS, 386, 1897
  * Meikle et al. (1996) Meikle W. P. S., et al., 1996, MNRAS, 281, 263
  * Noebauer et al. (2017) Noebauer U. M., Kromer M., Taubenberger S., Baklanov P., Blinnikov S., Sorokina E., Hillebrand t W., 2017, MNRAS, 472, 2787
  * Nomoto (1980) Nomoto K., 1980, Space Science Reviews, 27, 563
  * Nomoto (1982) Nomoto K., 1982, ApJ, 253, 798
  * Nugent et al. (1997) Nugent P., Baron E., Branch D., Fisher A., Hauschildt P. H., 1997, ApJ, 485, 812
  * Nugent et al. (2011) Nugent P. E., et al., 2011, Nature, 480, 344
  * Oliphant (2007) Oliphant T. E., 2007, Computing in Science & Engineering, 9, 10
  * Pakmor et al. (2022) Pakmor R., et al., 2022, MNRAS, 517, 5260
  * Pastorello et al. (2007) Pastorello A., et al., 2007, MNRAS, 377, 1531
  * Phillips et al. (1987) Phillips M. M., et al., 1987, PASP, 99, 592
  * Polin et al. (2019) Polin A., Nugent P., Kasen D., 2019, ApJ, 873, 84
  * Shen & Bildsten (2009) Shen K. J., Bildsten L., 2009, ApJ, 699, 1365
  * Shen & Moore (2014) Shen K. J., Moore K., 2014, ApJ, 797, 46
  * Shen et al. (2010) Shen K. J., Kasen D., Weinberg N. N., Bildsten L., Scannapieco E., 2010, ApJ, 715, 767
  * Shen et al. (2021) Shen K. J., Boos S. J., Townsley D. M., Kasen D., 2021, ApJ, 922, 68
  * Shingles et al. (2020) Shingles L. J., et al., 2020, MNRAS, 492, 2029
  * Shingles et al. (2023) Shingles L. J., Collins C. E., Holas A., Callan F., Sim S., 2023, artistools, doi:10.5281/zenodo.6418832, https://github.com/artis-mcrt/artistools
  * Sim (2007) Sim S. A., 2007, MNRAS, 375, 154
  * Taam (1980) Taam R. E., 1980, ApJ, 242, 749
  * Tanaka et al. (2008) Tanaka M., et al., 2008, ApJ, 677, 448
  * Teffs et al. (2020) Teffs J., Ertl T., Mazzali P., Hachinger S., Janka H. T., 2020, MNRAS, 499, 730
  * Townsley et al. (2019) Townsley D. M., Miles B. J., Shen K. J., Kasen D., 2019, ApJ, 878, L38
  * Waldman et al. (2011) Waldman R., Sauer D., Livne E., Perets H., Glasner A., Mazzali P., Truran J. W., Gal-Yam A., 2011, ApJ, 738, 21
  * Woosley & Kasen (2011) Woosley S. E., Kasen D., 2011, ApJ, 734, 38
  * Woosley & Weaver (1994) Woosley S. E., Weaver T. A., 1994, ApJ, 423, 371
  * Wyatt et al. (2021) Wyatt S. D., et al., 2021, ApJ, 914, 57


[1]
Richard Paul and Linda Elder.
Critical thinking: What, why, and how.
New directions for community colleges, 77(2):3–24, 1992.

[2]
Xuchao Zhang, Fanglan Chen, Chang-Tien Lu, and Naren Ramakrishnan.
Mitigating uncertainty in document classification.
In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 3126–3136. Association for Computational Linguistics, 2019.

[3]
Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto.
Multi-modal hallucination control by visual information grounding.
CoRR, abs/2403.14003, 2024.

[4]
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao.
Analyzing and mitigating object hallucination in large vision-language models.
CoRR, abs/2310.00754, 2023.

[5]
Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, Jitao Sang, and Haoyu Tang.
Evaluation and analysis of hallucination in large vision-language models.
CoRR, abs/2308.15126, 2023.

[6]
Neeraj Varshney and Chitta Baral.
Post-abstention: Towards reliably re-attempting the abstained instances in QA.
In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 967–982. Association for Computational Linguistics, 2023.

[7]
Ernest Davis.
Unanswerable questions about images and texts.
Frontiers Artif. Intell., 3:51, 2020.

[8]
Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa.
Unsolvable problem detection: Evaluating trustworthiness of vision language models.
CoRR, abs/2403.20331, 2024.

[9]
Aroma Mahendru, Viraj Prabhu, Akrit Mohapatra, Dhruv Batra, and Stefan Lee.
The promise of premise: Harnessing question premises in visual question answering.
In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 926–935. Association for Computational Linguistics, 2017.

[10]
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach.
Reliable visual question answering: Abstain rather than answer incorrectly.
In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision - ECCV 2022 - 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, volume 13696 of Lecture Notes in Computer Science, pages 148–166. Springer, 2022.

[11]
Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang.
An llm-free multi-dimensional benchmark for mllms hallucination evaluation.
CoRR, abs/2311.07397, 2023.

[12]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen.
Evaluating object hallucination in large vision-language models.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 292–305. Association for Computational Linguistics, 2023.

[13]
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.
MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI.
CoRR, abs/2311.16502, 2023.

[14]
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.
Making the V in VQA matter: Elevating the role of image understanding in visual question answering.
Int. J. Comput. Vis., 127(4):398–414, 2019.

[15]
GPT-4 technical report.
CoRR, abs/2303.08774, 2023.

[16]
Yanyang Guo, Fangkai Jiao, Zhiqi Shen, Liqiang Nie, and Mohan S. Kankanhalli.
UNK-VQA: A dataset and A probe into multi-modal large models' abstention ability.
CoRR, abs/2310.10942, 2023.

[17]
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht.
Obtaining well calibrated probabilities using bayesian binning.
In Blai Bonet and Sven Koenig, editors, Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2901–2907. AAAI Press, 2015.

[18]
Hanning Zhang, Shizhe Diao, Yong Lin, Yi Ren Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang.
R-tuning: Instructing large language models to say `i don't know'.

[19]
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.
Direct preference optimization: Your language model is secretly a reward model.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.

[20]
Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, and Jason Baldridge.
DOCCI: descriptions of connected and contrasting images.
CoRR, abs/2404.19753, 2024.

[21]
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang.
Grounded SAM: assembling open-world models for diverse visual tasks.
CoRR, abs/2401.14159, 2024.

[22]
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.
Resolution-robust large mask inpainting with fourier convolutions.
arXiv preprint arXiv:2109.07161, 2021.

[23]
Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi.
The generative AI paradox: "what it can create, it may not understand".
CoRR, abs/2311.00059, 2023.

[24]
Kushal Kafle and Christopher Kanan.
An analysis of visual question answering algorithms.
In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 1983–1991. IEEE Computer Society, 2017.

[25]
Oscar Mañas, Benno Krojer, and Aishwarya Agrawal.
Improving automatic VQA evaluation using large language models.
In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 4171–4179. AAAI Press, 2024.

[26]
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al.
Mistral 7b.
arXiv preprint arXiv:2310.06825, 2023.

[27]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, 2023.

[28]
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.

[29]
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond.

[30]
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang.
Mitigating hallucination in large multi-modal models via robust instruction tuning.
In The Twelfth International Conference on Learning Representations, 2023.

[31]
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell.
Aligning large multimodal models with factually augmented RLHF.
CoRR, abs/2309.14525, 2023.

[32]
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P. Bigham.
Vizwiz grand challenge: Answering visual questions from blind people.
In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3608–3617. Computer Vision Foundation / IEEE Computer Society, 2018.

[33]
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian.
Qa-lora: Quantization-aware low-rank adaptation of large language models.
CoRR, abs/2309.14717, 2023.

[34]
C.K. Chow.
An optimum character recognition system using decision functions.
IRE Transactions on Electronic Computers, 1957.

[35]
Claudio De Stefano, Carlo Sansone, and Mario Vento.
To reject or not to reject: that is the question-an answer in case of neural classifiers.
IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 30(1):84–94, 2000.

[36]
Ran El-Yaniv et al.
On the foundations of noise-free selective classification.
Journal of Machine Learning Research, 11(5), 2010.

[37]
Amita Kamath, Robin Jia, and Percy Liang.
Selective question answering under domain shift.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5684–5696. Association for Computational Linguistics, 2020.

[38]
Neeraj Varshney, Swaroop Mishra, and Chitta Baral.
Investigating selective prediction approaches across several tasks in iid, ood, and adversarial settings.
In Findings of the Association for Computational Linguistics (ACL), 2022.

[39]
Tejas Srinivasan, Jack Hessel, Tanmay Gupta, Bill Yuchen Lin, Yejin Choi, Jesse Thomason, and Khyathi Raghavi Chandu.
Selective "selective prediction": Reducing unnecessary abstention in vision-language reasoning.
CoRR, abs/2402.15610, 2024.

[40]
Andy Zeng, Maria Attarian, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, et al.
Socratic models: Composing zero-shot multimodal reasoning with language.
In International Conference on Learning Representations (ICLR), 2022.

[41]
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.
Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface.
arXiv preprint arXiv:2303.17580, 2023.

[42]
Haoxuan You, Rui Sun, Zhecan Wang, Long Chen, Gengyu Wang, Hammad A Ayyubi, Kai-Wei Chang, and Shih-Fu Chang.
Idealgpt: Iteratively decomposing vision and language reasoning via large language models.
arXiv preprint arXiv:2305.14985, 2023.

[43]
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang.
Mm-react: Prompting chatgpt for multimodal reasoning and action.
arXiv preprint arXiv:2303.11381, 2023.

[44]
Archiki Prasad, Elias Stengel-Eskin, and Mohit Bansal.
Rephrase, augment, reason: Visual grounding of questions for vision-language models.
arXiv preprint arXiv:2310.05861, 2023.

[45]
Tanmay Gupta and Aniruddha Kembhavi.
Visual programming: Compositional visual reasoning without training.
In Computer Vision and Pattern Recognition (CVPR), 2023.

[46]
Dídac Surís, Sachit Menon, and Carl Vondrick.
Vipergpt: Visual inference via python execution for reasoning.
Proceedings of IEEE International Conference on Computer Vision (ICCV), 2023.

[47]
Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, and Dan Klein.
Modular visual question answering via code generation.
arXiv preprint arXiv:2306.05392, 2023.

[48]
Zhiying Zhu, Zhiqing Sun, and Yiming Yang.
Halueval-wild: Evaluating hallucinations of language models in the wild.
CoRR, abs/2403.04307, 2024.

[49]
Haoqiang Kang, Juntong Ni, and Huaxiu Yao.
Ever: Mitigating hallucination in large language models through real-time verification and rectification.
CoRR, abs/2311.09114, 2023.

[50]
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen.
Woodpecker: Hallucination correction for multimodal large language models.
CoRR, abs/2310.16045, 2023.

[51]
Junliang Luo, Tianyu Li, Di Wu, Michael Jenkin, Steve Liu, and Gregory Dudek.
Hallucination detection and hallucination mitigation: An investigation.
CoRR, abs/2401.08358, 2024.

[52]
Tianyu Liu, Yizhe Zhang, Christopher John Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and William B. Dolan.
A token-level reference-free hallucination detection benchmark for free-form text generation.
ArXiv, abs/2104.08704, 2021.

[53]
Chunting Zhou, Jiatao Gu, Mona T. Diab, Paco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad.
Detecting hallucinated content in conditional neural sequence generation.
ArXiv, abs/2011.02593, 2020.

[54]
Nouha Dziri, Andrea Madotto, Osmar Zaiane, and A. Bose.
Neural path hunter: Reducing hallucination in dialogue systems via path grounding.
ArXiv, abs/2104.08455, 2021.

[55]
Mengyao Cao, Yue Dong, and Jackie Chi Kit Cheung.
Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.
In Annual Meeting of the Association for Computational Linguistics, 2021.

[56]
Potsawee Manakul, Adian Liusie, and Mark John Francis Gales.
Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.
ArXiv, abs/2303.08896, 2023.

[57]
Yuheng Zha, Yichi Yang, Ruichen Li, and Zhiting Hu.
Alignscore: Evaluating factual consistency with a unified alignment function.
In Annual Meeting of the Association for Computational Linguistics, 2023.

[58]
Jiaming Shen, Jialu Liu, Daniel Finnie, Negar Asgharipour Rahmati, Michael Bendersky, and Marc Najork.
“why is this misleading?”: Detecting news headline hallucinations with explanations.
Proceedings of the ACM Web Conference 2023, 2023.

[59]
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen.
Halueval: A large-scale hallucination evaluation benchmark for large language models.
ArXiv, abs/2305.11747, 2023.

[60]
Ziwei Ji, Zihan Liu, Nayeon Lee, Tiezheng Yu, Bryan Wilie, Mini Zeng, and Pascale Fung.
Rho ($$\rho$$): Reducing hallucination in open-domain dialogues with knowledge grounding.
In Annual Meeting of the Association for Computational Linguistics, 2022.

[61]
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston.
Retrieval augmentation reduces hallucination in conversation.
In Conference on Empirical Methods in Natural Language Processing, 2021.

[62]
Justin Johnson, Andrej Karpathy, and Li Fei-Fei.
Densecap: Fully convolutional localization networks for dense captioning.
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4565–4574, 2015.

[63]
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko.
Object hallucination in image captioning.
In Conference on Empirical Methods in Natural Language Processing, 2018.

[64]
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen.
Evaluating object hallucination in large vision-language models.
In Conference on Empirical Methods in Natural Language Processing, 2023.

[65]
Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung.
Negative object presence evaluation (nope) to measure object hallucination in vision-language models.
ArXiv, abs/2310.05338, 2023.

[66]
Anish Gunjal, Jihan Yin, and Erhan Bas.
Detecting and preventing hallucinations in large vision language models.
In AAAI Conference on Artificial Intelligence, 2023.

[67]
Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du.
FAITHSCORE: evaluating hallucinations in large vision-language models.
CoRR, abs/2311.01477, 2023.

[68]
Anisha Gunjal, Jihan Yin, and Erhan Bas.
Detecting and preventing hallucinations in large vision language models.
In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 18135–18143. AAAI Press, 2024.

[69]
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou.
Hallucination of multimodal large language models: A survey.
CoRR, abs/2404.18930, 2024.

[70]
Zhiyang Chen, Yousong Zhu, Yufei Zhan, Zhaowen Li, Chaoyang Zhao, Jinqiao Wang, and Ming Tang.
Mitigating hallucination in visual language models with visual supervision.
CoRR, abs/2311.16479, 2023.

[71]
Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim.
Mitigating fine-grained hallucination by fine-tuning large vision-language models with caption rewrites.
In Stevan Rudinac, Alan Hanjalic, Cynthia C. S. Liem, Marcel Worring, Bjorn Jonsson, Bei Liu, and Yoko Yamakata, editors, MultiMedia Modeling - 30th International Conference, MMM 2024, Amsterdam, The Netherlands, January 29 - February 2, 2024, Proceedings, Part IV, volume 14557 of Lecture Notes in Computer Science, pages 32–45. Springer, 2024.

[72]
Zhiying Zhu, Zhiqing Sun, and Yiming Yang.
Halueval-wild: Evaluating hallucinations of language models in the wild.
ArXiv, abs/2403.04307, 2024.

[73]
Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi.
Fine-grained hallucination detection and editing for language models.
ArXiv, abs/2401.06855, 2024.

[74]
Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang.
Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data.
CoRR, abs/2311.13614, 2023.

[75]
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong.
Silkie: Preference distillation for large visual language models.
CoRR, abs/2312.10665, 2023.

[76]
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.
Yi: Open foundation models by 01.ai, 2024.

[77]
Microsoft Azure.

[78]
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.
Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.
In KDD, 2020.

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
All the claims made in the abstract and introduction are supported by the benchmark dataset, a new metric, and empirical evaluation.

* Did you describe the limitations of your work?
We discuss limitations in Appendix <ref>.

* Did you discuss any potential negative societal impacts of your work?
We do not see significant negative societal impacts from .

* Have you read the ethics review guidelines and ensured that your paper conforms to them?
We have reviewed the Code of Ethics.

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?
The paper does not include theoretical results.

* Did you include complete proofs of all theoretical results?

* If you ran experiments (e.g. for benchmarks)...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
We will provide code, data samples and instructions in supplementary material.

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Training details are described in <ref>

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
We did not include error bars as the experiments are computationally expensive, and existing literature does not have the convention to report error bras.

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Computing resource details are described in <ref>.

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?
We properly cited all code, data, and models used in this paper.

* Did you mention the license of the assets?
We provide MIT License software to be used with our data.

* Did you include any new assets either in the supplemental material or as a URL?
We provide data samples of our dataset in supplementary material.

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?
Doesn't apply to this work since we sourced data from publicly available datasets such as COCO and DOCCI.

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
The publicly available datasets from which our data is constructed do not have any PII instances and are devoid of toxic content.

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?

We didn't conduct research with human subjects.

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
We didn't conduct research with human subjects.

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

§ LIMITATIONS

While our covers various categories of multimodal uncertainty, and showed improvements over the base model when finetuned with it, there are potential limitations to be acknowledged.
Though our synthetic data is rigorously quality-checked,
it is possible that the synthetic generation pipeline may not capture all the nuances of real-world uncertain scenarios. Additionally, the most effective way to improve model performance on our benchmark currently is SFT with LoRA, which is more resource-intensive compared to techniques such as selective prediction that makes decisions based on the prediction probabilities during inference. Moreover, providing a reasonable or best guess based on existing knowledge can be more suitable than either answering or abstaining, which we leave as future work.

§ BROADER IMPACT

Current models are incentivized to predict definitive answers even in uncertain scenarios. This can lead to outputs with unwarranted confidence, which is particularly problematic in high-stakes applications such as medical diagnosis or financial forecasting. This tendency can result in misleading information and erroneous decisions. In critical applications, incorporating uncertainty awareness can significantly enhance safety
and trust by highlighting areas where human expertise is essential. Our proposed taxonomy and data creation pipeline can be adapted to various scenarios, provided domain-specific inpainting techniques are available. Additionally, when models are trained with , it can facilitate more efficient resource allocation, as models can identify when additional data or analysis is required, ultimately leading to more robust and trustworthy models. Specifically, identifying the category of epistemic and aleatoric awareness from can help identify better means to tackle the uncertainty. Finally, our confidence-weighted metric allows for comprehensive performance evaluation across a wide range of domains, encompassing both unimodal and multimodal scenarios.

§ SAMPLES VISUALIZING BENCHMARK

We visualize some samples from each fine-grained category of the epistemic and aleatoric awareness. For the category of extraneous, our data is made of samples where the answer differs for the same question when the image is perturbed. For the rest of the categories, the dataset contains samples where the same image is paired with answerable and unanswerable questions.

Figure <ref> shows the category of knowledge awareness; as we can see the unanswerable questions ask about information that is hard to identify from the context of the image and requires additional knowledge.
Similarly, Figure <ref> shows examples from the complexity awareness in the epistemic category. The unanswerable questions are too tedious to arrive at an answer while the answerable questions still require some efforts, such as counting but is not laborious to answer.

figureSamples from Knowledge Awareness
(Epistemic) category

figureSamples from Complexity Awareness
(Epistemic) category

For the extraneous sub-category of the epistemic awareness, we perturb the image to mask and remove the target object about which the question seeks information. Samples from this category are shown in Fig <ref>. As we can see the target objects in the questions are `cat' and `statue'. These objects are removed from the image and inpainted to get a natural-looking image to obtain a perturbed image. The resulting image paired with the same question now becomes unanswerable. The answerable question is paired with the original unperturbed question to have a definitive answer (which is the standard setup for most VQA based benchmarks).

figureSamples from Extraneous Awareness (Epistemic) category

Figures <ref> and <ref>
show samples from in the aleatoric category, particularly of the temporal and ambiguity awareness sub-categories respectively. The temporal sub-category, as we can see, contains questions about the current happenings or state of the image for the answerable part and the unanswerable questions ask about the future that is not directly predictable from the current state.
The ambiguous awareness category contains questions with a definitive answer for the answerable type and questions with many plausible answers but cannot choose a single definite answer for the unanswerable type.

figureSamples from Temporal Awareness
(Aleatoric) category

figureSamples from Ambiguous Awareness
(Aleatoric) category

figureVisualization of model predictions and the corresponding $\text{LAVE}_{\text{idk}}$ accuracy, $P_{\text{pred}}$ and confidence-weighted accuracy. The base model here is Qwen-VL-Chat [29]. Our confidence-weighted accuracy is represented as CW Acc in this figure.

§ SAMPLES VISUALIZING PREDICTIONS AND CONFIDENCE-WEIGHTED METRIC

Our proposed confidence-weighted accuracy takes into account the prediction probability and the correctness of the predicted answer to give a holistic score.
Figure <ref> presents the visualization of model predictions and the corresponding $\text{LAVE}_{\text{idk}}$ accuracy, $P_{\text{pred}}$ and confidence-weighted accuracy. We show that the proposed confidence-weighted accuracy gives less score for a correct answer with lower confidence, and penalizes more for an incorrect answer with higher confidence.
In addition, our visualization shows that Qwen-VL-Chat [29] is able to say equivalents of “I don't know" more confidently from (a) and (b), after continued finetuning on our data with SFT-LoRA.

Examples (a) and (b) show cases where the base model is less confident for a correct answer. Our metric gives a partial score for the correctness owing to the prediction probability. After finetuning, as the prediction probability of the correct answer increases, our confidence-weighted accuracy increases accordingly. In case (c), the base model predicts an incorrect answer with high confidence. Our metric penalizes this more heavily with a high negative score. After finetuning, the prediction is rectified and the scores are adjusted accordingly. In the case of (d), the base model predicts the incorrect answer but with low confidence. Our metric still gives a negative score but penalizes less compared to (c). The cases of (c) and (d) differentiate answering incorrectly with high and low probabilities respectively.

Moreover, after finetuning with our , we see the corrected predictions with relatively higher probabilities for correctness, which are reflected in our confidence-weighted metric score. These probabilities of the model predictions are not reflected in the $\text{LAVE}_{\text{idk}}$ accuracy.

§ ADDITIONAL RESULTS

LLaVA with LoRA-SFT. We include results with LoRA-SFT on LLaVA-v1.5-7b in Table <ref>, which show consistent performance improvement when trained with our data.

Comparing 7B to 13B models.

We conduct experiments to study the performance of a larger model across different uncertainty awareness categories. These results are presented in Table <ref>.

We observe consistent performance improvements over LLaVA-1.5-7B-LoRA and LLaVA-1.5-13B-LoRA [27] with the augmentation of during the instruction-tuning phase.  When instruction-tuned with only our data (i.e., Ours-only), compared to the results on the 7B-LoRA model, a larger model 13B-LoRA only marginally improves on confidence-weighted accuracy and ECE (IDK). However, when mixing our data with LLaVA instruction tuning data (i.e., Ours+LLaVA Data), the resulting 13B model clearly outperforms 7B on both metrics.

In addition, we observe that the model performance on $\text{LAVE}_{\text{idk}}$ metrics stay on par for 7B and 13B models with the same training data,

while they can still be differentiated by our proposed metric, which further highlights the importance of confidence-weighted accuracy.

LAVE IDK Judgement Accuracy As our metric relies on the accuracy of LAVE refusal judgment, we experiment with different LLMs on 100 samples from extraneous split. Table <ref> presents the results with Mistral-7B [26], Yi-34B-4bits [76] and GPT4, in comparison with Human. Given the high performance, and also considering the latency and cost of model inference, we decide to use the smaller model Mistral-7B in our evaluation.

§ IMPLEMENTATION DETAILS

For Thresholding baselines, we perform grid search among $(0.1, 0.2, ... 0.9)$ and $(0.91, 0.92, ... 0.99)$ to decide the optimal threshold for each split. The latter range is included, as we observe that the models are often over-confident in their own predictions.

For SFT/Instruction-tuning with LoRA, we follow the instructions provided by Qwen-VL and LLaVA official implementations, with exactly the same setting of learning rate and LoRA configurations.  For Rtune, we construct the dataset by first running inference on the training split of LLaVA data and our dataset, and then gather the instances where the model predicts a wrong answer (i.e., receives a LAVE accuracy of 0). With the constructed dataset, we tune Qwen-VL with the same training configuration as SFT. For DPO, we follow the implementations of Silkie [75].

All experiments are conducted with V100s on Microsoft Azure [77], adopting mixed-precision training with DeepSpeed [78] stage 3. To match the batch size suggested in official implementations, we train the models on 64 V100s for 1 epoch with a batch size of 2 per GPU.

For evaluation on Vizwiz, we first use LAVE refusal prompt to judge whether the prediction is IDK. If so, we convert the answer to “unanswerable” and use the standard VQA-based VizWiz evaluation.

§ ADDITIONAL DETAILS ON DATA CREATION

§.§ More Details on sourcing from image

The masks of salient objects are generated by Grounded-SAM [21] with boxthreshold of 0.3 and textthreshold of 0.25. The mask is dilated with kernel size 20 and then input to LaMa inpainting model [22] to remove the object.

For VQA images, we use GPT-4 to first identify the salient objects given the question-answer pairs, which will use as text queries to Grounded-SAM.

For GQA images, we identify objects in the scene graphs that is associated with a question as the salient object. Then we traverse the scene graphs to find all other objects with the same label. Since GQA also offers groundtruth bounding box (bbox) annotations, we use the mask generated by Grounded-SAM from GT bbox, following by inpainting to remove all such objects. In this way, the same question becomes unanswerable for the perturbed image, and we replace the answer with IDK answers by randomly sample from (1) “I don't know."; (2) “I don't see any [Object]."; (3) “There is no [Object] in the image."; and (4) “I can't see any [Object].".

§.§ Prompts for Data Creation

r submission must satisfy.

The only supported style file for NeurIPS 2024 is ,
written for . There are no supported style sheets for Microsoft Word, RTF, or other formats. The  style file contains three optional arguments: , which
creates a camera-ready copy, , which creates a preprint for
submission to, e.g., arXiv, and , which will not load the
package for you in case of package clash.

Preprint option
If you wish to post a preprint of your work online, e.g., on arXiv, using the
NeurIPS style, please use the  option. This will create a version of your work with the text “Preprint. Work in progress.”
in the footer. This version may be distributed as you see fit. Please do
not use the  option, which should only be used for
papers accepted to the NeurIPS Track on Datasets and Benchmarks.

At submission time, please omit the  and
options. This will add line numbers to aid
review. Please do not refer to these line numbers in your paper as they
will be removed during generation of camera-ready copies. Note that submissions to the NeurIPS Track on Datasets and Benchmarks are reviewed in a single-blind fashion and therefore not anonymous. This is because datasets can typically not be shared in a non-anonymous way. If you feel strongly that your work should be submitted anonymously, please use the  option. This will create a version of your work with all author names hidden.

The file  may be used as a “shell” for writing your
paper. All you have to do is replace the author, title, abstract, and text of
the paper with your own.

The formatting instructions contained in these style files are summarized in
Sections <ref>, <ref>, and <ref> below.

§ GENERAL FORMATTING INSTRUCTIONS

The text must be confined within a rectangle 5.5 inches (33 picas) wide and
9 inches (54 picas) long. The left margin is 1.5 inch (9 picas).  Use 10 point
type with a vertical spacing (leading) of 11 points.  Times New Roman is the
preferred typeface throughout, and will be selected for you by default.
Paragraphs are separated by 1/2 line space (5.5 points), with no

The paper title should be 17 point, initial caps/lower case, bold, centered
between two horizontal rules. The top rule should be 4 points thick and the
bottom rule should be 1 point thick. Allow 1/4 inch space above and
below the title to rules. All pages should start at 1 inch (6 picas) from the
top of the page.

Authors' names are set in boldface, and each name is
centered above the corresponding address. The lead author's name is to be listed
first (left-most), and the co-authors' names (if different address) are set to
follow. If there is only one co-author, list both author and co-author side by

Please pay special attention to the instructions in Section <ref>
regarding figures, tables, acknowledgments, and references.

§ HEADINGS: FIRST LEVEL

All headings should be lower case (except for first word and proper nouns),
flush left, and bold.

First-level headings should be in 12-point type.

§.§ Headings: second level

Second-level headings should be in 10-point type.

§.§.§ Headings: third level

Third-level headings should be in 10-point type.

There is also a  command available, which sets the heading in
bold, flush left, and inline with the text, with the heading followed by 1 em
of space.

§ CITATIONS, FIGURES, TABLES, REFERENCES

These instructions apply to everyone.

§.§ Citations within the text

The  package will be loaded for you by default.  Citations may be
author/year or numeric, as long as you maintain internal consistency.  As to the
format of the references themselves, any style is acceptable as long as it is
used consistently.

The documentation for  may be found at

Of note is the command , which produces citations appropriate for
use in inline text.  For example,

Hasselmo, et al. (1995) investigated…

If you wish to load the  package with options, you may add the
following before loading the  package:

If  clashes with another package you load, you can add the optional
argument  when loading the style file:

As submission is double blind, refer to your own published work in the third
person. That is, use “In the previous work of Jones et al. [4],” not “In our
previous work [4].” If you cite your other papers that are not widely available
(e.g., a journal paper under review), use anonymous author names in the
citation, e.g., an author of the form “A. Anonymous.”

§.§ Footnotes

Footnotes should be used sparingly.  If you do require a footnote, indicate
footnotes with a number[Sample of the first footnote.] in the
text. Place the footnotes at the bottom of the page on which they appear.
Precede the footnote with a horizontal rule of 2 inches (12 picas).

Note that footnotes are properly typeset after punctuation
marks.[As in this example.]

§.§ Figures

[-.5cm]0cm4cm [-.5cm]4cm0cm
Sample figure caption.

All artwork must be neat, clean, and legible. Lines should be dark enough for
purposes of reproduction. The figure number and caption always appear after the
figure. Place one line space before the figure caption and one line space after
the figure. The figure caption should be lower case (except for first word and
proper nouns); figures are numbered consecutively.

You may use color figures.  However, it is best for the figure captions and the
paper body to be legible if the paper is printed in either black/white or in

§.§ Tables

All tables must be centered, neat, clean and legible.  The table number and
title always appear before the table.  See Table <ref>.

Place one line space before the table title, one line space after the
table title, and one line space after the table. The table title must
be lower case (except for first word and proper nouns); tables are
numbered consecutively.

Note that publication-quality tables do not contain vertical rules. We
strongly suggest the use of the  package, which allows for
typesetting high-quality, professional tables:

This package was used to typeset Table <ref>.

Sample table title

Name         Description         Size ($\mu$m)

Dendrite     Input terminal      $\sim$100

Axon         Output terminal     $\sim$10

Soma         Cell body           up to $10^6$

§ FINAL INSTRUCTIONS

Do not change any aspects of the formatting parameters in the style files.  In
particular, do not modify the width or length of the rectangle the text should
fit into, and do not change font sizes (except perhaps in the
References section; see below). Please note that pages should be

§ PREPARING PDF FILES

Please prepare submission files with paper size “US Letter,” and not, for
example, “A4.”

Fonts were the main cause of problems in the past years. Your PDF file must only
contain Type 1 or Embedded TrueType fonts. Here are a few instructions to
achieve this.

* You should directly generate PDF files using .

* You can check which fonts a PDF files uses.  In Acrobat Reader, select the
menu Files$>$Document Properties$>$Fonts and select Show All Fonts. You can
also use the program  which comes with  and is
available out-of-the-box on most Linux machines.

* The IEEE has recommendations for generating PDF files whose fonts are also
acceptable for NeurIPS. Please see

*  "patterned" shapes are implemented with bitmap fonts.  Use
"solid" shapes instead.

* The  package almost always uses bitmap fonts.  You should use
the equivalent AMS Fonts:

followed by, e.g., , , or
for $\mathbb{R}$, $\mathbb{N}$ or $\mathbb{C}$.  You can also use the following
workaround for reals, natural and complex:

Note that  is automatically loaded by the  package.

If your file contains type 3 fonts or non embedded TrueType fonts, we will ask
you to fix it.

§.§ Margins in

Most of the margin problems come from figures positioned by hand using
or other commands. We suggest using the command
from the  package. Always specify the
figure width as a multiple of the line width as in the example below:

See Section 4.4 in the graphics bundle documentation

A number of width problems arise when  cannot properly hyphenate a
line. Please give LaTeX hyphenation hints using the  command when

§.§ Suppressing line numbers in supplementary materials

If you need to suppress line numbers in the supplementary materials because
they interfere with the text, for instance because you are including a
data sheet in 2-column format, you can do so by
placing the following command before it:

Use unnumbered first level headings for the acknowledgments. All acknowledgments
go at the end of the paper before the list of references. Moreover, you are required to declare
funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
More information about this disclosure can be found at: <https://neurips.cc/Conferences/2024/PaperInformation/FundingDisclosure>.
You can use the  environment provided in the style file. As opposed to the main NeurIPS track, acknowledgements do not need to be hidden.

§ REFERENCES

References follow the acknowledgments. Use unnumbered first-level heading for
the references. Any choice of citation style is acceptable as long as you are
consistent. It is permissible to reduce the font size to  (9 point)
when listing the references.
Note that the Reference section does not count towards the page limit.

[1] Alexander, J.A. & Mozer, M.C. (1995) Template-based algorithms for
connectionist rule extraction. In G. Tesauro, D.S. Touretzky and T.K. Leen
(eds.), Advances in Neural Information Processing Systems 7,
pp. 609–616. Cambridge, MA: MIT Press.

[2] Bower, J.M. & Beeman, D. (1995) The Book of GENESIS: Exploring
Realistic Neural Models with the GEneral NEural SImulation System.  New York:

[3] Hasselmo, M.E., Schnell, E. & Barkai, E. (1995) Dynamics of learning and
recall at excitatory recurrent synapses and cholinergic modulation in rat
hippocampal region CA3. Journal of Neuroscience 15(7):5249-5262.

§ CHECKLIST

The checklist follows the references.  Please
read the checklist guidelines carefully for information on how to answer these
questions.  For each question, change the default  to ,
, or .  You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of
your paper or providing a brief inline description.  For example:

* Did you include the license to the code and datasets? See Section <ref>.

* Did you include the license to the code and datasets? The code and the data are proprietary.

* Did you include the license to the code and datasets?

Please do not modify the questions and only use the provided macros for your
answers.  Note that the Checklist section does not count towards the page
limit.  In your paper, please delete this instructions block and only keep the
Checklist section heading above along with the questions/answers below.

* For all authors...

* Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

* Did you describe the limitations of your work?

* Did you discuss any potential negative societal impacts of your work?

* Have you read the ethics review guidelines and ensured that your paper conforms to them?

* If you are including theoretical results...

* Did you state the full set of assumptions of all theoretical results?

* Did you include complete proofs of all theoretical results?

* If you ran experiments (e.g. for benchmarks)...

* Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

* Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

* Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

* Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

* If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

* If your work uses existing assets, did you cite the creators?

* Did you mention the license of the assets?

* Did you include any new assets either in the supplemental material or as a URL?

* Did you discuss whether and how consent was obtained from people whose data you're using/curating?

* Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

* If you used crowdsourcing or conducted research with human subjects...

* Did you include the full text of instructions given to participants and screenshots, if applicable?

* Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

* Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

§ APPENDIX

Include extra information in the appendix. This section will often be part of the supplemental material. Please see the call on the NeurIPS website for links to additional guides on dataset publication.

* Submission introducing new datasets must include the following in the supplementary materials:

* Dataset documentation and intended uses. Recommended documentation frameworks include datasheets for datasets, dataset nutrition labels, data statements for NLP, and accountability frameworks.

* URL to website/platform where the dataset/benchmark can be viewed and downloaded by the reviewers.

* URL to Croissant metadata record documenting the dataset/benchmark available for viewing and downloading by the reviewers. You can create your Croissant metadata using e.g. the Python library available here: https://github.com/mlcommons/croissant

* Author statement that they bear all responsibility in case of violation of rights, etc., and confirmation of the data license.

* Hosting, licensing, and maintenance plan. The choice of hosting platform is yours, as long as you ensure access to the data (possibly through a curated interface) and will provide the necessary maintenance.

* To ensure accessibility, the supplementary materials for datasets must include the following:

* Links to access the dataset and its metadata. This can be hidden upon submission if the dataset is not yet publicly available but must be added in the camera-ready version. In select cases, e.g when the data can only be released at a later date, this can be added afterward. Simulation environments should link to (open source) code repositories.

* The dataset itself should ideally use an open and widely used data format. Provide a detailed explanation on how the dataset can be read. For simulation environments, use existing frameworks or explain how they can be used.

* Long-term preservation: It must be clear that the dataset will be available for a long time, either by uploading to a data repository or by explaining how the authors themselves will ensure this.

* Explicit license: Authors must choose a license, ideally a CC license for datasets, or an open source license for code (e.g. RL environments).

* Add structured metadata to a dataset's meta-data page using Web standards (like schema.org and DCAT): This allows it to be discovered and organized by anyone. If you use an existing data repository, this is often done automatically.

* Highly recommended: a persistent dereferenceable identifier (e.g. a DOI minted by a data repository or a prefix on identifiers.org) for datasets, or a code repository (e.g. GitHub, GitLab,...) for code. If this is not possible or useful, please explain why.

* For benchmarks, the supplementary materials must ensure that all results are easily reproducible. Where possible, use a reproducibility framework such as the ML reproducibility checklist, or otherwise guarantee that all results can be easily reproduced, i.e. all necessary datasets, code, and evaluation procedures must be accessible and documented.

* For papers introducing best practices in creating or curating datasets and benchmarks, the above supplementary materials are not required.