# Optimal observing strategies for velocity-suppressed dark matter
annihilation

Nolan Smyth<EMAIL_ADDRESS>Gabriela Huckabee<EMAIL_ADDRESS>Stefano
Profumo<EMAIL_ADDRESS>Department of Physics, University of California,
Santa Cruz, 1156 High Street, Santa Cruz, CA 95064, USA  Santa Cruz Institute
for Particle Physics, 1156 High Street, Santa Cruz, CA 95064, USA

(August 27, 2024)

###### Abstract

Numerous particle models for the cosmological dark matter feature a pair-
annihilation rate that scales with powers of the relative velocity between the
annihilating particles. As a result, the annihilation rate in the central
regions of a dark matter halo can be significantly lower than at the halo’s
periphery for particular ambient gravitational potentials. While this might be
offset by an increasing dark matter pair number density in the inner halo, it
raises the question: what angular region for dark matter models with velocity-
suppressed annihilation rates optimizes signal-to-noise? Here, we consider
simplified background models for galactic and extragalactic targets and
demonstrate that the optimal observing strategy varies greatly case-by-case.
Generally, a bright central source warrants an annular region of interest,
while a flatter background warrants as large as possible an angular region,
possibly including the central regions.

††preprint: APS/123-QED

##  I Introduction

The production of Standard Model particles from the annihilation of dark
matter (DM) particles provides a detectable signal with which to indirectly
probe DM models (for reviews, see e.g. [1, 2, 3]). Dwarf spheroidal galaxies
(dSphs) are of particular interest in indirect detection searches as they are
generally DM-dominated and have a low astrophysical foreground [4]. Other
prime targets are the Galactic center region and nearby galaxies such as M31
[5, 6] and M87 [7, 2, 3].

The total photon flux from a specific astrophysical object due to DM
annihilation is proportional to that object’s J-factor, also sometimes called
the astrophysical factor. The J-factor is determined by the astrophysical
properties of the object, such as its DM density profile. In calculating the
predicted photon flux from DM annihilation, often one assumes a velocity-
independent cross section, which, in turn, implies a velocity-independent
J-factor. However, in cases where the velocity-independent $s$-wave channel is
subdominant to the velocity-dependent $p$-wave and $d$-wave channels, the
J-factor must effectively include the velocity-dependent contribution from the
cross section. As shown by [8], in the general case of a velocity-dependent
cross section, the J-factor scales as the moments of the DM velocity
distribution. They show that even in this general case, the J-factor is
strongly correlated with the DM density and weakly correlated with the
velocity dispersion.

However, we show in this work that for simple cases of $p$-wave and $d$-wave
dominated annihilation, the velocity-dependent J-factor implies that the
optimal signal-to-noise ratio is generally achieved for a different
observation strategy than in the velocity-independent, $s$-wave dominated
case. For example, we find an annular field of view to be superior to a disk
when the background gamma-ray signal is sharply peaked at the center of a DM
halo. This is due to two factors. First, the velocity distribution of DM
particles is peaked off-center in the halo, resulting in a boost to the
velocity-dependent J-factor in this region. Second, the benefits of including
the high-density region at the center of the halo in the J-factor are
outweighed by the detriment of a large background.

In this study we consider two broad classes of observational targets: (1) a
generic extragalactic target and (2) the Milky Way Galactic Center. The latter
has been the subject of much debate due to the extended gamma-ray excess
within the inner region of the bulge. Annihilating dark matter may be
responsible for this excess and is the focus of this paper, but unresolved
gamma-ray pulsars and other faint baryonic sources have also been put forth as
viable explanations [9, 10, 11, 12]. The extragalactic class, which includes
nearby galaxies such as M31, M87, and M33, is of interest because the central
bulge and stellar disk are resolvable as two distinct components, something
that is not possible in the Milky Way Galactic Center due to bright disk
contamination. We include in this category local dwarf spheroidal galaxies
(dSphs), satellites of the Milky Way, as the angular extent of these
satellites is quite similar to nearby galaxies in the Local Group. This set of
nearby galaxies and satellites has ideal conditions for the observation of DM
annihilation and has been the subject of extensive theoretical and
observational study [13, 14, 15, 16].

This paper is organized as follows. In sec. II, we summarize different
scenarios in which $p$-wave and $d$-wave annihilation processes are dominant
or comparable to $s$-wave channels. In sec. III we detail our approach and
methodology. In sec. IV, we present our findings. Lastly, we discuss the
implications of our results in sec. V.

##  II Velocity-Dependent Cross Sections

We are interested in models in which the DM annihilation cross section has a
non-trivial velocity dependence. In a model-independent approach, we
parameterize this dependence as

$\langle\sigma_{A}v_{\mathrm{rel}}\rangle=\langle\sigma_{A}v_{\mathrm{rel}}\rangle_{0}\Big{(}\frac{v_{\mathrm{rel}}}{c}\Big{)}^{n}.$
(1)

where $\sigma_{A}$ is the annihilation cross section and
$\langle\sigma_{A}v_{\mathrm{rel}}\rangle_{0}$ is the velocity-independent
piece of the thermally averaged cross section.

We aim to keep our approach as general as possible, but we will primarily
consider three cases for illustrative purposes:

  1. 1.

$n=0$: This is the $s$-wave channel, which is the typical case of a velocity-
independent cross section;

  2. 2.

$n=2$: The $p$-wave channel, which is relevant in particular models such as
Majorana fermion DM annihilating into fermion/anti-fermion pairs. In this
scenario, the $s$-wave channel is chirality-suppressed and so $p$-wave
annihilation is dominant [17];

  3. 3.

$n=4$: This is the $d$-wave channel, which becomes relevant in the case of
real scalar singlet DM annihilating into lepton/anti-lepton pairs [18]. Again,
the $s$-wave channel is chirality suppressed and now the $p$-wave channel
requires a $CP$-odd bilinear involving two real scalars, for which there is no
such operator. Similarly, for the case of scalar DM annihilation into a pair
of massless gauge bosons, the cross section is also $d$-wave suppressed [19].

Sommerfeld enhancement ($n=-1$) is also a well-motivated case, but will tend
to enhance the J-factor towards the center of a DM halo where the typical
velocity is lower. We do not expect this case to yield a significantly
different optimal observing strategy from the benchmark $s$-wave case. As
such, we will exclusively focus on the 3 aforementioned cases.

##  III Formalism

###  III.1 Velocity-Dependent J-Factor

The dark matter annihilation signal is proportional to the square of the DM
density integrated over the line-of-sight. The literature refers to this
integral as the J-factor, also called the astrophysical factor. In the
velocity-independent case, it can be understood as a measure of how many DM
pairs exist between an observer and their observation target. In the more
general case, a velocity dependence can appear through the cross section, and
the appropriate J-factor, which we indicate with the symbol $\mathcal{J}$, is
defined as

$\mathcal{J}=\int
dl\frac{\langle\sigma_{A}v_{\mathrm{rel}}\rangle}{(\sigma_{A}v_{\mathrm{rel}})_{0}}\rho(r)^{2},$
(2)

where $\rho$ is the DM density. Parameterizing the velocity-dependent
annihilation cross section as in (1), the J-factor scales as the moments of
the dark matter velocity [8].

$\mathcal{J}=\int dl\rho(r)^{2}\Big{(}\frac{\mu_{n}(r)}{c^{n}}\Big{)},$ (3)

where $\mu_{n}=\int
d^{3}v_{\mathrm{rel}}f(v_{\mathrm{rel}})v_{\mathrm{rel}}^{n}$ is the n-th
moment of the velocity distribution.

We assume that the dark matter follows a Maxwell-Boltzmann distribution

$f(\mathbf{v})\propto(\sigma_{v}^{2})^{-3/2}e^{-\mathbf{v^{2}}/\sigma_{v}^{2}},$
(4)

where the velocity dispersion is $\sigma_{v}^{2}=\frac{\langle
v^{2}\rangle}{3}$ from the equipartition theorem and we take $\langle
v^{2}\rangle$ to be the square of the circular velocity at a given radius,

$v_{c}(r)=\sqrt{\frac{2GM(<r)}{r}}.$ (5)

The average relative velocity of two dark matter particles of identical mass,
assuming uncorrelated velocities, is then given by

$\langle
v_{\mathrm{rel}}^{2}\rangle=\langle(\mathbf{v}-\mathbf{v}^{\prime})^{2}\rangle=\int
d^{3}\mathbf{v}\int
d^{3}\mathbf{v}^{\prime}(\mathbf{v}-\mathbf{v}^{\prime})^{2}f(\mathbf{v})f(\mathbf{v}^{\prime})\\\
=2\langle v^{2}\rangle=2v_{c}^{2}$ (6)

For the purposes of this paper, we will consider two cases for the DM density:
a “cuspy” Navarro-Frenk-White (NFW) profile and a “cored” Burkert profile [20,
21, 22]. Note that once we specify the density profile, the velocity is fully
determined in this simple model, in which we neglect the gravitational
potential of baryons. The NFW profile is given by [20]

$\rho(r)=\frac{\rho_{0}}{\left(\frac{r}{r_{s}}\right)\left(1+\frac{r}{r_{s}}\right)^{2}}$
(7)

The mass enclosed in a sphere of radius $r$ centered on a halo with an NFW
density profile is

$M(<r)=4\pi\rho_{0}r_{s}^{3}\left(\frac{r_{s}}{r_{s}+r}-1+\log\left({1+\frac{r}{r_{s}}}\right)\right).$
(8)

The calculation for the Burkert profile is analogous. Plugging (8) into (5),
we have everything we need to calculate the velocity-dependent J-factor.

###  III.2 J-factor Calculation

Sol$r_{d}$$l$$b$EG$r$ Figure 1: Geometric parameters used in J-factor
calculation. EG is a generic extragalactic body that hosts DM; $r_{d}$ is the
displacement between our star, Sol, and the center of EG; $l$ is the
displacment between Sol and a point in EG’s DM halo; $b$ is the angle made
between $r_{d}$ and $l$; and $r$ is the radial distance from the center of EG.

Assuming a particular density profile, the DM velocity is known at all
locations under the assumptions outlined in sec. III.1. We can therefore
perform the line-of-sight integral to calculate the J-factor per unit solid
angle as a function of the angle $b$, which is measured with respect to the
center of the target halo as shown in Fig. 1.

Expressing the distance to the center of a DM halo in terms of the distance
from the solar neighborhood, we have

$r=\sqrt{r_{d}^{2}-2lr_{d}\cos(b)+l^{2}},$ (9)

where $r_{d}$ is the distance between the Sun and the center of the DM halo
and $l$ is the distance from the Sun and the point of evaluation. We then
calculate the J-factor for a range of discrete values for $b$ from $0$ out to
a maximum angle $b_{\mathrm{max}}=\arctan(\frac{r_{t}}{r_{d}})$, corresponding
to a line tangent to the edge of the halo as defined by the tidal radius
$r_{t}$. We interpolate between these points to get $\mathcal{J}(b)$.

Because the ratio $r_{s}/r_{d}$ is nearly identical for M31, M87, and dSphs
such as Draco, the angular scale of these halos is quite similar. We will
therefore categorize all such halos as an extragalactic (EG) source case
study. Note that we do not necessarily expect the shape of the DM distribution
to be identical in each of these halos, but it is not necessary to separate
these cases further for the illustrative purposes of this work. Thus for the
EG case, we use model values corresponding to the dSph Draco:
$\rho_{0}=2.3\times 10^{8}M_{\odot}\mathrm{kpc}^{-3}$, $r_{d}=76$ kpc,
$r_{t}=0.97$ kpc, and $r_{s}=0.35$ kpc as calculated in [23]. For the Galactic
Center, we use $\rho_{0}=4.9\times 10^{6}M_{\odot}\mathrm{kpc}^{-3}$,
$r_{d}=8$ kpc, $r_{t}=90$ kpc, and $r_{s}=15.3$ kpc.

To determine the total J-factor for a particular solid angle field of view, we
integrate the J-factor over all relevant angles, as shown in Fig. 2. For an
annulus centered at $b=0$, making use of the spherical symmetry of the halo,
the total J-factor is

$2\pi\int_{\theta_{1}}^{\theta_{2}}\mathcal{J}(b)bdb.$ (10)

$\theta_{2}$$\theta_{1}$ Figure 2: The shaded annulus region is the area in
which we evaluate the signal-to-noise ratio.

###  III.3 Gamma-Ray Background

One of the challenges in conclusively detecting a signal from dark matter
annihilation is that the signal must be distinguishable from the astrophysical
background. At gamma-ray energies, key sources of background include inelastic
cosmic-ray interactions with ionized gas and inverse Compton scattering of
high-energy electrons off of the intervening photon background. Gamma-ray
point sources within or beyond the halo of interest contribute additional
noise.

There are certain characteristics of DM annihilation that may make it
distinguishable from the astrophysical background. In addition to model-
dependent spectral features, DM annihilation is expected to result in a signal
that is highly concentrated towards the central regions of the halo, but still
more extended than a point source. In order to extract a potential signal, it
is paramount to identify the optimal observation region, which, we define here
as the region with the largest signal-to-noise ratio.

Regarding gamma-ray sources within the DM halos of interest, studies using
data from the Fermi Large Area Telescope (LAT) have shown that no
statistically significant excess of gamma-ray emission is present in 8
observed dSph candidates [24]. However, in larger galaxies, a significant
gamma-ray background from the central regions of the halo has been detected.
See, for instance, promising targets for DM searches such as the M31
(Andromeda) galaxy, where a central diffuse emission has been detected [25,
5]; in other cases, the background source is genuinely a point-like object,
such as an active galactic nucleus as in the case of M87 [26].

For simplicity, we assume a model of gamma-ray background consisting of two
components. For the extragalactic case study, we use (1) an isotropic
component, modeling the diffuse background, and (2) a central point source,
suitably smeared to account for the finite instrumental point spread function
(see details below). The isotropic piece is aimed at capturing both the
diffuse gamma-ray background produced by interactions of high energy cosmic
rays with interstellar gas, and the extragalactic gamma-ray background. The
point source accounts for the contribution from gamma-ray emission in the
central region of galaxies from active galactic nuclei or other sources, such
as a dense population of unresolved millisecond pulsars [27, 28].

For the Galactic Center (GC), we expect that both the DM annihilation and
other astrophysical processes would result in spherically symmetric emission.
Data from the Fermi Gamma Ray Space Telescope show that gamma-ray emission
close to the Galactic Center is distributed as $r^{-1.55}$, where $r$ is the
distance to the Galactic Center [9]. This is perhaps due to the gas or inverse
Compton emission in the Galactic Bulge or from a population of unresolved
point sources. We expect the isotropic background to be subdominant for the
Galactic Center, and therefore ignore it. In this case then, the two
components of the background are (i) $N_{B}\propto\int dlr^{-1.55}$, and again
(ii) a central point source, possibly associated with the central supermassive
black hole Sag A∗ or a concentrated cluster of faint point sources.

We model the contribution from the point source as a Gaussian centered at the
center of the given DM halo ($b=0$). The photon count from this component of
the background is proportional to

$N_{G}(b)\propto\frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\big{(}\frac{b^{2}}{\sigma^{2}}\big{)}},$
(11)

such that $\sigma$ is the angular width of the central source accounting for
the instrument-dependent point spread function. We define $N_{G}(b)$ as the
contribution from the entire line-of-sight for a given angle $b$ so that the
total background from the point source is integrated over the field of view

$N_{Gtot}\propto\int_{\theta_{1}}^{\theta_{2}}N_{G}(b)bdb.$ (12)

The total isotropic contribution to the background is simply proportional to
the field of view

$N_{Itot}\propto N_{I0}(\theta_{2}^{2}-\theta_{1}^{2}),$ (13)

where $N_{I0}$ is the isotropic background per solid angle. Thus the total
background contribution is simply $N_{Itot}+N_{Gtot}$.

We normalize the isotropic background relative to the area of the Gaussian
point source (i.e. 1) so that $N_{Gtot}=\eta N_{I0}$, in which $\eta$ can take
on a range of values. For the Galactic Center case, we normalize the bulge
component of the background relative to the point source, such that
$N_{Gtot}=\eta N_{B}(0)$. For the Galactic Center, the total background is
thus $N_{Btot}+N_{Gtot}$ where
$N_{Btot}\propto\int_{\theta_{1}}^{\theta_{2}}N_{B}(b)bdb$.

A large $\eta$ corresponds to the case where the central gamma ray source
dominates the background, such as in the case of galaxies with an active
galactic nucleus. Conversely, a smaller $\eta$ corresponds to a smaller
contribution from the central region of a halo and a flatter background
profile. This is most relevant for dSphs where there is not typically a large
central gamma ray source, but a sub-dominant faint source below detection
threshold cannot be ruled out (see e.g. [29]).

The number of detected photons amounts to counting independent events that
randomly occur at a constant rate, and is therefore well described by a
Poisson distribution. Since the mean value of a Poisson distribution is also
its variance, the standard deviation is simply the square root of the count.
We therefore take the noise to be given by the square root of the number of
background gamma-ray events. Therefore, we define the “optimal observing
region” as the region that maximizes the quantity

$\frac{\mathcal{J}}{N_{\mathrm{{tot}}}^{1/2}}.$ (14)

##  IV Results

###  IV.1 Extragalactic Source

For the extragalactic case, we show $\mathcal{J}(b)$ per unit solid angle for
$s$-wave, $p$-wave, and $d$-wave annihilation in Fig.3. Note that due to the
assumption of the halo’s spherical symmetry, $\mathcal{J}$ is only a function
of one variable, the angle $b$.

Figure 3: $\mathcal{J}$ per solid angle as a function of the angle $b$ for
different annihilation channels, normalized to their respective values at
$b=10^{-3}$ degrees. The contribution to the J-factor from the velocity-
dependent $p$ and $d$-wave channels extends to larger angles when compared to
the velocity-independent $s$-wave case.

In the $s$-wave case, the J-factor per unit solid angle decreases rapidly with
increasing $b$, decreasing approximately to $90\%$ from $b=0.001$ to $b=0.01$
degrees. By comparison, the $p$\- and $d$-wave channels decrease at more
modest rates and contribute significantly to the J-factor out to larger
angles. This is because velocity increases as a function of $r$, the distance
from the center of the halo, up until $r\approx r_{t}$ at which point the DM
particles are tidally stripped away and the enclosed mass ceases to increase.
Thus, the line-of-sight integral through the halo will be enhanced when
including this higher velocity region due to the velocity-dependence of the
cross section. Because the noise grows as the square root of the background,
there must be a significant enhancement of $\mathcal{J}$ or a large spike in
the background signal at small $b$ in order for it to be prudent to exclude
the central region from the region of interest. We note that there are
uncertainties associated with the calculation of the tidal radius. However,
only a small fraction of the total annihilation flux comes from the outermost
regions of the halo, so we don’t expect changes in the tidal radius to affect
our results in any meaningful way (see the discussion in [23] for more
details).

Figure 4: $\frac{\mathcal{J}}{N_{\mathrm{{tot}}}^{1/2}}$ for $s$, $p$, and
$d$-wave annihilation for an extragalactic source using an NFW density profile
with $\sigma=0.05^{\circ}$ and $\eta=20$. The black dot is the field of view
used in a Fermi-LAT survey of MW satellites [30, 16], while the orange and
yellow dots correspond to observations of M31 using Fermi-LAT [14, 13].

We plot $\frac{\mathcal{J}}{N_{\mathrm{b}}^{1/2}}$ as a function of
$\theta_{1}$ and $\theta_{2}-\theta_{1}$, the angular extent of the inner side
of the annulus and the thickness of the annulus, respectively. Using an NFW
density profile for Draco, we show the signal-to-noise ratio for a dominant
central gamma-ray source ($\eta=20$) for the case of $s$-wave, $p$-wave, and
$d$-wave annihilation in Fig. 4. We see that for the $s$-wave channel, the
optimal strategy is a large circular field of view, including the entire
angular extent of the source. For the $p$-wave case, there is a much broader
range of observation strategies that yield large signal-to-noise ratios; the
disk field of view is still viable, but excluding the central region yields a
similar outcome. However, for the $d$-wave case, the optimal observation
strategy becomes an annulus, excluding the inner $0.1$ degrees of the halo and
having a slightly smaller field of view, with a width extending only about
$0.3$ degrees, i.e. cropping out both the inner and outer regions of the halo.

For reference, the dots shown on the EG figures represent the observation
strategies used in previous searches for gamma-ray emission due to dark matter
annihilation. The black dot corresponds to surveys of Milky Way dSphs using
Fermi-LAT data [16, 30]. The orange and yellow dots correspond to observations
of M31, again using Fermi-LAT [13, 14]. We note that a square field of view
(e.g. $10^{\circ}\times 10^{\circ}$) is represented as a circular field of
view (e.g. $\theta_{2}-\theta_{1}=10^{\circ}$) in the figure for ease of
comparison.

It is also worth keeping in mind that the angular resolution of Fermi-Lat
varies depending on the single-photon energy. For $>10$ GeV photons, the
resolution is $<0.15^{\circ}$, whereas for sub-GeV photons, the resolution is
$\gtrsim 1.0^{\circ}$. The typical energy range observed is $\approx 100$ MeV
- $500$ GeV. For comparison, next generation telescopes, such as AdEPT, should
achieve $0.1^{\circ}$ at $\approx 1$ GeV [31]. We see that even with current-
generation instruments, there is room for improvement in terms of the signal-
to-noise ratio for the given astrophysical background.

Using the cored Burkert density profile,

$\rho_{\mathrm{c}}=\frac{\rho_{0}}{\big{(}1+\frac{r}{r_{s}}\big{)}\big{(}1+(\frac{r}{r_{s}})^{2}\big{)}}$
(15)

we show the signal to noise ratio in Fig. 5 for $s$\- and $p$-wave channels.
The behavior of the $d$-wave channel in this case is quite similar to that of
the $p$-wave channel so we only show $s$ and $p$ for simplicity. The width of
the central point source is fixed at $\sigma=0.05$ degrees. Similarly to the
corresponding NFW profile examples, the best field of view for the velocity-
independent channel is either a disk or a thin annulus. For the velocity-
dependent channels, the optimal field of view is achieved with an annulus,
excluding the inner $0.05$ degrees of the halo, which corresponds to the width
of the point source. Note that the point source need not be extremely bright
for this to be the case; this behavior occurs for a moderate value of
$\eta=3$. The extent of the annulus is only about $0.4$ degrees, whereas the
total extent of sources like M31, M87, and Draco are about $0.7$ degrees, so
excluding the outermost region of these halos would be beneficial.

Figure 5: $\frac{\mathcal{J}}{N_{\mathrm{{tot}}}^{1/2}}$ for $s$ and $p$-wave
annihilation for an extragalactic source using a Burkert profile with
$\sigma=0.05^{\circ}$ and $\eta=3$. The black dot is the field of view used in
a Fermi-LAT survey of MW satellites [30, 16], while the orange and yellow dots
correspond to observations of M31 using Fermi-LAT [14, 13].

###  IV.2 Galactic Center

Figure 6: $\frac{\mathcal{J}}{N_{\mathrm{{tot}}}^{1/2}}$ for $s$ and $p$-wave
annihilation in the Galactic Center with $\sigma=0.5^{\circ}$ and $\eta=50$,
corresponding to a dominant central point source. The green dot is the field
of view used by a Fermi-LAT survey of the Galactic Center looking for $p$-wave
annihilation [32]. The red dot represents a previous search for $s$-wave
annihilation using Fermi-LAT with a 2 degree plane mask [33]. Figure 7:
$\frac{\mathcal{J}}{N_{\mathrm{{tot}}}^{1/2}}$ for $s$ and $p$-wave
annihilation in the Galactic Center with $\sigma=0.5^{\circ}$ and $\eta=0.1$,
corresponding to a dominant noise contribution from the GC bulge. The green
dot is the field of view used by a Fermi-LAT survey of the Galactic Center
looking for $p$-wave annihilation [32]. The red dot corresponds to a search
for $s$-wave annihilation using Fermi-LAT with a 2 degree plane mask [33].

We calculate $\frac{\mathcal{J}}{N_{\mathrm{b}}^{1/2}}$ for the Galactic
Center for $s$\- and $p$-wave annihilation under two scenarios: In the first,
the central gamma-ray source is large compared to the bulge component with
$\eta=50$, a case shown in Fig. 6. In the second scenario, the central gamma-
ray source is very small compared to the bulge component with $\eta=0.1$, a
case shown in Fig. 7. The second case can be thought of as corresponding to an
optimistic scenario where the bulk of the central Galactic emission stems from
dark matter, versus a pessimistic one where it does not (originating instead
from e.g. unresolved point sources).

Let us first consider the case with a bright central point source, shown in
Fig. 6. For the $s$-wave channel, the highest signal-to-noise ratio can be
obtained by using an annulus, blocking out the innermost degree and extending
out to tens of degrees. Indeed, this is similar to the approach taken by [33]
which uses a plane mask for the inner 2 degrees of the Galactic latitude and a
field of view extending out to 30 degrees. This approach is represented as a
red dot in Figs. 7 and 6. Note that the present work is done considering a
purely spherical halo, so the inner ring of the annulus would not be confined
to the plane of galactic latitude. Even still, it is interesting to find that
an existing analysis uses a mask ideal for detecting $s$-wave annihilation
according to this calculation.

The $p$-wave case yields a very different result. Here, the best approach is
to include as wide a region as possible. This is because the velocity
enhancement of the cross section leads to a non-negligible contribution to the
J-factor per solid angle, even far from the Galactic Center. In fact, one can
exclude as much as the inner 10 degrees of the Galactic Center, and still end
up with an ideal signal to noise ratio by using a very wide field of view.

The dots shown in the GC figures represent the observation strategies used in
previous gamma-ray surveys in the Galactic Center. The red dot represents a
survey using Fermi-LAT data for energies $2-20$ GeV with a $<2^{\circ}$ plane
mask with respect to the Galactic latitude [33]. For ease of comparison, we
represent this as a circular $2^{\circ}$ mask in the figures. Interestingly.
we find that the plane mask yields a very high signal-to-noise ratio for both
the $s$\- and $p$-wave scenarios given the astrophysical background used here.
This suggests that one does not necessarily need to search for a particular
annihilation channel at the exclusion of another.

The green dot corresponds to a search for $p$-wave annihilation of dark matter
using Fermi-LAT in the energy range $10-600$ GeV [32]. The region of interest
was $2^{\circ}\times 2^{\circ}$, approximately represented in the figure as
$\theta_{2}-\theta_{1}=2^{\circ}$. Here, the goal was to make use of the
increase in the DM velocity in the region of the central supermassive black
hole Sgr $A^{*}$. For this region of interest, we find a relatively low
signal-to-noise ratio for both $s$ and $p$-wave annihilation. However, we note
that the corresponding survey considered the existence of a localized dark
matter over-density near Sag $A^{*}$, which could significantly change the
J-factor [32]. We did not consider DM substructure in this work, so we leave
this possibility to future investigation.

We now turn to the second GC case, shown in Fig. 7, which has a small central
gamma-ray source compared to the bulge component. Here, the optimal strategy
for the $s$-wave channel is once again a disk. With no dominant point source,
the highest signal-to-noise ratio is attained by including the center of the
halo where the dark matter density is highest. Again in the $p$-wave case, the
velocity enhancement implies that a much larger field of view should be used,
regardless of whether or not the inner region of the halo is included in the
field of view.

##  V Discussion and conclusions

In this paper, we have investigated the behavior of the DM annihilation rate
through its dependence on the J-factor for cases in which the annihilation
cross section is velocity-dependent. Under simple assumptions for the
astrophysical background, we find that the annihilation signal-to-noise ratio
can be larger for an annulus field of view than for a disk when considering
$p$\- and $d$-wave channels, even when there is not an overwhelming background
from a central gamma-ray point source. This can be true for the extragalactic
case, regardless of whether the DM distribution is cuspy or cored. In the case
of the Galactic Center, the optimal observation strategy for velocity-
dependent channels is again quite different than for the velocity-independent
one, but this manifests differently than in the extragalactic scenario. Here,
we find that the $p$-wave signal-to-noise ratio is optimized for a much larger
field of view than that of the $s$-wave signal. Intriguingly, one can search
for both channels simultaneously and still have a very high signal-to-noise
ratio for each by masking the inner 1-5 degrees of the halo and extending the
field of view out to nearly 100 degrees.

We have taken a number of simplifying assumptions throughout this paper for
illustrative purposes. It is now worthwhile to mention and revisit these
assumptions. We have neglected the baryonic matter contribution to the
gravitational potential, which would contribute to the DM’s velocity
dispersion (this is expected to be a subdominant effect in some cases, for
instance dSph, but more relevant in others, such as the Galactic Center). We
have also ignored the possibility of DM substructure within the Milky Way and
extragalactic halos. Such substructures would alter both the density and the
velocity distribution of the dark matter, and therefore that would go beyond
our analysis.

Another simplifying assumption was that the DM relative velocity distribution
follows a Maxwell-Boltzmann distribution. In reality, this is an
oversimplification as the distribution is only truly Maxwell-Boltzmann for the
case of a singular isothermal sphere. More generally, one could consider an
anisotropic distribution and relate the DM velocity distribution to the
density profile through the Eddington inversion formula [34]. In this case,
the velocity dispersion would no longer be trivially related to the circular
velocity and one would need to more accurately determine the relative velocity
of the dark matter particles to calculate $\mathcal{J}$.

Specific values for the width of the central gamma-ray point source $\sigma$
and the relative normalization of the background sources $\eta$ were chosen ad
hoc on the basis that these served as reasonable benchmarks and demonstrated
non-trivial results for various annihilation channels. The purpose of this
more general work was to demonstrate how one might carry out such analysis and
to demonstrate a few interesting cases as examples. Ideally, one would analyze
a given target of observation and determine the optimal strategy for that
particular target using the approach and method presented here.

Lastly, we have examined both an NFW and core-type Burkert density profile for
the dark matter. In reality, the density distribution of DM halos is wide
ranging [for the case of dSph see e.g. 35]. As a result, the optimal
observational strategy for different dSphs may vary, even for the same
annihilation channel and similar central gamma-ray sources.

###### Acknowledgements.

This material is based upon work supported in part by the National Science
Foundation Graduate Research Fellowship under Grant No. DGE-1842400 to NS. SP
is partly supported by the U.S. Department of Energy grant number de-
sc0010107.

## Data availability

The code used in this article will be shared upon request to the authors.

## References

  * Particle Data Group  _et al._ [2020] Particle Data Group, P. A. Zyla, _et al._ , Review of Particle Physics, Progress of Theoretical and Experimental Physics 2020, 10.1093/ptep/ptaa104 (2020).
  * Profumo [2020] S. Profumo, Dark Matter Indirect Detection, EPJ Web of Conferences (2020).
  * Slatyer [2017] T. R. Slatyer, TASI Lectures on Indirect Detection of Dark Matter, arXiv:1710.05137 [astro-ph, physics:hep-ph] (2017).
  * Strigari [2018] L. E. Strigari, Dark Matter in Dwarf Spheroidal Galaxies and Indirect Detection: A Review, Reports on Progress in Physics 81, 056901 (2018), arXiv:1805.05883 .
  * Do  _et al._ [2020] A. Do, M. Duong, A. McDaniel, C. O’Connor, S. Profumo, J. Rafael, C. Sweeney, and W. Vera III, Cosmic-ray transport and gamma-ray emission in M31, arXiv:2012.14507 [astro-ph, physics:hep-ph] (2020).
  * McDaniel  _et al._ [2018] A. McDaniel, T. Jeltema, and S. Profumo, A Multi-Wavelength Analysis of Annihilating Dark Matter as the Origin of the Gamma-Ray Emission from M31, Physical Review D 97, 103021 (2018), arXiv:1802.05258 .
  * Profumo and Ubaldi [2011] S. Profumo and L. Ubaldi, Cosmic Ray-Dark Matter Scattering: A New Signature of (Asymmetric) Dark Matter in the Gamma Ray Sky, Journal of Cosmology and Astroparticle Physics 2011 (08), 020, arXiv:1106.4568 .
  * Board  _et al._ [2021] E. Board, N. Bozorgnia, L. E. Strigari, R. J. J. Grand, A. Fattahi, C. S. Frenk, F. Marinacci, and J. F. Navarro, Velocity-dependent J-factors for annihilation radiation from cosmological simulations, arXiv:2101.06284 [astro-ph, physics:hep-ph] (2021).
  * Hooper and Goodenough [2011] D. Hooper and L. Goodenough, Dark Matter Annihilation in The Galactic Center As Seen by the Fermi Gamma Ray Space Telescope, Physics Letters B 697, 412 (2011), arXiv:1010.2752 .
  * Daylan  _et al._ [2016] T. Daylan, D. P. Finkbeiner, D. Hooper, T. Linden, S. K. N. Portillo, N. L. Rodd, and T. R. Slatyer, The Characterization of the Gamma-Ray Signal from the Central Milky Way: A Compelling Case for Annihilating Dark Matter, Physics of the Dark Universe 12, 1 (2016), arXiv:1402.6703 .
  * Collaboration [2016] T. F.-L. Collaboration, Fermi-LAT Observations of High-Energy Gamma-Ray Emission Toward the Galactic Center, The Astrophysical Journal 819, 44 (2016), arXiv:1511.02938 .
  * Abazajian  _et al._ [2014] K. N. Abazajian, N. Canac, S. Horiuchi, and M. Kaplinghat, Astrophysical and Dark Matter Interpretations of Extended Gamma-Ray Emission from the Galactic Center, Physical Review D 90, 023526 (2014), arXiv:1402.4090 .
  * Mauro  _et al._ [2019] M. D. Mauro, X. Hou, C. Eckner, G. Zaharijas, and E. Charles, Search for $\gamma$-ray emission from dark matter particle interactions from the Andromeda and Triangulum galaxies with the Fermi Large Area Telescope, PHYS. REV. D , 26 (2019).
  * Feng  _et al._ [2019] L. Feng, Z. Li, M. Su, P.-H. T. Tam, and Y. Chen, Searching for GeV gamma-ray emission from the bulge of M31, Research in Astronomy and Astrophysics 19, 046 (2019), arXiv:1810.10721 .
  * Collaboration [2015] F.-L. Collaboration, Searching for Dark Matter Annihilation from Milky Way Dwarf Spheroidal Galaxies with Six Years of Fermi-LAT Data, Physical Review Letters 115, 231301 (2015), arXiv:1503.02641 .
  * Abdo  _et al._ [2010] A. A. Abdo _et al._ , Observations of Milky Way Dwarf Spheroidal galaxies with the Fermi-LAT detector and constraints on Dark Matter models, The Astrophysical Journal 712, 147 (2010), arXiv:1001.4531 .
  * Kumar and Marfatia [2013] J. Kumar and D. Marfatia, Matrix element analyses of dark matter scattering and annihilation, Physical Review D 88, 014035 (2013), arXiv:1305.1611 .
  * Giacchino  _et al._ [2013] F. Giacchino, L. Lopez-Honorez, and M. H. G. Tytgat, Scalar Dark Matter Models with Significant Internal Bremsstrahlung, Journal of Cosmology and Astroparticle Physics 2013 (10), 025, arXiv:1307.6480 .
  * Han  _et al._ [2016] C. Han, H. M. Lee, M. Park, and V. Sanz, The diphoton resonance as a gravity mediator of dark matter, Physics Letters B 755, 371 (2016), arXiv:1512.06376 .
  * Navarro  _et al._ [1997] J. F. Navarro, C. S. Frenk, and S. D. M. White, A Universal Density Profile from Hierarchical Clustering, The Astrophysical Journal 490, 493 (1997), arXiv:astro-ph/9611107 .
  * Bergstrom and Hooper [2006] L. Bergstrom and D. Hooper, Dark Matter and Gamma-Rays From Draco: MAGIC, GLAST and CACTUS, Physical Review D 73, 063510 (2006), arXiv:hep-ph/0512317 .
  * Burkert [1995] A. Burkert, The Structure of Dark Matter Haloes in Dwarf Galaxies, The Astrophysical Journal 447, 10.1086/309560 (1995), arXiv:astro-ph/9504041 .
  * Klop  _et al._ [2017] N. Klop, F. Zandanel, K. Hayashi, and S. Ando, Impact of axisymmetric mass models for dwarf spheroidal galaxies on indirect dark matter searches, Physical Review D 95, 123012 (2017), arXiv:1609.03509 .
  * Drlica-Wagner [2015] A. Drlica-Wagner, SEARCH FOR GAMMA-RAY EMISSION FROM DES DWARF SPHEROIDAL GALAXY CANDIDATES WITH FERMI-LAT DATA, The Astrophysical Journal Letters , 8 (2015).
  * Karwin  _et al._ [2021] C. Karwin, S. Murgia, I. Moskalenko, S. Fillingham, A.-K. Burns, and M. Fieg, Dark Matter Interpretation of the Fermi-LAT Observations Toward the Outer Halo of M31, Physical Review D 103, 023027 (2021), arXiv:2010.08563 .
  * Collaboration  _et al._ [2009] T. F. L. Collaboration, A. A. Abdo, D. E. Harris, F. Massaro, and L. Stawarz, Fermi Large Area Telescope Gamma-Ray Detection of the Radio Galaxy M87, The Astrophysical Journal 707, 55 (2009), arXiv:0910.3565 .
  * Bartels  _et al._ [2016] R. Bartels, S. Krishnamurthy, and C. Weniger, Strong support for the millisecond pulsar origin of the Galactic center GeV excess, arXiv:1506.05104 [astro-ph, physics:hep-ph] 10.1103/PhysRevLett.116.051102 (2016).
  * Hooper and Linden [2018] D. Hooper and T. Linden, Millisecond Pulsars, TeV Halos, and Implications For The Galactic Center Gamma-Ray Excess, Physical Review D 98, 043005 (2018), arXiv:1803.08046 .
  * Gonzalez-Morales  _et al._ [2014] A. X. Gonzalez-Morales, S. Profumo, and F. S. Queiroz, Effect of Black Holes in Local Dwarf Spheroidal Galaxies on Gamma-Ray Constraints on Dark Matter Annihilation, Physical Review D 90, 103508 (2014), arXiv:1406.2424 .
  * Albert  _et al._ [2017] A. Albert _et al._ , Searching for Dark Matter Annihilation in Recently Discovered Milky Way Satellites with Fermi-LAT, The Astrophysical Journal 834, 110 (2017), arXiv:1611.03184 .
  * Hunter  _et al._ [2014] S. D. Hunter, P. F. Bloser, G. O. Depaola, M. P. Dion, G. A. DeNolfo, A. R. Hanu, M. L. Iparraguirre, J. Legere, M. L. McConnell, S. F. Nowicki, J. M. Ryan, S. Son, and F. W. Stecker, A Pair Production Telescope for Medium-Energy Gamma-Ray Polarimetry, Astroparticle Physics 59, 18 (2014), arXiv:1311.2059 .
  * Johnson  _et al._ [2019] C. Johnson, R. Caputo, C. Karwin, S. Murgia, S. Ritz, and J. Shelton, Search for gamma-ray emission from $p$-wave dark matter annihilation in the Galactic Center, Physical Review D 99, 103007 (2019), arXiv:1904.06261 .
  * Leane and Slatyer [2019] R. K. Leane and T. R. Slatyer, Dark Matter Strikes Back at the Galactic Center, Physical Review Letters 123, 241101 (2019), arXiv:1904.08430 .
  * Widrow [2000] L. M. Widrow, Distribution Functions for Cuspy Dark Matter Density Profiles, The Astrophysical Journal Supplement Series 131, 39 (2000).
  * Oman  _et al._ [2015] K. A. Oman, J. F. Navarro, A. Fattahi, C. S. Frenk, T. Sawala, S. D. M. White, R. Bower, R. A. Crain, M. Furlong, M. Schaller, J. Schaye, and T. Theuns, The unexpected diversity of dwarf galaxy rotation curves, Monthly Notices of the Royal Astronomical Society 452, 3650 (2015).


# Beyond temperature scaling:
Obtaining well-calibrated multi-class probabilities with Dirichlet calibration
Supplementary material

Meelis Kull
Department of Computer Science
University of Tartu
<EMAIL_ADDRESS>
&Miquel Perello-Nieto
Department of Computer Science
University of Bristol
<EMAIL_ADDRESS>
&Markus Kängsepp
Department of Computer Science
University of Tartu
<EMAIL_ADDRESS>
&Telmo Silva Filho
Department of Statistics
Universidade Federal da Paraíba
<EMAIL_ADDRESS>
&Hao Song
Department of Computer Science
University of Bristol
<EMAIL_ADDRESS>
&Peter Flach
Department of Computer Science
University of Bristol and
The Alan Turing Institute
<EMAIL_ADDRESS>

###### Contents

  1. A Source code
  2. B Proofs
  3. C Dirichlet calibration
    1. C.1 Reliability diagram examples
  4. D Experimental setup
    1. D.1 Datasets and performance estimation
    2. D.2 Full example of statistical analysis
  5. E Results
    1. E.1 Final ranking tables for all metrics
    2. E.2 Final critical difference diagrams for every metric
    3. E.3 Best calibrator hyperparameters
    4. E.4 Comparison of classifiers
    5. E.5 Deep neural networks

##  Appendix A Source code

The instructions and code for the experiments can be found on
https://dirichletcal.github.io/.

##  Appendix B Proofs

######  Theorem 1 (Equivalence of generative, linear and canonical
parametrisations).

The parametric families
$\hat{\boldsymbol{\mu}}_{DirGen}(\mathbf{q};\boldsymbol{\alpha},\boldsymbol{\pi})$,
$\hat{\boldsymbol{\mu}}_{DirLin}(\mathbf{q};\mathbf{W},\mathbf{b})$ and
$\hat{\boldsymbol{\mu}}_{Dir}(\mathbf{q};\mathbf{A},\mathbf{c})$ are equal,
i.e. they contain exactly the same calibration maps.

###### Proof.

We will prove that:

  1. 1.

every function in
$\hat{\boldsymbol{\mu}}_{DirGen}(\mathbf{q};\boldsymbol{\alpha},\boldsymbol{\pi})$
belongs to
$\hat{\boldsymbol{\mu}}_{DirLin}(\mathbf{q};\mathbf{W},\mathbf{b})$;

  2. 2.

every function in
$\hat{\boldsymbol{\mu}}_{DirLin}(\mathbf{q};\mathbf{W},\mathbf{b})$ belongs to
$\hat{\boldsymbol{\mu}}_{Dir}(\mathbf{q};\mathbf{A},\mathbf{c})$;

  3. 3.

every function in
$\hat{\boldsymbol{\mu}}_{Dir}(\mathbf{q};\mathbf{A},\mathbf{c})$ belongs to
$\hat{\boldsymbol{\mu}}_{DirGen}(\mathbf{q};\boldsymbol{\alpha},\boldsymbol{\pi})$.

#### 1.

Consider a function
$\hat{\mu}(\mathbf{q})=\hat{\boldsymbol{\mu}}_{DirGen}(\mathbf{q};\boldsymbol{\alpha},\boldsymbol{\pi})$.
Let us start with an observation that any vector
$\mathbf{x}=(x_{1},\dots,x_{k})\in(0,\infty)^{k}$ with only positive elements
can be renormalised to add up to $1$ using the expression
$\boldsymbol{\sigma}(\mathbf{ln}(\mathbf{x}))$, since:

$\displaystyle\boldsymbol{\sigma}(\mathbf{ln}(\mathbf{x}))=\mathbf{exp}(\mathbf{ln}(\mathbf{x}))/(\sum_{i}\exp(\ln(x_{i})))=\mathbf{x}/(\sum_{i}x_{i})$

where $\mathbf{exp}$ is an operator applying exponentiation element-wise.
Therefore,

$\displaystyle\hat{\mu}(\mathbf{q})=\boldsymbol{\sigma}(\mathbf{ln}(\pi_{1}f_{1}(\mathbf{q}),\dots,\pi_{k}f_{k}(\mathbf{q})))$

where $f_{i}(\mathbf{q})$ is the probability density function of the
distribution $Dir(\boldsymbol{\alpha}^{(i)})$ where
$\boldsymbol{\alpha}^{(i)}$ is the $i$-th row of matrix $\boldsymbol{\alpha}$.
Hence,
$f_{i}(\mathbf{q})=\frac{1}{B(\boldsymbol{\alpha}^{(i)})}\prod_{j=1}^{k}q_{j}^{\alpha_{ij}-1}$,
where $B(\cdot)$ denotes the multivariate beta function. Let us define a
matrix $\mathbf{W}$ and vector $\mathbf{b}$ as follows:

$\displaystyle w_{ij}=\alpha_{ij}-1,\qquad
b_{i}=\ln(\pi_{i})-\ln(B(\boldsymbol{\alpha}^{(i)}))$

with $w_{ij}$ and $\alpha_{ij}$ denoting elements of matrices $\mathbf{W}$ and
$\boldsymbol{\alpha}$, respectively, and $b_{i},\pi_{i}$ denoting elements of
vectors $\mathbf{b}$ and $\boldsymbol{\pi}$. Now we can write

$\displaystyle\ln(\pi_{i}f_{i}(\mathbf{q}))$
$\displaystyle=\ln(\pi_{i})-\ln(B(\boldsymbol{\alpha}^{(i)}))+\ln\prod_{j=1}^{k}q_{j}^{\alpha_{ij}-1}$
$\displaystyle=\ln(\pi_{i})-\ln(B(\boldsymbol{\alpha}^{(i)}))+\sum_{j=1}^{k}(\alpha_{ij}-1)\ln(q_{j})$
$\displaystyle=b_{i}+\sum_{j=1}^{k}w_{ij}\ln(q_{j})$

and substituting this back into $\hat{\mu}(\mathbf{q})$ we get:

$\displaystyle\hat{\mu}(\mathbf{q})$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{ln}(\pi_{1}f_{1}(\mathbf{q}),\dots,\pi_{k}f_{k}(\mathbf{q})))$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{b}+\mathbf{W}\mathbf{ln}(\mathbf{q}))=\hat{\mu}_{DirLin}(\mathbf{q};\mathbf{W},\mathbf{b})$

#### 2.

Consider a function
$\hat{\mu}(\mathbf{q})=\hat{\boldsymbol{\mu}}_{DirLin}(\mathbf{q};\mathbf{W},\mathbf{b})$.
Let us define a matrix $\mathbf{A}$ and vector $\mathbf{c}$ as follows:

$\displaystyle
a_{ij}=w_{ij}-\min_{i}w_{ij},\qquad\mathbf{c}=\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\mathbf{u}+\mathbf{b})$

with $a_{ij}$ and $w_{ij}$ denoting elements of matrices $\mathbf{A}$ and
$\mathbf{W}$, respectively, and $\mathbf{u}=(1/k,\dots,1/k)$ is a column
vector of length $k$. Note that
$\mathbf{A}\,\mathbf{x}=\mathbf{W}\,\mathbf{x}+const_{1}$ and
$\mathbf{ln}\,\boldsymbol{\sigma}(\mathbf{x})=\mathbf{x}+const_{2}$ for any
$x$ where $const_{1}$ and $const_{2}$ are constant vectors (all elements are
equal), but the constant depends on $\mathbf{x}$. Taking into account that
$\boldsymbol{\sigma}(\mathbf{v}+const)=\boldsymbol{\sigma}(\mathbf{v})$ for
any vector $\mathbf{v}$ and constant vector $const$, we obtain:

$\displaystyle\hat{\mu}_{Dir}(\mathbf{q};\mathbf{A},\mathbf{c})$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{A}\,\mathbf{ln}\,\frac{\mathbf{q}}{1/k}+\mathbf{ln}\,\mathbf{c})=\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\frac{\mathbf{q}}{1/k}+const_{1}+\mathbf{ln}\,\mathbf{c})$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\mathbf{q}-\mathbf{W}\,\mathbf{ln}\,\mathbf{u}+const_{1}+\mathbf{ln}\,\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\mathbf{u}+\mathbf{b}))$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\mathbf{q}-\mathbf{W}\,\mathbf{ln}\,\mathbf{u}+const_{1}+\mathbf{W}\,\mathbf{ln}\,\mathbf{u}+\mathbf{b}+const_{2})$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\mathbf{q}+\mathbf{b}+const_{1}+const_{2})=\boldsymbol{\sigma}(\mathbf{W}\,\mathbf{ln}\,\mathbf{q}+\mathbf{b})=\hat{\mu}_{DirLin}(\mathbf{q};\mathbf{W},\mathbf{b})$
$\displaystyle=\hat{\mu}(\mathbf{q})$

#### 3.

Consider a function
$\hat{\mu}(\mathbf{q})=\hat{\boldsymbol{\mu}}_{Dir}(\mathbf{q};\mathbf{A},\mathbf{c})$.
Let us define a matrix $\boldsymbol{\alpha}$, vector $\mathbf{b}$ and vector
$\pi$ as follows:

$\displaystyle\alpha_{ij}=a_{ij}+1,\qquad\mathbf{b}=\mathbf{ln}\,\mathbf{c}-\mathbf{A}\,\mathbf{ln}\,\mathbf{u},\qquad\pi_{i}=\exp(b_{i})\cdot
B(\boldsymbol{\alpha}^{(i)})$

with $\alpha_{ij}$ and $a_{ij}$ denoting elements of matrices
$\boldsymbol{\alpha}$ and $\mathbf{A}$, respectively, and
$\mathbf{u}=(1/k,\dots,1/k)$ is a column vector of length $k$. We can now
write:

$\displaystyle\hat{\mu}(\mathbf{q})$
$\displaystyle=\hat{\mu}_{Dir}(\mathbf{q};\mathbf{A},\mathbf{c})=\boldsymbol{\sigma}(\mathbf{A}\,\mathbf{ln}\,\frac{\mathbf{q}}{1/k}+\mathbf{ln}\,\mathbf{c})=\boldsymbol{\sigma}(\mathbf{A}\,\mathbf{ln}\,\mathbf{q}-\mathbf{A}\,\mathbf{ln}\,\mathbf{u}+\mathbf{ln}\,\mathbf{c})$
$\displaystyle=\boldsymbol{\sigma}((\boldsymbol{\alpha}-1)\mathbf{ln}\,\mathbf{q}+\mathbf{b})$

Element $i$ in the vector within the softmax is equal to:

$\displaystyle\sum_{j=1}^{k}(\boldsymbol{\alpha}_{ij}-1)\ln(q_{j})+b_{j}$
$\displaystyle=\sum_{j=1}^{k}(\boldsymbol{\alpha}_{ij}-1)\ln(q_{j})+\ln(\pi_{i}\cdot\frac{1}{B(\boldsymbol{\alpha}^{(i)})})$
$\displaystyle=\ln(\pi_{i}\cdot\frac{1}{B(\boldsymbol{\alpha}^{(i)})}\prod_{j=1}^{k}q_{j}^{\boldsymbol{\alpha}_{ij}-1})$
$\displaystyle=\ln(\pi_{i}\cdot f_{i}(\boldsymbol{\alpha}^{(i)}))$

and therefore:

$\displaystyle\hat{\mu}(\mathbf{q})=\boldsymbol{\sigma}((\boldsymbol{\alpha}-1)\mathbf{ln}(\mathbf{q})+\mathbf{b})=\boldsymbol{\sigma}(\ln(\pi_{i}\cdot
f_{i}(\boldsymbol{\alpha}^{(i)})))=\hat{\boldsymbol{\mu}}_{DirGen}(\mathbf{q};\boldsymbol{\alpha},\boldsymbol{\pi})$

∎

The following proposition proves that temperature scaling can be viewed as a
general-purpose calibration method, being a special case within the Dirichlet
calibration map family.

######  Proposition 1.

Let us denote the temperature scaling family by
$\hat{\mu}^{\prime}_{TempS}(\mathbf{z};t)=\boldsymbol{\sigma}(\mathbf{z}/t)$
where $\mathbf{z}$ are the logits. Then for any $t$, temperature scaling can
be expressed as

$\displaystyle\hat{\mu}^{\prime}_{TempS}(\mathbf{z};t)=\hat{\mu}_{DirLin}(\boldsymbol{\sigma}(\mathbf{z});\frac{1}{t}\mathbf{I},\mathbf{0})$

where $\mathbf{I}$ is the identity matrix and $\mathbf{0}$ is the vector of
zeros.

###### Proof.

Let us first observe that for any $\mathbf{x}\in\mathbb{R}^{k}$ there exists a
constant vector $const$ (all elements are equal) such that
$\mathbf{ln}\,\boldsymbol{\sigma}(\mathbf{x})=\mathbf{x}+const$. Furthermore,
$\boldsymbol{\sigma}(\mathbf{v}+const)=\boldsymbol{\sigma}(\mathbf{v})$ for
any vector $\mathbf{v}$ and any constant vector $const$. Therefore,

$\displaystyle\hat{\mu}_{DirLin}(\boldsymbol{\sigma}(\mathbf{z});\frac{1}{t}\mathbf{I},\mathbf{0})$
$\displaystyle=\boldsymbol{\sigma}(\frac{1}{t}\,\mathbf{I}\,\mathbf{ln}\,\boldsymbol{\sigma}(\mathbf{z})))$
$\displaystyle=\boldsymbol{\sigma}(\frac{1}{t}\,\mathbf{I}\,(\mathbf{z}+const))$
$\displaystyle=\boldsymbol{\sigma}(\frac{1}{t}\,\mathbf{I}\,\mathbf{z}+\frac{1}{t}\,\mathbf{I}\,const)$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{z}/t+const^{\prime})$
$\displaystyle=\boldsymbol{\sigma}(\mathbf{z}/t)$
$\displaystyle=\hat{\mu}^{\prime}_{TempS}(\mathbf{z};t)$

where $const^{\prime}=\frac{1}{t}\,\mathbf{I}\,const$ is a constant vector as
a product of a diagonal matrix with a constant vector. ∎

##  Appendix C Dirichlet calibration

In this section we show some examples of reliability diagrams and other plots
that can help to understand the representational power of Dirichlet
calibration compared with other calibration methods.

###  C.1 Reliability diagram examples

We will look at two examples of reliability diagrams on the original
classifier and after applying $6$ calibration methods. Figure 1 shows the
first example for the 3 class classification dataset _balance-scale_ and the
classifier MLP. This figure shows the confidence-reliability diagram in the
first column and the classwise-reliability diagrams in the other columns.
Figure 1(a) shows how posterior probabilities from the MLP have small gaps
between the true class proportions and the predicted means. This visualisation
may indicate that the original classifier is already well calibrated. However,
when we separate the reliability diagram per class, we notice that the
predictions for the first class are underconfident; as indicated by low mean
predictions containing high proportions of the true class. On the other hand,
classes 2 and 3 are overconfident in the regions of posterior probabilities
compressed between $[0.2,0.5]$ while being underconfident in higher regions.
The discrepancy shown by analysing the individual reliability diagrams seems
to compensate in the general picture of the aggregated one.

Table 1: Averaged results for the confidence-ECE and classwise-ECE metrics of $6$ calibrators applied on an MLP trained in the _balance-scale_ dataset. | DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
conf-ECE | $\mathbf{0.04_{1}}$ | $0.05_{3}$ | $0.13_{7}$ | $0.05_{2}$ | $0.08_{6}$ | $0.05_{4}$ | $0.08_{5}$
cw-ECE | $0.12_{2}$ | $0.13_{3}$ | $0.29_{7}$ | $\mathbf{0.12_{1}}$ | $0.17_{5}$ | $0.15_{4}$ | $0.20_{6}$

The following subfigures show how the different calibration methods try to
reduce ECE, occasionally increasing the error. As can be seen in Table 1,
Dirichlet L2 and One-vs.Rest isotonic regression obtain the lowest ECE while
One-vs.Rest frequency binning makes the original calibration worse. Looking at
Figure 1(i) it is possible to see how temperature scaling manages to reduce
the overall overconfidence in the higher range of probabilities for classes 2
and 3, but makes the situation worse in the interval $[0.2,0.6]$. However, it
manages to reduce the overall ECE.

(a) Uncalibrated

(b) Uncalibrated per class

(c) OvR Frequency Binning

(d) One-vs.-Rest Frequency Binning per class

(e) OvR Width Binning

(f) One-vs.-Rest Width Binning calibration per class

(g) OvR Isotonic calibration

(h) One-vs.-Rest Isotonic calibration per class

(i) Temperature Scaling

(j) Temperature Scaling

(k) OvR Beta calibration

(l) One-vs.-Rest Beta calibration per class

(m) Dirichlet L2

(n) Dirichlet with L2 regularisation per class

Figure 1: Confidence-reliability diagrams in the first column and classwise-
reliability diagrams in the remaining columns, for a real experiment with the
multilayer perceptron classifier on the balance-scale dataset and a subset of
the calibrators. All the test partitions from the 5 times 5-fold-cross-
validation have been aggregated to draw every plot.

In the second example we show $3$ calibration methods for a 4 class
classification problem (car dataset) applied on the scores of an Adaboost
SAMME classifier. Figure 2 shows one reliability diagram per class ($C_{1}$
_acceptable_ , $C_{2}$ _good_ , $C_{3}$ _unacceptable_ , and $C_{4}$ _very
good_).

From this Figure we can see that the uncalibrated model is underconfident for
classes 1, 2 and 3, showing posterior probabilities never higher than $0.7$,
while having true class proportions higher than $0.7$ in the mentioned
interval. We can see that after applying some of the calibration models the
posterior probabilities reach higher probability values.

Table 2: Averaged results for the confidence-ECE and classwise-ECE metrics of $6$ calibrators applied on an Adaboost SAMME trained in the _car_ dataset. | DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
conf-ECE | $\mathbf{0.07_{1}}$ | $0.10_{4}$ | $0.12_{5}$ | $0.07_{2}$ | $0.09_{3}$ | $0.14_{7}$ | $0.14_{6}$
cw-ECE | $0.18_{2}$ | $0.23_{3}$ | $0.29_{5}$ | $\mathbf{0.18_{1}}$ | $0.25_{4}$ | $0.32_{7}$ | $0.29_{6}$

As can be seen in Table 2, Dirichlet L2 and One-vs.Rest Isotonic Regression
obtain the lowest ECE while Temperature Scaling makes the original calibration
worse. Figure 2(d) shows how Dirichlet calibration with L2 regularisation
achieved the largest spread of probabilities, also reducing the error mean gap
with the predictions and the true class proportions. On the other hand,
temperature scaling reduced ECE for class 1, but hurt the overall performance
for the other classes.

(a) Uncalibrated

(b) One-vs.-Rest Isotonic calibration

(c) Temperature Scaling

(d) Dirichlet with L2 regularisation

Figure 2: Reliability diagrams per class for a real experiment with the
classifier Ada boost SAMME on the car dataset and $3$ calibrators. The test
partitions from the 5 times 5-fold-cross-validation have been aggregated to
draw every plot.

A more detailed depiction of the previous reliability diagrams can be seen in
Figure 3. In this case, the posterior probabilities are not introduced in
bins, but a boxplot summarises their full distribution. The first observation
here is, for the _good_ and _very good_ classes, the uncalibrated model tends
to predict probability vectors with small variance, i.e. the outputs do not
change much among different instances. Among the calibration approaches,
temperature scaling still maintains this low level of variance, while both
isotonic and Dirichlet L2 manage to show a higher variance on the outputs.
While this observation cannot be justified here without quantitative analysis,
another observation clearly shows an advantage of using Dirichlet L2. For the
_acceptable_ class, only Dirichlet L2 is capable of providing the highest mean
probability for the correct class, while the other three methods tend to put
higher probability mass on the _unacceptable_ class on average.

(a) Uncalibrated

(b) Isotonic

(c) Temperature scaling

(d) Dirichlet L2

Figure 3: Effect of Dirichlet Calibration on the scores of Ada boost SAMME on
the _car_ dataset which is composed of $4$ classes (_acceptable_ , _good_ ,
_unacceptable_ , and _very good_). The whiskers of each box indicate the 5th
and 95th percentile, the notch around the median indicates the confidence
interval. The green error bar to the right of each box indicates one standard
deviation on each side of the mean. In each subfigure, the first boxplot
corresponds to the posterior probabilities for the samples of class 1, divided
in 4 boxes representing the posterior probabilities for each class. A good
classifier should have the highest posterior probabilities in the box
corresponding to the true class. In Figure 3(a) it is possible to see that the
first class (_acceptable_) is missclassified as belonging to the third class
(_unacceptable_) with high probability values, while Dirichlet Calibration is
able to alleviate that problem. Also, for the second and fourth true classes
(_good_ , and _very good_) the original classifier uses a reduced domain of
probabilities (indicative of underconfidence), while Dirichlet calibration is
able to spread these probabilities with more meaningful values (as indicated
by a reduction of the calibration losses; See Figure 2).

##  Appendix D Experimental setup

In this section we provide the detailed description of the experimental setup
on a variety of non-neural classifiers and datasets. While our implementation
of Dirichlet calibration is based on standard Newton-Raphson with multinomial
logistic loss and L2 regularisation, as mentioned at the end of Section 3,
existing implementations of logistic regression (e.g. scikit-learn) with the
log transformed predicted probabilities can also be easily applied.

###  D.1 Datasets and performance estimation

The full list of datasets, and a brief description of each one including the
number of samples, features and classes is presented in Table 3.

Figure 4 shows how every dataset was divided in order to get an estimated
performance for every combination of dataset, classifier and calibrator. Each
dataset was divided using 5 times 5-fold-cross-validation to create 25 test
partitions. For each of the 25 partitions the corresponding training set was
divided further with a 3-fold-cross-validation for wich the bigger portions
were used to train the classifiers (and validate the calibratiors if they had
hyperparameters), and the small portion was used to train the calibrators. The
3 calibrators trained in the inner 3-folds were used to predict the
corresponding test partition, and their predictions were averaged in order to
obtain better estimates of their performance with the 7 different metrics
(accuracy, Brier score, log-loss, maximum calibration error, confidence-ECE,
classwise-ECE and the p test statistic of the ECE metrics). Finally, the 25
resulting measures were averaged.

| n_samples | n_features | n_classes
---|---|---|---
dataset |  |  |
abalone | 4177 | 8 | 3
balance-scale | 625 | 4 | 3
car | 1728 | 6 | 4
cleveland | 297 | 13 | 5
dermatology | 358 | 34 | 6
glass | 214 | 9 | 6
iris | 150 | 4 | 3
landsat-satellite | 6435 | 36 | 6
libras-movement | 360 | 90 | 15
mfeat-karhunen | 2000 | 64 | 10
mfeat-morphological | 2000 | 6 | 10
mfeat-zernike | 2000 | 47 | 10
optdigits | 5620 | 64 | 10
page-blocks | 5473 | 10 | 5
pendigits | 10992 | 16 | 10
segment | 2310 | 19 | 7
shuttle | 101500 | 9 | 7
vehicle | 846 | 18 | 4
vowel | 990 | 10 | 11
waveform-5000 | 5000 | 40 | 3
yeast | 1484 | 8 | 10

Table 3: Datasets used for the large-scale empirical comparison.

Figure 4: Partitions of each dataset in order to estimate out-of-sample
measures.

###  D.2 Full example of statistical analysis

The following is a full example of how the final rankings and statistical
tests are computed. For this example, we will focus on the metric log-loss,
and we will start with the naive Bayes classifier. Table 4 shows the estimated
log-loss by averaging the 5-times 5-fold cross-validation log-losses of the
inner 3-fold aggregated predictions. The sub-indices are the ranking of every
calibrator for each dataset (ties in the ranking share the averaged rank). The
resulting table of sub-indices is used to compute the Friedman test statistic,
resulting in a value of $73.8$ and a p-value of $6.71e^{-14}$ indicating
statistical difference between the calibration methods. The last row contains
the average ranks of the full table, which is shown in the corresponding
critical difference diagram in Figure 5(a). The critical difference uses the
Bonferroni-Dunn one-tailed statistical test to compute the minimum ranking
distance that is shown in the Figure, indicating that for this particular
classifier and metric the Dirichlet calibrator with L2 regularisation is
significantly better than the other methods.

Table 4: Ranking of calibration methods applied on the classifier naive Bayes with log-loss (Friedman statistic test = 73.8, p-value = 6.71E-14) | DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
abalone | $\mathbf{0.89_{1}}$ | $0.89_{4}$ | $0.89_{2}$ | $0.90_{5}$ | $0.92_{6}$ | $0.89_{3}$ | $1.95_{7}$
balance-sc | $\mathbf{0.21_{1}}$ | $0.30_{2}$ | $0.36_{5}$ | $0.32_{3}$ | $0.35_{4}$ | $0.41_{6}$ | $0.47_{7}$
car | $\mathbf{0.38_{1}}$ | $0.59_{4}$ | $0.56_{2}$ | $0.57_{3}$ | $0.67_{5}$ | $1.56_{6.5}$ | $1.56_{6.5}$
cleveland | $\mathbf{1.02_{1}}$ | $1.30_{4}$ | $1.12_{2}$ | $1.38_{5}$ | $1.14_{3}$ | $2.18_{6}$ | $2.49_{7}$
dermatolog | $\mathbf{0.20_{1}}$ | $0.41_{5}$ | $0.23_{2}$ | $0.39_{3}$ | $0.40_{4}$ | $2.51_{7}$ | $2.51_{6}$
glass | $\mathbf{1.11_{1}}$ | $1.50_{4}$ | $1.14_{3}$ | $1.64_{5}$ | $1.12_{2}$ | $3.14_{6.5}$ | $3.14_{6.5}$
iris | $\mathbf{0.11_{1}}$ | $0.26_{5}$ | $0.33_{6}$ | $0.34_{7}$ | $0.21_{4}$ | $0.13_{3}$ | $0.13_{2}$
landsat-sa | $\mathbf{0.36_{1}}$ | $0.56_{2}$ | $0.58_{4}$ | $0.58_{3}$ | $0.74_{5}$ | $3.87_{6.5}$ | $3.87_{6.5}$
libras-mov | $\mathbf{0.97_{1}}$ | $1.36_{2}$ | $1.67_{4}$ | $1.91_{5}$ | $1.45_{3}$ | $4.90_{6.5}$ | $4.90_{6.5}$
mfeat-karh | $\mathbf{0.20_{1}}$ | $0.22_{2}$ | $0.38_{6}$ | $0.38_{5}$ | $0.29_{4}$ | $0.23_{3}$ | $0.44_{7}$
mfeat-morp | $\mathbf{0.72_{1}}$ | $0.91_{5}$ | $0.82_{2}$ | $0.87_{3}$ | $0.88_{4}$ | $1.75_{6.5}$ | $1.75_{6.5}$
mfeat-zern | $\mathbf{0.59_{1}}$ | $0.71_{2}$ | $0.82_{4}$ | $0.84_{6}$ | $0.84_{5}$ | $0.77_{3}$ | $1.73_{7}$
optdigits | $0.45_{2}$ | $0.57_{4}$ | $0.47_{3}$ | $\mathbf{0.44_{1}}$ | $0.84_{5}$ | $3.14_{6}$ | $3.14_{7}$
page-block | $\mathbf{0.17_{1}}$ | $0.21_{4}$ | $0.20_{3}$ | $0.18_{2}$ | $0.21_{5}$ | $0.74_{6.5}$ | $0.74_{6.5}$
pendigits | $\mathbf{0.19_{1}}$ | $0.46_{3}$ | $0.48_{4}$ | $0.46_{2}$ | $0.58_{5}$ | $1.30_{6.5}$ | $1.30_{6.5}$
segment | $\mathbf{0.28_{1}}$ | $0.62_{5}$ | $0.46_{3}$ | $0.45_{2}$ | $0.56_{4}$ | $1.39_{6.5}$ | $1.39_{6.5}$
vehicle | $\mathbf{0.99_{1}}$ | $1.09_{3}$ | $1.05_{2}$ | $1.13_{5}$ | $1.10_{4}$ | $1.16_{6}$ | $2.30_{7}$
vowel | $\mathbf{0.54_{1}}$ | $0.81_{2}$ | $1.07_{6}$ | $1.08_{7}$ | $0.90_{5}$ | $0.85_{4}$ | $0.84_{3}$
waveform-5 | $\mathbf{0.33_{1}}$ | $0.37_{2}$ | $0.38_{3}$ | $0.38_{4}$ | $0.46_{6}$ | $0.43_{5}$ | $0.80_{7}$
yeast | $\mathbf{1.18_{1}}$ | $1.41_{4}$ | $1.31_{2}$ | $1.33_{3}$ | $1.43_{5}$ | $5.10_{6.5}$ | $5.10_{6.5}$
avg rank | 1.05 | 3.40 | 3.40 | 3.95 | 4.40 | 5.53 | 6.28

(a) Average over all datasets for Naive Bayes classifier

(b) Average over all classifiers

Figure 5: Critical Difference diagrams for the averaged ranking results of the
metric Log-loss.

The same process is applied to each of the $11$ classifiers for every metric.
Table 6 shows the final average results of all classifiers. Notice that the
row corresponding to naive Bayes has the rounded average rankings from Figure
5(a).

##  Appendix E Results

In this Section we present all the final results, including ranking tables for
every metric, critical difference diagrams, the best hyperparameters selected
for Dirichlet calibration with L2 regularisation, Frequency binning and Width
binning; a comparison of how calibrated the $11$ classifiers are, and
additional results on deep neural networks.

###  E.1 Final ranking tables for all metrics

We present here all the final ranking tables for all metrics (Tables 6, 6, 7,
8, 9, 10, E.1, and E.1). For each ranking, a lower value is indicative of a
better metric value (eg. a higher accuracy corresponds to a lower ranking,
while a lower log-loss corresponds to a lower ranking as well). Additional
details on how to interpret the tables can be found in Section D.2.

Table 5: Rankings for Accuracy

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{2.5}$ | $4.1$ | $2.9$ | $4.5$ | $3.1$ | $5.3$ | $5.5$
forest | $4.0$ | $\mathbf{3.0}$ | $5.6$ | $3.2$ | $4.4$ | $3.9$ | $3.9$
knn | $5.0$ | $3.9$ | $4.8$ | $\mathbf{3.1}$ | $3.1$ | $4.0$ | $4.0$
lda | $3.5$ | $5.1$ | $4.9$ | $3.7$ | $5.0$ | $3.0$ | $\mathbf{2.9}$
logistic | $\mathbf{2.1}$ | $3.7$ | $5.3$ | $4.0$ | $3.6$ | $4.6$ | $4.7$
mlp | $2.9$ | $\mathbf{2.8}$ | $5.9$ | $3.7$ | $4.5$ | $4.0$ | $4.3$
nbayes | $\mathbf{1.4}$ | $3.8$ | $3.0$ | $2.9$ | $5.0$ | $6.0$ | $6.0$
qda | $\mathbf{2.7}$ | $3.6$ | $3.9$ | $2.9$ | $3.8$ | $5.6$ | $5.6$
svc-linear | $\mathbf{1.8}$ | $3.5$ | $5.7$ | $2.8$ | $4.3$ | $5.1$ | $4.8$
svc-rbf | $3.3$ | $3.5$ | $3.8$ | $\mathbf{3.2}$ | $3.6$ | $5.0$ | $5.5$
tree | $3.7$ | $4.8$ | $4.5$ | $5.0$ | $4.3$ | $2.8$ | $2.8$
avg rank | 2.99 | 3.78 | 4.58 | 3.55 | 4.07 | 4.48 | 4.54

Table 6: Rankings for log-loss | DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{1.4}$ | $3.1$ | $3.2$ | $4.3$ | $3.5$ | $5.9$ | $6.6$
forest | $4.2$ | $\mathbf{1.9}$ | $4.7$ | $4.1$ | $2.9$ | $5.2$ | $5.2$
knn | $3.8$ | $4.8$ | $3.0$ | $\mathbf{1.6}$ | $2.0$ | $6.5$ | $6.5$
lda | $\mathbf{1.6}$ | $2.2$ | $5.2$ | $5.2$ | $3.5$ | $4.6$ | $5.7$
logistic | $\mathbf{1.3}$ | $2.1$ | $5.8$ | $6.1$ | $3.5$ | $3.6$ | $5.6$
mlp | $\mathbf{2.2}$ | $2.3$ | $6.5$ | $6.2$ | $4.7$ | $2.9$ | $3.4$
nbayes | $\mathbf{1.1}$ | $3.4$ | $3.4$ | $4.0$ | $4.4$ | $5.5$ | $6.3$
qda | $\mathbf{1.7}$ | $2.7$ | $5.6$ | $4.6$ | $3.4$ | $4.2$ | $5.8$
svc-linear | $\mathbf{1.3}$ | $2.3$ | $6.1$ | $6.1$ | $4.3$ | $3.0$ | $4.8$
svc-rbf | $2.6$ | $\mathbf{2.2}$ | $4.3$ | $4.8$ | $4.5$ | $4.0$ | $5.6$
tree | $3.9$ | $5.1$ | $3.4$ | $\mathbf{2.1}$ | $2.4$ | $5.6$ | $5.6$
avg rank | 2.25 | 2.92 | 4.66 | 4.48 | 3.54 | 4.61 | 5.54

Table 7: Rankings for Brier score

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{1.6}$ | $3.0$ | $3.3$ | $3.6$ | $3.6$ | $6.3$ | $6.5$
forest | $4.4$ | $\mathbf{1.8}$ | $5.4$ | $1.9$ | $3.9$ | $5.3$ | $5.3$
knn | $3.9$ | $3.5$ | $5.3$ | $\mathbf{1.9}$ | $3.8$ | $4.8$ | $4.8$
lda | $\mathbf{1.8}$ | $3.2$ | $5.3$ | $2.2$ | $3.9$ | $6.0$ | $5.8$
logistic | $\mathbf{1.6}$ | $2.7$ | $6.1$ | $2.5$ | $4.3$ | $4.3$ | $6.4$
mlp | $3.0$ | $\mathbf{2.2}$ | $6.6$ | $2.8$ | $5.2$ | $3.9$ | $4.2$
nbayes | $\mathbf{1.2}$ | $3.5$ | $4.2$ | $2.3$ | $4.9$ | $5.7$ | $6.2$
qda | $\mathbf{1.9}$ | $2.9$ | $5.8$ | $2.1$ | $4.4$ | $5.1$ | $5.7$
svc-linear | $\mathbf{1.5}$ | $2.8$ | $6.5$ | $2.6$ | $4.6$ | $4.1$ | $5.8$
svc-rbf | $3.0$ | $\mathbf{2.5}$ | $4.7$ | $2.8$ | $4.7$ | $4.5$ | $5.8$
tree | $3.4$ | $4.2$ | $6.5$ | $4.7$ | $5.4$ | $\mathbf{1.9}$ | $2.0$
avg rank | 2.48 | 2.94 | 5.43 | 2.67 | 4.43 | 4.72 | 5.33

Table 8: Rankings for MCE

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{3.0}$ | $3.4$ | $3.5$ | $3.4$ | $3.6$ | $5.3$ | $5.9$
forest | $4.2$ | $\mathbf{3.2}$ | $4.8$ | $3.8$ | $3.4$ | $4.1$ | $4.5$
knn | $4.2$ | $4.7$ | $4.2$ | $3.7$ | $\mathbf{3.3}$ | $4.0$ | $4.0$
lda | $\mathbf{2.0}$ | $3.2$ | $4.8$ | $4.5$ | $4.0$ | $5.0$ | $4.5$
logistic | $3.4$ | $3.8$ | $5.4$ | $4.8$ | $\mathbf{2.5}$ | $3.5$ | $4.7$
mlp | $3.2$ | $4.2$ | $4.7$ | $4.6$ | $\mathbf{3.0}$ | $3.7$ | $4.5$
nbayes | $\mathbf{2.6}$ | $3.0$ | $3.6$ | $3.0$ | $4.2$ | $5.6$ | $5.9$
qda | $3.2$ | $\mathbf{2.2}$ | $5.1$ | $3.7$ | $4.2$ | $4.2$ | $5.4$
svc-linear | $\mathbf{2.8}$ | $4.2$ | $5.5$ | $3.7$ | $3.5$ | $4.1$ | $4.2$
svc-rbf | $5.0$ | $4.6$ | $3.9$ | $3.5$ | $\mathbf{3.3}$ | $3.5$ | $4.2$
tree | $4.3$ | $4.2$ | $4.5$ | $\mathbf{3.4}$ | $3.7$ | $4.0$ | $4.0$
avg rank | 3.44 | 3.73 | 4.53 | 3.83 | 3.50 | 4.27 | 4.71

Table 9: Rankings for confidence-ECE

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{1.7}$ | $2.7$ | $4.3$ | $2.7$ | $4.2$ | $6.0$ | $6.4$
forest | $4.2$ | $2.2$ | $5.7$ | $\mathbf{1.4}$ | $4.4$ | $5.1$ | $5.1$
knn | $3.0$ | $\mathbf{3.0}$ | $6.1$ | $3.5$ | $5.8$ | $3.3$ | $3.3$
lda | $\mathbf{2.0}$ | $2.9$ | $5.9$ | $2.1$ | $4.0$ | $5.7$ | $5.5$
logistic | $2.2$ | $3.0$ | $6.3$ | $\mathbf{1.9}$ | $4.7$ | $3.8$ | $6.1$
mlp | $3.5$ | $2.7$ | $6.6$ | $\mathbf{1.4}$ | $5.7$ | $4.0$ | $4.2$
nbayes | $\mathbf{2.1}$ | $2.8$ | $5.2$ | $2.4$ | $4.3$ | $5.3$ | $5.9$
qda | $3.1$ | $2.3$ | $6.5$ | $\mathbf{1.7}$ | $4.7$ | $4.6$ | $5.1$
svc-linear | $2.7$ | $2.8$ | $6.7$ | $\mathbf{2.0}$ | $4.9$ | $3.4$ | $5.5$
svc-rbf | $3.7$ | $3.4$ | $6.5$ | $2.9$ | $4.5$ | $\mathbf{2.7}$ | $4.3$
tree | $2.6$ | $3.6$ | $6.8$ | $4.8$ | $5.7$ | $\mathbf{2.2}$ | $2.3$
avg rank | 2.80 | 2.86 | 6.05 | 2.42 | 4.81 | 4.17 | 4.89

Table 10: Rankings for classwise-ECE

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{1.9}$ | $3.2$ | $4.3$ | $4.3$ | $4.1$ | $5.0$ | $5.1$
forest | $4.0$ | $2.1$ | $5.8$ | $\mathbf{1.1}$ | $4.0$ | $5.5$ | $5.4$
knn | $4.0$ | $3.9$ | $6.0$ | $3.6$ | $5.6$ | $2.5$ | $2.5$
lda | $2.4$ | $2.8$ | $5.8$ | $\mathbf{2.0}$ | $4.1$ | $5.3$ | $5.7$
logistic | $2.2$ | $2.5$ | $6.2$ | $\mathbf{2.0}$ | $4.4$ | $4.5$ | $6.1$
mlp | $3.0$ | $2.3$ | $6.6$ | $\mathbf{1.7}$ | $5.5$ | $4.3$ | $4.5$
nbayes | $\mathbf{1.9}$ | $3.5$ | $5.0$ | $2.5$ | $4.0$ | $5.4$ | $5.7$
qda | $2.7$ | $2.6$ | $6.4$ | $\mathbf{1.8}$ | $4.6$ | $5.0$ | $4.9$
svc-linear | $2.5$ | $2.6$ | $6.7$ | $\mathbf{2.5}$ | $4.6$ | $3.6$ | $5.5$
svc-rbf | $\mathbf{2.7}$ | $2.9$ | $6.5$ | $3.1$ | $4.5$ | $3.6$ | $4.7$
tree | $3.1$ | $4.1$ | $6.5$ | $4.7$ | $5.5$ | $\mathbf{1.9}$ | $2.0$
avg rank | 2.76 | 2.95 | 5.97 | 2.67 | 4.65 | 4.25 | 4.75

Table 11: Rankings for p-confidence-ECE

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{1.8}$ | $2.9$ | $4.3$ | $3.1$ | $4.4$ | $5.6$ | $5.7$
forest | $3.7$ | $2.2$ | $6.0$ | $\mathbf{1.9}$ | $4.7$ | $4.7$ | $4.9$
knn | $\mathbf{2.6}$ | $2.6$ | $5.7$ | $2.9$ | $5.1$ | $4.5$ | $4.6$
lda | $\mathbf{1.9}$ | $3.0$ | $6.1$ | $2.2$ | $3.9$ | $5.3$ | $5.5$
logistic | $2.6$ | $2.9$ | $6.3$ | $\mathbf{1.8}$ | $4.7$ | $3.7$ | $6.0$
mlp | $3.4$ | $2.8$ | $6.6$ | $\mathbf{2.2}$ | $5.7$ | $3.5$ | $3.9$
nbayes | $\mathbf{2.1}$ | $2.7$ | $5.2$ | $2.5$ | $4.5$ | $4.9$ | $6.1$
qda | $3.0$ | $2.1$ | $6.5$ | $\mathbf{2.1}$ | $4.6$ | $4.6$ | $5.2$
svc-linear | $2.5$ | $3.0$ | $6.6$ | $\mathbf{2.4}$ | $5.0$ | $3.3$ | $5.1$
svc-rbf | $3.4$ | $3.4$ | $6.3$ | $3.0$ | $5.0$ | $\mathbf{2.8}$ | $4.2$
tree | $\mathbf{2.5}$ | $3.7$ | $6.4$ | $4.3$ | $5.7$ | $2.6$ | $2.7$
avg rank | 2.69 | 2.87 | 6.00 | 2.58 | 4.85 | 4.11 | 4.90

Table 12: Rankings for p-classwise-ECE

| DirL2 | Beta | FreqB | Isot | WidB | TempS | Uncal
---|---|---|---|---|---|---|---
adas | $\mathbf{2.4}$ | $3.2$ | $4.1$ | $4.2$ | $3.9$ | $5.0$ | $5.2$
forest | $3.5$ | $\mathbf{2.3}$ | $5.7$ | $3.0$ | $3.6$ | $5.0$ | $5.0$
knn | $2.5$ | $4.0$ | $4.5$ | $\mathbf{2.1}$ | $3.2$ | $5.8$ | $6.0$
lda | $\mathbf{1.9}$ | $3.1$ | $5.8$ | $3.0$ | $3.5$ | $5.0$ | $5.8$
logistic | $\mathbf{2.2}$ | $2.8$ | $6.4$ | $3.0$ | $4.2$ | $3.9$ | $5.5$
mlp | $\mathbf{2.2}$ | $2.9$ | $6.7$ | $4.0$ | $5.2$ | $3.0$ | $4.1$
nbayes | $\mathbf{1.4}$ | $3.6$ | $4.8$ | $2.6$ | $4.2$ | $5.3$ | $6.1$
qda | $\mathbf{2.2}$ | $2.8$ | $6.3$ | $2.5$ | $3.8$ | $4.8$ | $5.6$
svc-linear | $\mathbf{2.3}$ | $2.7$ | $6.7$ | $3.8$ | $4.0$ | $3.7$ | $4.8$
svc-rbf | $\mathbf{2.9}$ | $3.0$ | $6.3$ | $3.5$ | $4.1$ | $3.9$ | $4.3$
tree | $\mathbf{2.4}$ | $4.3$ | $5.9$ | $4.2$ | $5.2$ | $3.0$ | $3.0$
avg rank | 2.34 | 3.15 | 5.73 | 3.27 | 4.11 | 4.37 | 5.02

###  E.2 Final critical difference diagrams for every metric

In order to perform a final comparison between calibration methods, we
considered every combination of dataset and classifier as a group
$n=\\#datasets\times\\#classifiers$, and ranked the results of the $k$
calibration methods. With this setting, we have performed the Friedman
statistical test followed by the one-tailed Bonferroni-Dunn test to obtain
critical differences (CDs) for every metric (See Figure 6). The results showed
Dirichlet L2 as the best calibration method for the measures accuracy, log-
loss and p-cw-ece with statistical significance (See Figures 6(a) 6(c), and
6(h)), and in the group of the best calibration methods in the rest of the
metrics with statistical significance, but no difference within the group. It
is worth mentioning that Figure 6(c) showed statistical difference between
Dirichlet L2, OvR Beta, OvR width binning, and the rest of the calibrators in
one group; in the mentioned order.

(a) Accuracy

(b) Brier score

(c) Log-loss score

(d) Maximum Calibration Error

(e) conf-ece

(f) cw-ece

(g) p-conf-ece

(h) p-cw-ece

Figure 6: Critical difference of the average of multiclass classifiers.

(a) p-conf-ece

(b) p-cw-ece

Figure 7: Proportion of times each calibrator passes a calibration p-test with
a p-value higher than 0.05.

###  E.3 Best calibrator hyperparameters

(a) Dirichlet L2 $\lambda$

(b) OvR Frequency binning #bins

(c) OvR Width binning #bins

Figure 8: Histogram of the selected hyperparameters during the inner 3-fold-
cross-validation

Figure 8 shows the best hyperparameters for every inner 3-fold-cross-
validation. Dirichlet L2 (Figure 8(a)) shows a preference for regularisation
hyperparameter $\lambda=1e^{-3}$ and lower values. Our current minimum
regularisation value of $1e^{-7}$ is also being selected multiple times,
indicating that lower values may be optimal in several occasions. However,
this fact did not seem to hurt the overall good results in our experiments.
One-vs.-Rest frequency binning tends to prefer $10$ bins of equal number of
samples, while One-vs.Rest width binning prefers $5$ equal sized bins (See
Figures 8(b) and 8(c) respectively).

###  E.4 Comparison of classifiers

In this Section we compare all the classifiers without post-hoc calibration on
$17$ of the datasets; from the total of $21$ datasets _shuttle_ , _yeast_ ,
_mfeat-karhunen_ and _libras-movement_ were removed from this analysis as at
least one classifier was not able to complete the experiment.

(a) Accuracy

(b) Log-loss

(c) Brier score

(d) Maximum Calibration Error

(e) conf-ece

(f) cw-ece

(g) p-conf-ece

(h) p-cw-ece

Figure 9: Critical difference of uncalibrated classifiers.

Figure 9 shows the Critical Difference diagram for all the $8$ metrics. In
particular, the MLP and the SVC with linear kernel are always in the group
with the higher rankings and never in the lowest. Similarly, random forest is
consistently in the best group, but in the worst group as well in $4$ of the
measures. SVC with radial basis kernel is in the best group $6$ times, but $3$
times in the worst. On the other hand, naive Bayes and Adaboost SAMME are
consistently in the worst group and never in the best one. The rest of the
classifiers did not show a clear ranking position.

(a) p-conf-ece

(b) p-cw-ece

Figure 10: Proportion of times each classifier is already calibrated with
different p-tests.

Figures 10(b) and 10(a) show the proportion of times each classifier passed
the p-conf-ECE and p-cw-ECE statistical test for all datasets and cross-
validation folds.

###  E.5 Deep neural networks

In this section, we provide further discussion about results from the deep
networks experiments. These are given in the form of critical difference
diagrams (Figure 11) and tables (Tables E.5-E.5) both including the following
measures: error rate, log-loss, Brier score, maximum calibration error (MCE),
confidence-ECE (conf-ECE), classwise-ECE (cw-ECE), as well as significance
measures p-conf-ECE and p-cw-ECE.

In addition, Table E.5 compares MS-ODIR and vector scaling on log-loss. On the
table, we also added MS-ODIR-zero which was obtained from the respective MS-
ODIR model by replacing the off-diagonal entries with zeroes. Each experiment
is replicated three times with different splits on datasets. This is done to
compare the stability of the methods. In each replication, the best scoring
model is written in bold.

Finally, Figure 12 shows that temperature scaling systematically under-
estimates class 4 probabilities on the model c10_resnet_wide32 on CIFAR-10.

(a) Error Rate

(b) Brier score

(c) Log-loss score

(d) Maximum calibration error

(e) conf-ece

(f) cw-ece

(g) p-conf-ece

(h) p-cw-ece

Figure 11: Critical difference of the deep neural networks. Figure 12: Reliability diagrams of c10_resnet_wide32 on CIFAR-10: (a) classwise-reliability for class 4 after temperature scaling; (b) classwise-reliability for class 4 after Dirichlet calibration. Table 13: Scores and ranking of calibration methods for log-loss. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.39098_{6}$ | $\mathbf{0.19497_{1}}$ | $0.19692_{4}$ | $0.19536_{2}$ | $0.19743_{5}$ | $0.19634_{3}$
c10_densenet40 | $0.42821_{6}$ | $0.22509_{5}$ | $\mathbf{0.22048_{1}}$ | $0.22371_{4}$ | $0.22270_{3}$ | $0.22240_{2}$
c10_lenet5 | $0.82326_{6}$ | $0.80031_{5}$ | $0.74418_{2}$ | $0.74441_{3}$ | $0.74704_{4}$ | $\mathbf{0.74262_{1}}$
c10_resnet110 | $0.35827_{6}$ | $0.20926_{5}$ | $\mathbf{0.20303_{1}}$ | $0.20511_{3}$ | $0.20595_{4}$ | $0.20375_{2}$
c10_resnet110_SD | $0.30325_{6}$ | $0.17760_{5}$ | $0.17694_{4}$ | $0.17608_{3}$ | $0.17549_{2}$ | $\mathbf{0.17537_{1}}$
c10_resnet_wide32 | $0.38170_{6}$ | $0.19148_{5}$ | $0.18464_{4}$ | $0.18203_{2}$ | $0.18276_{3}$ | $\mathbf{0.18165_{1}}$
c100_convnet | $1.64120_{6}$ | $\mathbf{0.94162_{1}}$ | $1.18945_{5}$ | $0.96121_{2}$ | $0.96369_{4}$ | $0.96141_{3}$
c100_densenet40 | $2.01740_{6}$ | $1.05713_{2}$ | $1.25293_{5}$ | $1.05909_{4}$ | $1.05831_{3}$ | $\mathbf{1.05084_{1}}$
c100_lenet5 | $2.78365_{6}$ | $2.64979_{5}$ | $2.59482_{4}$ | $2.48951_{2}$ | $2.51590_{3}$ | $\mathbf{2.48670_{1}}$
c100_resnet110 | $1.69371_{6}$ | $1.09169_{3}$ | $1.21239_{5}$ | $1.09607_{4}$ | $1.08916_{2}$ | $\mathbf{1.07370_{1}}$
c100_resnet110_SD | $1.35250_{6}$ | $0.94214_{3}$ | $1.19837_{5}$ | $0.94477_{4}$ | $\mathbf{0.92341_{1}}$ | $0.92731_{2}$
c100_resnet_wide32 | $1.80215_{6}$ | $0.94453_{3}$ | $1.08711_{5}$ | $0.95288_{4}$ | $0.93650_{2}$ | $\mathbf{0.93273_{1}}$
SVHN_convnet | $0.20460_{6}$ | $0.15142_{5}$ | $0.14246_{3}$ | $0.13791_{2}$ | $0.14388_{4}$ | $\mathbf{0.13760_{1}}$
SVHN_resnet152_SD | $0.08542_{6}$ | $\mathbf{0.07861_{1}}$ | $0.08463_{5}$ | $0.08038_{2}$ | $0.08124_{4}$ | $0.08100_{3}$
avg rank | 6.0 | 3.5 | 3.79 | 2.93 | 3.14 | 1.64
Table 14: Scores and ranking of calibration methods for Brier score. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.01090_{6}$ | $\mathbf{0.00952_{1}}$ | $0.00969_{5}$ | $0.00955_{3}$ | $0.00958_{4}$ | $0.00953_{2}$
c10_densenet40 | $0.01274_{6}$ | $0.01100_{4}$ | $0.01102_{5}$ | $0.01097_{2}$ | $0.01097_{3}$ | $\mathbf{0.01097_{1}}$
c10_lenet5 | $0.03788_{6}$ | $0.03748_{5}$ | $0.03510_{2}$ | $0.03511_{3}$ | $0.03523_{4}$ | $\mathbf{0.03502_{1}}$
c10_resnet110 | $0.01102_{6}$ | $0.00979_{4}$ | $0.00979_{5}$ | $0.00977_{2}$ | $0.00978_{3}$ | $\mathbf{0.00976_{1}}$
c10_resnet110_SD | $0.00981_{6}$ | $0.00874_{4}$ | $0.00877_{5}$ | $0.00867_{3}$ | $0.00867_{2}$ | $\mathbf{0.00866_{1}}$
c10_resnet_wide32 | $0.01047_{6}$ | $0.00924_{5}$ | $0.00909_{4}$ | $\mathbf{0.00888_{1}}$ | $0.00891_{3}$ | $0.00889_{2}$
c100_convnet | $0.00425_{5}$ | $\mathbf{0.00358_{1}}$ | $0.00441_{6}$ | $0.00358_{2}$ | $0.00362_{4}$ | $0.00361_{3}$
c100_densenet40 | $0.00491_{6}$ | $0.00401_{3}$ | $0.00468_{5}$ | $0.00400_{2}$ | $0.00403_{4}$ | $\mathbf{0.00400_{1}}$
c100_lenet5 | $0.00813_{6}$ | $0.00792_{5}$ | $0.00786_{4}$ | $0.00760_{2}$ | $0.00767_{3}$ | $\mathbf{0.00760_{1}}$
c100_resnet110 | $0.00453_{6}$ | $0.00392_{3}$ | $0.00438_{5}$ | $0.00391_{2}$ | $0.00393_{4}$ | $\mathbf{0.00391_{1}}$
c100_resnet110_SD | $0.00418_{5}$ | $0.00367_{4}$ | $0.00456_{6}$ | $0.00364_{3}$ | $\mathbf{0.00360_{1}}$ | $0.00361_{2}$
c100_resnet_wide32 | $0.00432_{6}$ | $0.00355_{4}$ | $0.00401_{5}$ | $0.00354_{3}$ | $0.00352_{2}$ | $\mathbf{0.00351_{1}}$
SVHN_convnet | $0.00776_{6}$ | $0.00598_{5}$ | $0.00555_{3}$ | $\mathbf{0.00530_{1}}$ | $0.00561_{4}$ | $0.00532_{2}$
SVHN_resnet152_SD | $0.00297_{3}$ | $\mathbf{0.00291_{1}}$ | $0.00305_{6}$ | $0.00293_{2}$ | $0.00299_{5}$ | $0.00298_{4}$
avg rank | 5.64 | 3.5 | 4.71 | 2.21 | 3.29 | 1.64
Table 15: Scores and ranking of calibration methods for confidence-ECE. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.04760_{6}$ | $0.01065_{5}$ | $0.00769_{2}$ | $0.00960_{4}$ | $\mathbf{0.00740_{1}}$ | $0.00782_{3}$
c10_densenet40 | $0.05500_{6}$ | $0.00946_{2}$ | $\mathbf{0.00568_{1}}$ | $0.01097_{5}$ | $0.01018_{4}$ | $0.00988_{3}$
c10_lenet5 | $0.05180_{6}$ | $0.01665_{5}$ | $0.01383_{3}$ | $0.01367_{2}$ | $\mathbf{0.01310_{1}}$ | $0.01468_{4}$
c10_resnet110 | $0.04750_{6}$ | $0.01132_{5}$ | $\mathbf{0.00680_{1}}$ | $0.01086_{3}$ | $0.01130_{4}$ | $0.01059_{2}$
c10_resnet110_SD | $0.04113_{6}$ | $\mathbf{0.00555_{1}}$ | $0.00646_{4}$ | $0.00815_{5}$ | $0.00579_{3}$ | $0.00566_{2}$
c10_resnet_wide32 | $0.04505_{6}$ | $0.00784_{4}$ | $\mathbf{0.00524_{1}}$ | $0.00837_{5}$ | $0.00769_{3}$ | $0.00727_{2}$
c100_convnet | $0.17614_{6}$ | $\mathbf{0.01367_{1}}$ | $0.14347_{5}$ | $0.02069_{3}$ | $0.01965_{2}$ | $0.02660_{4}$
c100_densenet40 | $0.21156_{6}$ | $\mathbf{0.00902_{1}}$ | $0.12380_{5}$ | $0.01138_{2}$ | $0.01224_{3}$ | $0.02197_{4}$
c100_lenet5 | $0.12125_{6}$ | $0.01499_{4}$ | $0.01369_{2}$ | $0.02003_{5}$ | $\mathbf{0.01294_{1}}$ | $0.01407_{3}$
c100_resnet110 | $0.18480_{6}$ | $\mathbf{0.02380_{1}}$ | $0.14535_{5}$ | $0.02822_{4}$ | $0.02693_{2}$ | $0.02735_{3}$
c100_resnet110_SD | $0.15861_{5}$ | $\mathbf{0.01214_{1}}$ | $0.15920_{6}$ | $0.02283_{4}$ | $0.01296_{2}$ | $0.02246_{3}$
c100_resnet_wide32 | $0.18784_{6}$ | $\mathbf{0.01472_{1}}$ | $0.13509_{5}$ | $0.01891_{3}$ | $0.01718_{2}$ | $0.02581_{4}$
SVHN_convnet | $0.07755_{6}$ | $0.01179_{4}$ | $0.01910_{5}$ | $0.00997_{2}$ | $\mathbf{0.00934_{1}}$ | $0.01037_{3}$
SVHN_resnet152_SD | $0.00862_{6}$ | $0.00607_{4}$ | $0.00691_{5}$ | $\mathbf{0.00582_{1}}$ | $0.00595_{2}$ | $0.00604_{3}$
avg rank | 5.93 | 2.79 | 3.57 | 3.43 | 2.21 | 3.07
Table 16: Scores and ranking of calibration methods for classwise-ECE. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.10375_{6}$ | $0.04423_{4}$ | $0.04262_{2}$ | $0.04507_{5}$ | $\mathbf{0.04259_{1}}$ | $0.04352_{3}$
c10_densenet40 | $0.11430_{6}$ | $0.03977_{5}$ | $\mathbf{0.03412_{1}}$ | $0.03687_{4}$ | $0.03609_{2}$ | $0.03678_{3}$
c10_lenet5 | $0.19849_{6}$ | $0.17141_{5}$ | $\mathbf{0.05185_{1}}$ | $0.05891_{4}$ | $0.05705_{2}$ | $0.05862_{3}$
c10_resnet110 | $0.09846_{6}$ | $0.04344_{5}$ | $\mathbf{0.03206_{1}}$ | $0.03950_{4}$ | $0.03653_{3}$ | $0.03615_{2}$
c10_resnet110_SD | $0.08647_{6}$ | $0.03071_{4}$ | $0.03148_{5}$ | $0.02937_{3}$ | $0.02713_{2}$ | $\mathbf{0.02681_{1}}$
c10_resnet_wide32 | $0.09530_{6}$ | $0.04775_{5}$ | $0.03153_{3}$ | $0.02947_{2}$ | $0.03164_{4}$ | $\mathbf{0.02921_{1}}$
c100_convnet | $0.42414_{6}$ | $\mathbf{0.22683_{1}}$ | $0.40185_{5}$ | $0.24041_{3}$ | $0.24063_{4}$ | $0.23958_{2}$
c100_densenet40 | $0.47026_{6}$ | $0.18664_{2}$ | $0.32985_{5}$ | $\mathbf{0.18630_{1}}$ | $0.18879_{3}$ | $0.19112_{4}$
c100_lenet5 | $0.47264_{6}$ | $0.38481_{5}$ | $0.21865_{4}$ | $0.21348_{2}$ | $\mathbf{0.20293_{1}}$ | $0.21379_{3}$
c100_resnet110 | $0.41644_{6}$ | $0.20095_{3}$ | $0.35885_{5}$ | $\mathbf{0.18639_{1}}$ | $0.19442_{2}$ | $0.20270_{4}$
c100_resnet110_SD | $0.37518_{6}$ | $0.20310_{4}$ | $0.37346_{5}$ | $0.18895_{3}$ | $\mathbf{0.17015_{1}}$ | $0.18552_{2}$
c100_resnet_wide32 | $0.42027_{6}$ | $0.18573_{4}$ | $0.33258_{5}$ | $0.17951_{2}$ | $\mathbf{0.17082_{1}}$ | $0.17966_{3}$
SVHN_convnet | $0.15935_{6}$ | $0.03830_{4}$ | $0.04276_{5}$ | $0.02638_{2}$ | $\mathbf{0.02480_{1}}$ | $0.02750_{3}$
SVHN_resnet152_SD | $0.01940_{2}$ | $\mathbf{0.01849_{1}}$ | $0.02184_{6}$ | $0.01988_{3}$ | $0.02120_{5}$ | $0.02088_{4}$
avg rank | 5.71 | 3.71 | 3.79 | 2.79 | 2.29 | 2.71
Table 17: Scores and ranking of calibration methods for MCE. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.59173_{6}$ | $0.23150_{4}$ | $0.12432_{2}$ | $0.24830_{5}$ | $0.12831_{3}$ | $\mathbf{0.07621_{1}}$
c10_densenet40 | $0.33396_{6}$ | $0.09929_{2}$ | $0.11679_{4}$ | $\mathbf{0.07858_{1}}$ | $0.12046_{5}$ | $0.11297_{3}$
c10_lenet5 | $0.11281_{6}$ | $0.09158_{3}$ | $\mathbf{0.05112_{1}}$ | $0.09009_{2}$ | $0.09996_{4}$ | $0.10061_{5}$
c10_resnet110 | $0.29580_{6}$ | $0.23639_{4}$ | $0.24405_{5}$ | $\mathbf{0.08331_{1}}$ | $0.13130_{2}$ | $0.22678_{3}$
c10_resnet110_SD | $0.32484_{6}$ | $\mathbf{0.07823_{1}}$ | $0.23064_{5}$ | $0.13309_{3}$ | $0.14276_{4}$ | $0.08422_{2}$
c10_resnet_wide32 | $0.37215_{4}$ | $\mathbf{0.07060_{1}}$ | $0.49283_{6}$ | $0.41567_{5}$ | $0.26539_{3}$ | $0.26372_{2}$
c100_convnet | $0.36391_{6}$ | $0.13689_{4}$ | $0.23333_{5}$ | $0.07235_{2}$ | $\mathbf{0.07043_{1}}$ | $0.08171_{3}$
c100_densenet40 | $0.45400_{6}$ | $\mathbf{0.02213_{1}}$ | $0.19748_{5}$ | $0.04074_{2}$ | $0.04293_{3}$ | $0.05004_{4}$
c100_lenet5 | $0.20097_{6}$ | $0.05836_{3}$ | $\mathbf{0.05678_{1}}$ | $0.06774_{4}$ | $0.05749_{2}$ | $0.08939_{5}$
c100_resnet110 | $0.39882_{6}$ | $0.07099_{2}$ | $0.20732_{5}$ | $0.08026_{4}$ | $0.07354_{3}$ | $\mathbf{0.06678_{1}}$
c100_resnet110_SD | $0.48291_{6}$ | $0.04099_{2}$ | $0.24578_{5}$ | $0.05979_{3}$ | $\mathbf{0.04038_{1}}$ | $0.06612_{4}$
c100_resnet_wide32 | $0.45639_{6}$ | $\mathbf{0.03606_{1}}$ | $0.19370_{5}$ | $0.05521_{2}$ | $0.06605_{4}$ | $0.06468_{3}$
SVHN_convnet | $0.30011_{5}$ | $0.40691_{6}$ | $\mathbf{0.16154_{1}}$ | $0.18458_{3}$ | $0.16312_{2}$ | $0.18588_{4}$
SVHN_resnet152_SD | $0.25032_{5}$ | $\mathbf{0.18244_{1}}$ | $0.23895_{4}$ | $0.19649_{2}$ | $0.23092_{3}$ | $0.80082_{6}$
avg rank | 5.71 | 2.5 | 3.86 | 2.79 | 2.86 | 3.29
Table 18: Scores and ranking of calibration methods for error rate (%). |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $6.18000_{2}$ | $6.18000_{2}$ | $6.38000_{6}$ | $\mathbf{6.12000_{1}}$ | $6.36000_{5}$ | $6.32000_{4}$
c10_densenet40 | $7.58000_{5}$ | $7.58000_{5}$ | $\mathbf{7.49000_{1}}$ | $7.53000_{4}$ | $7.52000_{3}$ | $7.50000_{2}$
c10_lenet5 | $27.26000_{5}$ | $27.26000_{5}$ | $\mathbf{25.25000_{1}}$ | $25.44000_{2}$ | $25.49000_{3}$ | $25.50000_{4}$
c10_resnet110 | $6.44000_{1}$ | $6.44000_{1}$ | $6.54000_{6}$ | $6.49000_{4}$ | $6.47000_{3}$ | $6.49000_{4}$
c10_resnet110_SD | $5.96000_{5}$ | $5.96000_{5}$ | $5.90000_{4}$ | $\mathbf{5.77000_{1}}$ | $5.83000_{3}$ | $5.81000_{2}$
c10_resnet_wide32 | $6.07000_{5}$ | $6.07000_{5}$ | $5.94000_{4}$ | $5.76000_{2}$ | $\mathbf{5.74000_{1}}$ | $5.81000_{3}$
c100_convnet | $26.12000_{1}$ | $26.12000_{1}$ | $30.96000_{6}$ | $26.22000_{3}$ | $26.56000_{4}$ | $26.60000_{5}$
c100_densenet40 | $30.00000_{3}$ | $30.00000_{3}$ | $33.48000_{6}$ | $29.87000_{2}$ | $30.16000_{5}$ | $\mathbf{29.61000_{1}}$
c100_lenet5 | $66.41000_{5}$ | $66.41000_{5}$ | $65.97000_{4}$ | $62.53000_{2}$ | $63.59000_{3}$ | $\mathbf{62.44000_{1}}$
c100_resnet110 | $28.52000_{4}$ | $28.52000_{4}$ | $30.04000_{6}$ | $\mathbf{28.36000_{1}}$ | $28.40000_{2}$ | $28.45000_{3}$
c100_resnet110_SD | $27.17000_{4}$ | $27.17000_{4}$ | $31.43000_{6}$ | $26.96000_{3}$ | $26.50000_{2}$ | $\mathbf{26.42000_{1}}$
c100_resnet_wide32 | $26.18000_{4}$ | $26.18000_{4}$ | $27.69000_{6}$ | $26.07000_{2}$ | $26.08000_{3}$ | $\mathbf{26.06000_{1}}$
SVHN_convnet | $3.82750_{5}$ | $3.82750_{5}$ | $3.42811_{3}$ | $\mathbf{3.34728_{1}}$ | $3.51845_{4}$ | $3.37105_{2}$
SVHN_resnet152_SD | $1.84773_{2}$ | $1.84773_{2}$ | $1.90535_{6}$ | $\mathbf{1.80547_{1}}$ | $1.87462_{4}$ | $1.87462_{4}$
avg rank | 4.14 | 4.14 | 4.64 | 2.11 | 3.25 | 2.71
Table 19: Scores and ranking of calibration methods for p-confidence-ECE. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.0_{6}$ | $0.032_{4}$ | $0.363_{2}$ | $0.019_{5}$ | $\mathbf{0.461_{1}}$ | $0.052_{3}$
c10_densenet40 | $0.0_{4}$ | $0.002_{2}$ | $\mathbf{0.525_{1}}$ | $0.000_{4}$ | $0.000_{4}$ | $0.000_{4}$
c10_lenet5 | $0.0_{6}$ | $0.008_{5}$ | $0.027_{4}$ | $0.084_{3}$ | $\mathbf{0.155_{1}}$ | $0.144_{2}$
c10_resnet110 | $0.0_{4}$ | $0.000_{4}$ | $\mathbf{0.246_{1}}$ | $0.000_{4}$ | $0.000_{4}$ | $0.000_{4}$
c10_resnet110_SD | $0.0_{6}$ | $0.105_{4}$ | $\mathbf{0.179_{1}}$ | $0.003_{5}$ | $0.114_{3}$ | $0.124_{2}$
c10_resnet_wide32 | $0.0_{6}$ | $0.017_{3}$ | $\mathbf{0.281_{1}}$ | $0.005_{4}$ | $0.005_{4}$ | $0.076_{2}$
c100_convnet | $0.0_{5}$ | $\mathbf{0.174_{1}}$ | $0.000_{5}$ | $0.049_{2}$ | $0.021_{3}$ | $0.000_{5}$
c100_densenet40 | $0.0_{5}$ | $\mathbf{0.817_{1}}$ | $0.000_{5}$ | $0.617_{2}$ | $0.238_{3}$ | $0.000_{5}$
c100_lenet5 | $0.0_{6}$ | $0.153_{4}$ | $0.217_{3}$ | $0.001_{5}$ | $0.395_{2}$ | $\mathbf{0.422_{1}}$
c100_resnet110 | $0.0_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$
c100_resnet110_SD | $0.0_{4}$ | $0.009_{2}$ | $0.000_{4}$ | $0.000_{4}$ | $\mathbf{0.060_{1}}$ | $0.000_{4}$
c100_resnet_wide32 | $0.0_{4}$ | $\mathbf{0.022_{1}}$ | $0.000_{4}$ | $0.000_{4}$ | $0.001_{2}$ | $0.000_{4}$
mnist_mlp | $0.0_{6}$ | $0.616_{3}$ | $\mathbf{0.948_{1}}$ | $0.486_{4}$ | $0.455_{5}$ | $0.677_{2}$
SVHN_convnet | $0.0_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$
SVHN_resnet152_SD | $0.0_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$ | $0.000_{3}$
avg rank | 4.93 | 2.97 | 2.9 | 3.9 | 2.97 | 3.33
Table 20: Scores and ranking of calibration methods for p-classwise-ECE. |  | general-purpose calibrators | calibrators using logits
---|---|---|---
| Uncal | TempS | Dir-L2 | Dir-ODIR | VecS | MS-ODIR
c10_convnet | $0.0_{6}$ | $0.0104_{4}$ | $\mathbf{0.1276_{1}}$ | $0.0038_{5}$ | $0.0340_{2}$ | $0.0114_{3}$
c10_densenet40 | $0.0_{4}$ | $0.0000_{4}$ | $\mathbf{0.0093_{1}}$ | $0.0000_{4}$ | $0.0000_{4}$ | $0.0000_{4}$
c10_lenet5 | $0.0_{5}$ | $0.0000_{5}$ | $\mathbf{0.6014_{1}}$ | $0.0390_{4}$ | $0.1230_{2}$ | $0.0501_{3}$
c10_resnet110 | $0.0_{4}$ | $0.0000_{4}$ | $\mathbf{0.0088_{1}}$ | $0.0000_{4}$ | $0.0000_{4}$ | $0.0000_{4}$
c10_resnet110_SD | $0.0_{6}$ | $0.0058_{5}$ | $0.0105_{3}$ | $0.0077_{4}$ | $0.1816_{2}$ | $\mathbf{0.2196_{1}}$
c10_resnet_wide32 | $0.0_{5}$ | $0.0000_{5}$ | $0.0096_{3}$ | $0.0158_{2}$ | $0.0006_{4}$ | $\mathbf{0.0249_{1}}$
c100_convnet | $0.0_{4}$ | $\mathbf{0.0770_{1}}$ | $0.0000_{4}$ | $0.0000_{4}$ | $0.0000_{4}$ | $0.0000_{4}$
c100_densenet40 | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
c100_lenet5 | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
c100_resnet110 | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
c100_resnet110_SD | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
c100_resnet_wide32 | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
mnist_mlp | $0.0_{6}$ | $\mathbf{0.5669_{1}}$ | $0.0842_{3}$ | $0.0022_{5}$ | $0.0280_{4}$ | $0.1178_{2}$
SVHN_convnet | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
SVHN_resnet152_SD | $0.0_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$ | $0.0000_{3}$
avg rank | 4.37 | 3.63 | 2.77 | 3.77 | 3.37 | 3.1
Table 21: Comparison of MS-ODIR and vector scaling for log-loss. | Replication 1 | Replication 2 | Replication 3
---|---|---|---
| VecS | MS-ODIR | MS-ODIR-zero | VecS | MS-ODIR | MS-ODIR-zero | VecS | MS-ODIR | MS-ODIR-zero
c10_convnet | $0.19774$ | $\mathbf{0.19632}$ | $0.19632$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
c10_densenet40 | $\mathbf{0.22240}$ | $0.22240$ | $0.22240$ | $0.21316$ | $\mathbf{0.21186}$ | $0.21366$ | $0.21350$ | $\mathbf{0.21325}$ | $0.21327$
c10_lenet5 | $0.74688$ | $\mathbf{0.74262}$ | $0.74830$ | $0.69392$ | $\mathbf{0.69287}$ | $0.69335$ | $\mathbf{0.67955}$ | $0.67974$ | $0.68127$
c10_resnet110 | $0.20624$ | $\mathbf{0.20375}$ | $0.20537$ | $0.20064$ | $\mathbf{0.19803}$ | $0.20040$ | $0.19655$ | $\mathbf{0.19536}$ | $0.19739$
c10_resnet110_SD | $0.17545$ | $\mathbf{0.17537}$ | $0.17539$ | $0.18123$ | $\mathbf{0.18094}$ | $0.18097$ | $\mathbf{0.17799}$ | $0.17829$ | $0.17829$
c10_resnet_wide32 | $0.18274$ | $\mathbf{0.18165}$ | $0.18302$ | $0.18522$ | $\mathbf{0.18364}$ | $0.18546$ | $0.17431$ | $\mathbf{0.17274}$ | $0.17448$
c100_convnet | $0.96311$ | $\mathbf{0.96141}$ | $0.96149$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
c100_densenet40 | $1.05714$ | $\mathbf{1.05084}$ | $1.06804$ | $1.06366$ | $\mathbf{1.05456}$ | $1.07107$ | $1.07704$ | $\mathbf{1.06918}$ | $1.08559$
c100_lenet5 | $2.51695$ | $\mathbf{2.48670}$ | $2.57932$ | $2.21546$ | $\mathbf{2.20054}$ | $2.22360$ | $2.28054$ | $\mathbf{2.27887}$ | $2.29485$
c100_resnet110 | $1.08824$ | $\mathbf{1.07370}$ | $1.10137$ | $1.09066$ | $\mathbf{1.08267}$ | $1.11116$ | $1.11977$ | $\mathbf{1.10672}$ | $1.13900$
c100_resnet110_SD | $\mathbf{0.92275}$ | $0.92731$ | $0.92730$ | $0.87758$ | $\mathbf{0.87698}$ | $0.87701$ | $\mathbf{0.88523}$ | $0.88731$ | $0.88727$
c100_resnet_wide32 | $0.93724$ | $\mathbf{0.93273}$ | $0.94060$ | $0.93291$ | $\mathbf{0.92531}$ | $0.94854$ | $0.93183$ | $\mathbf{0.92439}$ | $0.94568$
SVHN_convnet | $0.14392$ | $\mathbf{0.13760}$ | $0.14507$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
SVHN_resnet152_SD | $0.08131$ | $\mathbf{0.08100}$ | $0.08100$ | $0.12728$ | $\mathbf{0.12723}$ | $0.12639$ | $0.12559$ | $\mathbf{0.12453}$ | $0.12381$