# New energy spectra in neutrino and photon detectors to reveal hidden dark
matter signals

Wim Beenakker<EMAIL_ADDRESS>Institute for Mathematics,
Astrophysics and Particle Physics, Radboud University Nijmegen, Heyendaalseweg
135, Nijmegen, the Netherlands  Institute of Physics, University of Amsterdam,
Science Park 904, 1018 XE Amsterdam, The Netherlands  Sascha Caron
<EMAIL_ADDRESS>Institute of Physics, University of Amsterdam, Science Park
904, 1018 XE Amsterdam, The Netherlands  Nikhef, Science Park, Amsterdam, The
Netherlands  Jochem Kip<EMAIL_ADDRESS>Institute for Mathematics,
Astrophysics and Particle Physics, Radboud University Nijmegen, Heyendaalseweg
135, Nijmegen, the Netherlands  Roberto Ruiz de Austri<EMAIL_ADDRESS>Instituto de Física Corpuscular, CSIC-Universitat de València, E-46980
Paterna, Valencia, Spain  Zhongyi Zhang<EMAIL_ADDRESS>Institute for
Mathematics, Astrophysics and Particle Physics, Radboud University Nijmegen,
Heyendaalseweg 135, Nijmegen, the Netherlands  Nikhef, Science Park,
Amsterdam, The Netherlands

###### Abstract

Neutral particles capable of travelling cosmic distances from a source to
detectors on Earth are limited to photons and neutrinos. Examination of the
Dark Matter annihilation/decay spectra for these particles reveals the
presence of continuum spectra (e.g. due to fragmentation and W or Z decay) and
peaks (due to direct annihilations/decays). However, when one explores
extensions of the Standard Model (BSM), unexplored spectra emerge that differ
significantly from those of the Standard Model (SM) for both neutrinos and
photons. In this paper, we argue for the inclusion of important spectra that
include peaks as well as previously largely unexplored entities such as boxes
and combinations of box, peak and continuum decay spectra.

##  1 Introduction

The search for Dark Matter (DM) by indirect detection is the subject of many
studies. A large number of experiments have investigated the cosmic
antiproton, positron, photon and neutrino spectra. Notable experiments include
but are not limited to, AMS-02 [1], Fermi-LAT [2], Icecube [3], ANTARES [4],
H.E.S.S. [5], Pierre Auger [6], and VERITAS [7] which have been measuring
charged cosmic rays, gamma rays, and neutrinos for decades. With the upcoming
construction of new experiments such as KM3Net [8], the CTA observatory [9],
and GRAND [10] the sensitivity to potential neutral particles arising from DM
annihilation or decay will increase significantly.
Historically the investigated particle spectra resulting from annihilating or
decaying DM have focussed on processes involving two Standard Model (SM) final
state particles, e.g. DM DM $\to b\bar{b}$, DM DM $\to\tau^{+}\tau^{-}$, etc,
which subsequently undergo decay, fragmentation and hadronization to produce
antiproton, positron, photon, and neutrino spectra. These spectra are produced
by well-understood mechanisms, leading to detailed analyses that can produce
strong limits on the properties of DM. Some notable examples are the potential
AMS-02 antiproton excess and the Fermi LAT gamma-ray excess, both of which
have been explained by DM annihilation with a DM mass in the
$\mathcal{O}$(100) GeV region [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26, 27]. However, if one considers possible extensions beyond the
standard model (BSM), other, still largely unexplored spectra are possible,
which may differ considerably from the known standard spectra.
In this paper, we describe a new type of neutrino and photon spectrum in a
largely model-independent way. It contains a combination of a well-defined
peak, a box and a combination of neutrino/photon spectra produced by the
decay, fragmentation and hadronization of SM particles. Of course, a clearly
defined peak has been investigated by Icecube [28], ANTARES [29] and Fermi-LAT
[30] among others, as it is easily obtained by DM particles directly
annihilating or decaying into the relevant final state resulting in a
comparatively clean signal. However, a box shape and, of course, the
combination of all three features leads to a significantly different spectrum.
These types of spectra have been largely overlooked in both experimental and
phenomenological research. To facilitate the search for these spectra, we
provide a code to obtain a user-defined non-standard neutrino or photon
spectrum by specifying the appropriate parameters.
This paper is structured as follows. First, we detail the physics of the non-
standard spectra and provide the relevant expressions of the kinematics. Next,
some example BSM models are given that can produce non-standard neutrino or
photon spectra. Then we detail the working of the sampling code and its
verification, after which we provide some elementary parameter sets that
capture the most important features of the spectra in order to facilitate
experimental searches. We end with our conclusions.

##  2 Theoretical background

There are only two types of neutral particles that can travel cosmic distances
from a source to detectors on or near Earth, namely photons and neutrinos. In
the following subsections, the kinematics of the spectrum of a particle will
be discussed as model-independently as possible. Both neutral particles can be
described by the same kinematics, since they are both massless or have
negligible mass. The main differences, of course, lie in the possible models
that can produce such spectra. However, no such assumptions will be made in
the following subsections.
Moreover, both DM annihilation and DM decay can produce cosmic rays. The
kinematics for DM decay is identical to DM annihilation, with the only
difference being that the initial energy for (non-relativistic) DM decay is
$M_{\rm DM}$, while for DM annihilation it is $2M_{\rm DM}$. Thus, to obtain
the kinematic expressions for the DM decay from those for the DM annihilation,
one simply has to replace $M_{\rm DM}$ by $\frac{1}{2}M_{\rm DM}$
In the following subsections, we assume the standard scenario that two DM
particles annihilate to neutrinos in order to simplify the discussion.

###  2.1 The box

The simplest non-standard spectrum arises from two DM particles annihilating
into two BSM particles, $X$, which subsequently decay into neutrinos:

$\displaystyle{\rm DM}\,{\rm DM}\to XX$ $\displaystyle X\to\nu\nu\,,$ (1)

where $\nu$ is any SM neutrino. More complicated decays of $X$ are of course
possible, which will be discussed in section 2.3. Depending on the model, of
course, the $XX$-pair can also be a $X\overline{X}$-pair, and any $\nu$ may
well be an $\overline{\nu}$. This kind of DM annihilation leads to a ’box’
shape of the neutrino spectrum. In the rest frame of the particle, $X$ both
neutrinos produced by the decay of $X$ have a clearly defined energy of
$M_{X}/2$, where $M_{X}$ is the mass of $X$. However, the neutrinos need to be
Lorentz boosted into the center-of-mass frame of the annihilating DM particles
as they are now evaluated in the rest frame of the particle $X$, which of
course differs from the center-of-mass frame. Assuming that ${\rm DM}$ has
zero momentum at annihilation, using spherical coordinates in the rest frame
of $X$, and choosing the motion of $X$ in the $z$ direction and thus boosted
in the $z$ direction, the energy of the neutrinos results in:

$\displaystyle
E_{\text{box}}=\frac{M_{X}}{2}\left(\cosh{(\eta)}+\sinh{(\eta)}\cos{(\theta)}\right)$
$\displaystyle\eta=\cosh^{-1}{\left(\frac{M_{\rm DM}}{M_{X}}\right)}\,.$ (2)

Here $M_{\rm DM}$ is the DM mass. The $\cos(\theta)$ term is due to the
spherical coordinates in the rest frame of $X$. To obtain a uniform
distribution of points on a sphere, i.e. an isotropic decay, $\cos(\theta)$
must be sampled uniformly between 0 and 1, as opposed to sampling $\theta$
uniformly between 0 and $2\pi$111See
http://corysimon.github.io/articles/uniformdistn-on-sphere/ for an in-depth
explanation.. This uniform distribution in $\cos(\theta)$ results in the
neutrino having an equal probability of having any energy within the bounds of
$\cos(\theta)=-1$ and $+1$, thereby resulting in a flat ’box’ shape.
However, a particle $X$ will in general not decay isotropically if, for
example, it is polarized, or has an asymmetric coupling to different
helicities [31]. The probabilty amplitude for the decay of $X$ must be
proportional to $\langle m^{\prime},S|R(\theta)|m,S\rangle$, with
$|m,S\rangle$ and $|m^{\prime},S\rangle$ the initial and final state
respectively, where $S$ is the total spin of $X$, $m$ the angular momentum of
$X$ in its flight direction, $m^{\prime}$ the helicity diffence between the
decay products of $X$, and $R(\theta)$ a rotation operator. The complete
results for various spin and helicity configurations are provided in [31],
which we have implemented here. The shapes of the boxes are given by:

$\displaystyle\left.\frac{dN_{\text{box}}}{dE}\right|_{S=0,m=0}$
$\displaystyle\propto 1\,,$
$\displaystyle\left.\frac{dN_{\text{box}}}{dE}\right|_{S=1/2,m=\pm 1/2}$
$\displaystyle\propto
1\pm(C_{1/2}-C_{-1/2})\frac{2E-E_{X}}{\sqrt{E_{X}^{2}-M_{X}^{2}}}\,,$
$\displaystyle\left.\frac{dN_{\text{box}}}{dE}\right|_{S=1,m=0}$
$\displaystyle\propto 4EE_{X}-4E^{2}-M_{X}^{2}\,,$
$\displaystyle\left.\frac{dN_{\text{box}}}{dE}\right|_{S=1,m=\pm 1}$
$\displaystyle\propto
E_{X}^{2}-2EE_{X}+2E^{2}-\frac{1}{2}M_{X}^{2}\pm(C_{1}-C_{-1})(2E-E_{X})\sqrt{E_{X}^{2}-M_{X}^{2}}\,.$
(3)

Here $E_{X}$ is the energy of $X$ in the CM frame, $C_{m^{\prime}}$ are model-
dependent positive normalized coefficients,
$\sum_{m^{\prime}}C_{m^{\prime}}=1$, that determine how $X$ couples to the
various polarizations. Note that for $S=1/2$ the flat box is regained for
either $C_{1/2}=C_{-1/2}$, or when $X$ has no preferred helicity. For a vector
$X$ the flat box is regained when $X$ is unpolarized.

###  2.2 The peak

Another process is the annihilation of two DM particles into a neutrino and a
BSM particle $X$:

$\displaystyle{\rm DM}{\rm DM}\to\nu X\,.$ (4)

The neutrino, which comes directly from DM annihilation, forms a clearly
defined peak in the spectrum with an energy of

$\displaystyle E_{\rm peak}=\frac{4M_{\rm DM}^{2}-M_{X}^{2}}{4M_{\rm DM}}\,.$
(5)

###  2.3 Alternative decay modes for particle $X$

In most realistic BSM models, a possible particle $X$ does not have a 100%
branching ratio into $\nu\nu$, but can have multiple different decay modes,
e.g. ${\rm BR}[X\to\nu Z]=0.5$ and ${\rm BR}[X\to W^{+}e^{-}]=0.5$. The
specific decay modes depend, of course, on the details and parameters of the
chosen BSM model. Here, the assumption is explicitly made that $X$ can only
decay into SM particles. The SM particles222Naturally except for neutrinos and
$e^{\pm}$. will undergo fragmentation, hadronization, and decay, thereby also
producing neutrinos or radiating off photons. A labelling is made of the
neutrinos that come from the various stages of DM annihilation into neutrinos:
the neutrinos that form the peak are called primary neutrinos, those in the
box are called secondary, and all neutrinos that come from the fragmentation
and hadronization of an SM particle are called standard neutrinos:

(6)

The other products of fragmentation, hadronization and decay, such as
$e^{\pm}$, $\bar{p}$, etc., are ignored here. Of course, the process ${\rm
DM}\,{\rm DM}\to XX$ has no primary neutrinos.
The energy range of the box depends, of course, on how the particle $X$ is
produced and what decay modes it has:

$\displaystyle E_{\text{box}}=\frac{M_{X}^{2}-M_{\rm
SM}^{2}}{2M_{X}}\left(\cosh{(\eta)}+\sinh{(\eta)}\cos{(\theta)}\right)\,,$ (7)
$\displaystyle\eta=\begin{cases}\cosh^{-1}{\left(\frac{4M_{{\rm
DM}}^{2}+M_{X}^{2}}{4M_{{\rm DM}}M_{X}}\right)}&\text{for}\quad{\rm DM}{\rm
DM}\rightarrow X\nu\,,\\\ \cosh^{-1}{\left(\frac{M_{{\rm
DM}}}{M_{X}}\right)}&\text{for}\quad{\rm DM}{\rm DM}\rightarrow
XX\,.\end{cases}$

The production mode of $X$ changes the total energy of the particle $X$ at
creation and thus $\eta$, while the decay mode changes the energy of the
secondary neutrino, giving the term $M_{\text{SM}}^{2}$. This results in the
total width of the box being

$\displaystyle E_{\text{box},\text{max}}-E_{\text{box},\text{min}}\equiv\Delta
E_{\text{box}}=\frac{M_{X}^{2}-M_{\rm SM}^{2}}{M_{X}}\sqrt{\Omega^{2}-1}\,,$
(8) $\displaystyle\Omega=\begin{cases}\frac{4M_{{\rm
DM}}^{2}+M_{X}^{2}}{4M_{{\rm DM}}M_{X}}&\text{for}\quad{\rm DM}{\rm
DM}\rightarrow\nu X\,,\\\ \frac{M_{\rm DM}}{M_{X}}&\text{for}\quad{\rm DM}{\rm
DM}\rightarrow XX\,.\end{cases}$

Notably, from this equation it can be seen that for ${\rm DM\,DM}\to\nu X$ the
energy of the peak is equal to the width of the box if $M_{\text{SM}}=0$. In
general $E_{\text{peak}}=\Delta E_{\text{box}}/(1-(M_{\text{SM}}/M_{X})^{2})$.
So for boxes with the same width, the peak is at the same energy, regardless
of the energy of the box. Or, put another way, the position of the peak
directly provides the width of the box up to the specific decay mode of $X$.
Of course, the mass of the particles DM and $X$ can also be expressed by the
initial and final energy of the box:

$\displaystyle
M_{X}=\sqrt{E_{\text{box},\text{max}}E_{\text{box},\text{min}}}+\sqrt{E_{\text{box},\text{max}}E_{\text{box},\text{min}}+M_{\text{SM}}^{2}}\,,$
(9) $\displaystyle
M_{\text{DM}}=\begin{cases}\frac{E_{\text{box},\text{max}}M_{X}^{2}}{M_{X}^{2}-M_{\text{SM}}^{2}}&\text{for}\quad{\rm
DM}{\rm DM}\rightarrow\nu X\,,\\\
\frac{(E_{\text{box},\text{max}}+E_{\text{box},\text{min}})M_{X}^{2}}{M_{X}^{2}-M_{\text{SM}}^{2}}&\text{for}\quad{\rm
DM}{\rm DM}\rightarrow XX\,.\\\ \end{cases}$

For this expression, the decay mode of $X$ must be fixed to know
$M_{\text{SM}}^{2}$.
The standard neutrinos produced by the hadronization and fragmentation of the
SM particle can best be determined with programs such as Pythia [32] and are
not analytically determinable. However, pre-computed spectra of ${\rm DM}{\rm
DM}\to$ SM SM can be used to sample these neutrinos. For ${\rm DM}{\rm DM}\to$
SM SM the energy of the SM particle is simply $E_{\rm SM}=M_{\rm DM}$. For
$X\to$ SM1 SM2, i.e. two different SM particles, the energy of particle SM1
and SM2 is respectively given by:

$\displaystyle E_{{\rm SM}_{1}}=\frac{M_{X}^{2}+M_{{\rm SM}_{1}}^{2}-M_{{\rm
SM}_{2}}^{2}}{2M_{X}}\,,$ $\displaystyle E_{{\rm
SM}_{2}}=\frac{M_{X}^{2}-M_{{\rm SM}_{1}}^{2}+M_{{\rm
SM}_{2}}^{2}}{2M_{X}}\,.$ (10)

Thus, to obtain the correct neutrino spectrum of SM1, the spectrum of ${\rm
DM}{\rm DM}\to{\rm SM}_{1}{\rm SM}_{1}$ is used, where $M_{\rm DM}=E_{{\rm
SM}_{1}}$, and similarly for the neutrino spectrum of SM2. Since this neutrino
spectrum of SM1/2 is in the rest state of $X$, any neutrino sampled from this
spectrum must be Lorentz boosted into the CM frame of annihilating DM
particles, identical to the neutrinos coming directly from the decay of $X$.
Additionally, when using the pre-computed spectra of ${\rm DM}{\rm DM}\to$ SM1
SM1 to determine the spectrum of a single particle SM1, its spectrum d$N$/d$E$
is overestimated by a factor of 2 and thus needs to be compensated for.
Furthermore, any correlations between the annihilation products of the pre-
computed spectra are assumed to be negligible [33, 34].
The average number of primary and secondary neutrinos per collision can be
straightforwardly determined via counting:

Process | $N_{\rm prim}$ | $N_{\rm sec}$
---|---|---
${\rm DM}{\rm DM}\to\nu X$ | 1 |  ${\rm BR}[X\to\nu{\rm SM}$]
${\rm DM}{\rm DM}\to XX$ | 0 |  $2\cdot{\rm BR}[X\to\nu{\rm SM}$]
Table 1: The average number of primary (peak) and secondary (box) neutrinos
per DM annihilation.

Note that when $X$ decays into $\nu\nu$, the number of secondary neutrinos is
doubled and the number of standard neutrinos is zero for that decay mode. The
number of standard neutrinos is determined by the integral of their spectrum,
$dN/dE$ and therefore varies from case to case.

##  3 Model examples

The only requirement to obtain these spectra is a DM particle capable of
either self-annihilation or decay into a mediating particle $X$ that couples
to neutrinos and/or photons. Of course, there are some limitations on the
possible interactions and decay modes of the DM and $X$ particles. For
example, if $X$ is a spin $\frac{1}{2}$ particle, then the decay mode
$X\to\nu\nu$ is forbidden by conservation of angular momentum, making a pure
box shape impossible. Since the shape of the neutrino spectrum is determined
by its kinematics, the spin of the DM or the $X$-particle is irrelevant when
it comes to the position of the peak or the box; only the possible couplings
are affected.
Two examples of candidates for a particle $X$ coupling to neutrinos are:
$Z^{\prime}$ bosons, introduced in a $U_{L_{\mu}-L_{\tau}}(1)$[35, 36, 37, 38]
gauge extensions of the SM and unstable heavy neutrinos such as those in an
inverse seesaw mechanism[39, 40, 41]. A heavy neutrino $\nu_{H}$ and DM
particle $\phi_{\rm DM}$ could, for example, have a non-standard muon neutron
spectrum via the decay chain $\phi_{\rm DM}\phi_{\rm DM}\to\nu_{\mu}\nu_{H}$
with $\nu_{H}\to\nu_{\mu}Z$, and the $Z$ boson gives a continuous neutrino
spectrum. The number of potential models can also be much larger; additional
gauge groups, different neutrino mass mechanisms or Higgs mechanisms specific
to neutrinos could all produce peak or box shapes. Of course, the number of
different DM candidates is very large. These include particles that are added
manually, such as complex scalar DM, or those that arise as a consequence of
the theory itself, e.g. neutralinos in supersymmetry.
Note that a single DM species annihilating with itself cannot have a neutrino
spectrum consisting only of primary and secondary neutrinos, i.e. ${\rm
DMDM}\to\nu X$ with $X\to\nu\nu$. The initial state has an even number of
fermions, while the final state has an odd number of fermions, which is of
course impossible.333Two different DM particles annihilating would circumvent
such a constraint, e.g. a sneutrino and a heavy neutrino as DM candidates
could produce such a spectrum. For photons, of course, there are no such
constraints. It is possible that a process such as ${\rm DMDM}\to\nu\nu X$
with $X\to\nu\nu$, but here no peak is formed because the energy of the
primary neutrinos and $X$ is not uniquely determined by the two-body phase
space. However, a spectrum with a peak and a box is easily obtained via the
decay of fermionic DM. For example, a heavy neutrino decaying into a $\nu$ and
a scalar, which subsequently decays into two $\nu$’s.
In terms of models that might produce non-standard photon spectra, there are a
plethora [42, 43, 44, 45, 46] of models proposed [47] to explain the 2015 750
GeV di-photon excess in ATLAS and CMS. More specifically [48] and [49] detail
some concrete models regarding photon boxes. It should be noted that if $X$ is
electrically charged, as is very possible for a particle coupling to photons,
the possible energy range of the spectrum is limited by constraints on the
mass of $X$, e.g. by LEP searches. This is in contrast to a particle $X$ that
couples to neutrinos via the weak force, which can more easily evade
experimental searches. Remarkably, $X$ does not have to be electrically
charged to couple to photons. The neutral pion $\pi^{0}$, for example, has a
decay mode into two photons. A non-standard photon spectrum could, for
example, be made by a pion-like BSM particle $\Pi^{0}$ and a DM particle
$\phi_{\rm DM}$ by $\phi_{\rm DM}\phi_{\rm DM}\to\Pi^{0}\Pi^{0}$ and
$\Pi^{0}\to\gamma\gamma$.

##  4 The sampling code

###  4.1 Sampling procedure

The sampling of the neutrinos is done by sampling a primary, secondary or
standard neutrino according to the probability $N_{\rm prim}/N$, $N_{\rm
sec}/N$ and $N_{\rm stand}/N$, respectively, where $N=N_{\rm prim}+N_{\rm
sec}+N_{\rm stand}$. Here $N_{\rm prim}$, $N_{\rm sec}$, and $N_{\rm stand}$
refer to the number of primary, secondary and standard neutrinos, as defined
by the Feynman diagrams in (6). The primary neutrino is always sampled at
$E_{\text{peak}}$ since its energy is fixed and therefore has no distribution.
The secondary neutrinos are either sampled uniformly, when the box is flat, or
by rejection sampling if $X$ has a polarization, with the box shapes given by
Eq. (3). The limits of the box are given by Eq. (7). The standard neutrinos
are sampled from precomputed spectra, and subsequently Lorentz boosted into
the CM frame of the annihilating DM particles as described previously. We use
the precomputed spectra from [50, 51, 52]. In order to sample from these
precomputed spectra, we numerically integrate them in order to obtain their
cumulative distribution function. Subsequently, this cumulative distribution
is used in order to perform inverse transform sampling. Notably, secondary
photons can give rise to additional particles due to QED showering effects
[32]. We take these effects into account by using the aforementioned
precomputed spectra when a secondary particle is a photon.

###  4.2 Verification of spectra

Figure 1: Comparison of the spectra between MadGraph/Pythia (orange) and the
sampling code (blue). The upper left and lower two spectra were generated with
simplified models, while the upper right spectrum is an example point from a
full model [53]. The lower two plots show the same model, but evaluated at
different energies.

Figure 2: A comparison between the spectra generated by MadGraph/Pythia
(orange) and the sampling code (blue) for two different spectra. The spectra
of these plots are the same as the bottom two in 1, but plotted
logarithmically. Note that the box now has a slope, which is purely a binning
artifact. Since logarithmically equidistant bins are wider for higher values
of $E$ in absolute terms, a flat box in linearly equidistant bins will then
have a slope when binned logarithmically due to the difference in entries per
bin.

In order to verify the accuracy of this sampler, we perform cross-checks with
multiple spectra computed with MadGraph5 v3.1.1[54] and Pythia 8.309 across a
range of masses and decay modes. We deem a simple visual inspection of the
spectra to be sufficient to validate the fidelity of the sampler. In figure 1
four different example spectra are shown of which the process is indicated in
the relevant plot. The top left and bottom two plots are generated through a
simplified model in which $X$ is a spin-0 or spin-1/2 particle respectively,
while the spectrum from of top right plot is used from [53]. The decay mode of
$X$ of the two spectra from the simplified models both have a branching ratio
of 100% into a single decay mode, while the spectrum in the top right has a
more complicated decay mode of ${\rm BR}[X\to\nu h]={\rm BR}[X\to\nu Z]=1/4$
and ${\rm BR}[X\to W^{\pm}e^{\mp}]={\rm BR}[X\to W^{\pm}\mu^{\mp}]={\rm
BR}[X\to W^{\pm}\tau^{\mp}]=1/6$, which shows that the sampler can indeed
handle more complicated decay modes.
In Figure 2, the two lower spectra of Figure 1 are plotted on a logarithmic
scale, showing that the sampler at low energies performs accurately. There is
some erratic behaviour at $E\approx 10^{-4}{\rm GeV}$, but this is simply an
artefact of a finite number of Monte Carlo samples.
It should be noted that the normalization of the sampler and the spectra
produced by MadGraph and Pythia are different, as the former only samples a
spectrum while the latter fully computes an annihilation process. However,
this difference can be easily remedied by counting the average number of
neutrinos per event, which then directly provides the relative normalization
factor. The spectra calculated by MadGraph and Pythia are the sum of 100,000
iterations, while the sampler’s spectra are the result of 1,000,000 samples,
which are then normalized.
The normalization factor for an arbitrary spectrum can either be determined by
counting the number of particles in a peak or box, which is directly related
to the average number of particles per DM annihilation/decay as given by Tab.
1, or can be provided by the sampling code via the norm_const() function. This
function provides the normalization factor
$(N_{\text{prim}}+N_{\text{sec}}+N_{\text{stand}})/N_{\text{samples}}$. The
sampled spectra are not normalized, such that they can easily be used for
experimental searches, while the normalization is readily performed for
phenomenological purposes.
Remarkably, although the spectra shown are $\mathcal{O}$(1-10)TeV, there is no
inherent scale for these spectra, since both $M_{\rm DM}$ and $M_{X}$ are a
priori both unconstrained. Similarly, the detectability of any spectrum
depends strongly on its annihilation cross-section $\langle\sigma v\rangle$,
which is of course model specific and which we do not comment on here.

###  4.3 Required input

The code can sample ${\rm DM}{\rm DM}\to\nu/\gamma X$, ${\rm DM}{\rm DM}\to
XX$, ${\rm DM}\to\nu/\gamma X$, or ${\rm DM}\to XX$ in which $X$ can have any
decay products and branching ratios into SM particles. The required input is
as follows:

  * •

How the spectra is produced: DM decay (1) or DM annihilation (2)

  * •

The particle type: $\nu_{e}$, $\nu_{\mu}$, $\nu_{\tau}$, or $\gamma$.

  * •

Which process: ${\rm DM}{\rm DM}\to\nu/\gamma X$ (1) or ${\rm DM}{\rm DM}\to
XX$ (2).

  * •

The DM mass $M_{\rm DM}$ in GeV.

  * •

The $X$ mass $M_{X}$ in GeV. The helicity/polarization of $X$ can be given,
with $m=-1,-1/2,0,1/2,1$. If no value is provided $X$ is assumed to be
unpolarized.

  * •

The decay modes of $X$ with the format being ‘$\rm BR$,daughter1,daughter2’.
The total branching ratio must sum to 1. An optional parameter
$C_{m^{\prime}}-C_{-m^{\prime}}$ can be passed in order to fix the shape of
the box according to Eq. (3). When no argument is given
$C_{m^{\prime}}-C_{-m^{\prime}}$ is assumed to be 0.

  * •

The number of samples from the spectrum.

  * •

The path to the csv file where the points are saved. When no input is given,
no save is made. The keyword ‘plot’ or ’logplot’ can be entered to plot the
sampled data linearly or logarithmically.

Acceptable daughter particles of $X$ decay are vl, e, mu, tau, h, z, w, ga, u,
d, s, c, b, t, and g. The neutrino type is only important for the production
of standard neutrinos; especially, the spectrum of tau neutrinos can differ
significantly as compared to the spectra of electron and muon neutrinos when
all other parameters are identical. The input can be provided manually or via
an input file that is passed as an argument.
It is noteworthy that the masses of the DM and $X$ particles are not
constrained, while the interpolated spectra used for the daughter particles
are tabulated up to 10,000 GeV, so the sampler cannot capture the spectra of
the daughter particles with energy greater than 10,000 GeV.
One disadvantage of these non-standard spectra is, of course, the enlargement
of the parameter space. In the typical indirect search for DM, with a fixed
channel, only the mass of the DM particle is important. In these non-standard
spectra, however, the dimensionality of the parameter space is at least 2,
namely the DM mass and the mass of the BSM particle $X$.

##  5 Conclusion

In this study, we have presented a novel approach for neutrino and photon
spectra that is largely model-independent. We have developed a user-specified
Monte Carlo sampler to efficiently sample these spectra, which can be found in
this github repository. While previous experimental searches have focused on
neutrino and photon spectra with lines (peaks) and continuous spectra, more
complex spectra have been largely overlooked. It is important to note that the
spectra we have discussed are not constrained by mass, allowing for
arbitrarily high or low energy ranges. Furthermore, the inclusion of a two-
dimensional parameter space that includes the mass of dark matter and the mass
of a BSM particle X, increases the complexity compared to the typical one-
dimensional parameter space. This sampler may prove valuable not only for
conventional high-energy astroparticle searches, but also for low-energy
searches.

## Acknowledgements

R. RdA is supported by PID2020-113644GB-I00 from the Spanish Ministerio de
Ciencia e Innovación.

## References

  * [1] M. Aguilar, L. Ali Cavasonza, G. Ambrosi, L. Arruda, N. Attig, F. Barao et al., _The alpha magnetic spectrometer (ams) on the international space station: Part ii — results from the first seven years_ , _Physics Reports_ 894 (2021) 1.
  * [2] The Fermi-LAT Collaboration collaboration, _Searching for dark matter annihilation from milky way dwarf spheroidal galaxies with six years of fermi large area telescope data_ , _Phys. Rev. Lett._ 115 (2015) 231301.
  * [3] N. Iovine and J.A. Aguilar, _Indirect search for dark matter in the galactic centre with icecube_ , arXiv:2107.11224.
  * [4] A. Albert, M. André, M. Anghinolfi, G. Anton, M. Ardid, J.-J. Aubert et al., _Search for dark matter towards the galactic centre with 11 years of ANTARES data_ , _Physics Letters B_ 805 (2020) 135439.
  * [5] H.E.S.S. Collaboration collaboration, _Search for dark matter annihilation signals in the h.e.s.s. inner galaxy survey_ , _Phys. Rev. Lett._ 129 (2022) 111101.
  * [6] R. Aloisio, _Pierre auger observatory and super heavy dark matter_ , _EPJ Web of Conferences_ 280 (2023) 07001.
  * [7] B. Zitzer and V. Collaboration, _The veritas dark matter program_ , arXiv:1708.07447.
  * [8] S. Adriá n-Martínez, M. Ageron, F. Aharonian, S. Aiello, A. Albert, F. Ameli et al., _Letter of intent for KM3net 2.0_ , _Journal of Physics G: Nuclear and Particle Physics_ 43 (2016) 084001.
  * [9] _Science with the Cherenkov Telescope Array_ , WORLD SCIENTIFIC (feb, 2018), 10.1142/10986.
  * [10] O. Martineau-Huynh, _The GRAND project and GRANDProto300 experiment_ , _EPJ Web of Conferences_ 210 (2019) 06007.
  * [11] M.-Y. Cui, X. Pan, Q. Yuan, Y.-Z. Fan and H.-S. Zong, _Revisit of cosmic ray antiprotons from dark matter annihilation with updated constraints on the background model from AMS-02 and collider data_ , _Journal of Cosmology and Astroparticle Physics_ 2018 (2018) 024.
  * [12] I. Cholis, T. Linden and D. Hooper, _A robust excess in the cosmic-ray antiproton spectrum: Implications for annihilating dark matter_ , _Phys. Rev. D_ 99 (2019) 103026.
  * [13] M.-Y. Cui, Q. Yuan, Y.-L.S. Tsai and Y.-Z. Fan, _Possible dark matter annihilation signal in the ams-02 antiproton data_ , _Phys. Rev. Lett._ 118 (2017) 191101.
  * [14] A. Cuoco, M. Krämer and M. Korsmeier, _Novel dark matter constraints from antiprotons in light of ams-02_ , _Phys. Rev. Lett._ 118 (2017) 191102.
  * [15] A. Cuoco, J. Heisig, M. Korsmeier and M. Krämer, _Probing dark matter annihilation in the galaxy with antiprotons and gamma rays_ , _Journal of Cosmology and Astroparticle Physics_ 2017 (2017) 053.
  * [16] A. Cuoco, J. Heisig, L. Klamt, M. Korsmeier and M. Krämer, _Scrutinizing the evidence for dark matter in cosmic-ray antiprotons_ , _Phys. Rev. D_ 99 (2019) 103014.
  * [17] S.-J. Lin, X.-J. Bi and P.-F. Yin, _Investigating the dark matter signal in the cosmic ray antiproton flux with the machine learning method_ , _Phys. Rev. D_ 100 (2019) 103014.
  * [18] A. Reinert and M.W. Winkler, _A precision search for WIMPs with charged cosmic rays_ , _Journal of Cosmology and Astroparticle Physics_ 2018 (2018) 055.
  * [19] A. Achterberg, S. Amoroso, S. Caron, L. Hendriks, R.R. de Austri and C. Weniger, _A description of the galactic center excess in the minimal supersymmetric standard model_ , _Journal of Cosmology and Astroparticle Physics_ 2015 (2015) 006.
  * [20] F. Calore, I. Cholis and C. Weniger, _Background model systematics for the fermi GeV excess_ , _Journal of Cosmology and Astroparticle Physics_ 2015 (2015) 038.
  * [21] T. Daylan, D.P. Finkbeiner, D. Hooper, T. Linden, S.K. Portillo, N.L. Rodd et al., _The characterization of the gamma-ray signal from the central milky way: A case for annihilating dark matter_ , _Physics of the Dark Universe_ 12 (2016) 1.
  * [22] C. Gordon and O. Macías, _Dark matter and pulsar model constraints from galactic center fermi-lat gamma-ray observations_ , _Phys. Rev. D_ 88 (2013) 083521.
  * [23] D. Hooper and L. Goodenough, _Dark matter annihilation in the galactic center as seen by the fermi gamma ray space telescope_ , _Physics Letters B_ 697 (2011) 412.
  * [24] D. Hooper and T. Linden, _Origin of the gamma rays from the galactic center_ , _Phys. Rev. D_ 84 (2011) 123005.
  * [25] L. Goodenough and D. Hooper, _Possible evidence for dark matter annihilation in the inner milky way from the fermi gamma ray space telescope_ , arXiv:0910.2998.
  * [26] V. Vitale and A. Morselli, _Indirect search for dark matter from the center of the milky way with the fermi-large area telescope_ , arXiv:0912.3828.
  * [27] A. Achterberg, M. van Beekveld, S. Caron, G.A. Gómez-Vargas, L. Hendriks and R. Ruiz de Austri, _Implications of the Fermi-LAT Pass 8 Galactic Center excess on supersymmetric dark matter_ , _jcap_ 2017 (2017) 040 [1709.10429].
  * [28] T.I. Collaboration, R. Abbasi, M. Ackermann, J. Adams, S.K. Agarwalla, J.A. Aguilar et al., _Search for neutrino lines from dark matter annihilation and decay with icecube_ , arXiv2303.13663.
  * [29] A. Albert, M. André, M. Anghinolfi, G. Anton, M. Ardid, J.-J. Aubert et al., _Results from the search for dark matter in the milky way with 9 years of data of the ANTARES neutrino telescope_ , _Physics Letters B_ 769 (2017) 249.
  * [30] M. Ackermann, M. Ajello, A. Albert, B. Anderson, W.B. Atwood, L. Baldini et al., _Updated search for spectral lines from galactic dark matter interactions with pass 8 data from the fermi large area telescope_ , _Phys. Rev. D_ 91 (2015) 122002.
  * [31] C. Garcia-Cely and J. Heeck, _Indirect searches of dark matter via polynomial spectral features_ , _Journal of Cosmology and Astroparticle Physics_ 2016 (2016) 023.
  * [32] C. Bierlich, S. Chakraborty, N. Desai, L. Gellersen, I. Helenius, P. Ilten et al., _A comprehensive guide to the physics and usage of pythia 8.3_ , arXiv:2203.11601.
  * [33] G. Bélanger, F. Boudjema, P. Brun, A. Pukhov, S. Rosier-Lees, P. Salati et al., _Indirect search for dark matter with micrOMEGAs_2.4_ , _Computer Physics Communications_ 182 (2011) 842.
  * [34] A. Arbey, F. Mahmoudi and G. Robbins, _SuperIso relic v4: A program for calculating dark matter and flavour physics observables in supersymmetry_ , _Computer Physics Communications_ 239 (2019) 238.
  * [35] X.G. He, G.C. Joshi, H. Lew and R.R. Volkas, _New- ${Z}^{{}^{\prime}}$ phenomenology_, _Phys. Rev. D_ 43 (1991) R22.
  * [36] R. FOOT, _New physics from electric charge quantization?_ , _Modern Physics Letters A_ 06 (1991) 527.
  * [37] X.-G. He, G.C. Joshi, H. Lew and R.R. Volkas, _Simplest ${Z}^{{}^{\prime}}$ model_, _Phys. Rev. D_ 44 (1991) 2118.
  * [38] J. Heeck and W. Rodejohann, _Gauged ${L}_{\mu}-{L}_{\tau}$ symmetry at the electroweak scale_, _Phys. Rev. D_ 84 (2011) 075007.
  * [39] E. Ma, _Radiative inverse seesaw mechanism for nonzero neutrino mass_ , _Phys. Rev. D_ 80 (2009) 013013.
  * [40] R.N. Mohapatra and J.W.F. Valle, _Neutrino mass and baryon-number nonconservation in superstring models_ , _Phys. Rev. D_ 34 (1986) 1642.
  * [41] M. Gonzalez-Garcia, A. Santamaria and J. Valle, _Isosinglet-neutral heavy-lepton production in z-decays and neutrino mass_ , _Nuclear Physics B_ 342 (1990) 108.
  * [42] R. Garisto, _Editorial: Theorists react to the cern 750 gev diphoton data_ , _Phys. Rev. Lett._ 116 (2016) 150001.
  * [43] Y. Nakai, R. Sato and K. Tobioka, _Footprints of new strong dynamics via anomaly and the 750 gev diphoton_ , _Phys. Rev. Lett._ 116 (2016) 151802.
  * [44] G. Li, Y.-n. Mao, Y.-L. Tang, C. Zhang, Y. Zhou and S.-h. Zhu, _Pseudoscalar decaying only via loops as an explanation for the 750 gev diphoton excess_ , _Phys. Rev. Lett._ 116 (2016) 151803.
  * [45] C. Petersson and R. Torre, _750 gev diphoton excess from the goldstino superpartner_ , _Phys. Rev. Lett._ 116 (2016) 151804.
  * [46] W.S. Cho, D. Kim, K. Kong, S.H. Lim, K.T. Matchev, J.-C. Park et al., _750 gev diphoton excess may not imply a 750 gev resonance_ , _Phys. Rev. Lett._ 116 (2016) 151805.
  * [47] W. Altmannshofer, J. Galloway, S. Gori, A.L. Kagan, A. Martin and J. Zupan, _750 gev diphoton excess_ , _Phys. Rev. D_ 93 (2016) 095015.
  * [48] A. Ibarra, S.L. Gehler and M. Pato, _Dark matter constraints from box-shaped gamma-ray features_ , _Journal of Cosmology and Astroparticle Physics_ 2012 (2012) 043.
  * [49] A. Ibarra, A.S. Lamperstorfer, S. Ló pez-Gehler, M. Pato and G. Bertone, _On the sensitivity of CTA to gamma-ray boxes from multi-TeV dark matter_ , _Journal of Cosmology and Astroparticle Physics_ 2015 (2015) 048.
  * [50] A. Jueid, J. Kip, R.R. de Austri and P. Skands, _Impact of QCD uncertainties on antiproton spectra from dark-matter annihilation_ , _Journal of Cosmology and Astroparticle Physics_ 2023 (2023) 068.
  * [51] S. Amoroso, S. Caron, A. Jueid, R.R. de Austri and P. Skands, _Estimating QCD uncertainties in monte carlo event generators for gamma-ray dark matter searches_ , _Journal of Cosmology and Astroparticle Physics_ 2019 (2019) 007.
  * [52] A. Jueid, J. Kip, R.R. de Austri and P. Skands, _The strong force meets the dark sector: a robust estimate of qcd uncertainties for anti-matter dark matter searches_ , arXiv:2303.11363.
  * [53] M. van Beekveld, W. Beenakker, S. Caron, J. Kip, R.R. de Austri and Z. Zhang, _Non-standard neutrino spectra from annihilating neutralino dark matter_ , _SciPost Physics Core_ 6 (2023) 006.
  * [54] J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer et al., _The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations_ , _Journal of High Energy Physics_ 2014 (2014) 1.


# SSE: A Metric for Evaluating Search System Explainability

Catherine Chen<EMAIL_ADDRESS>Brown UniversityProvidenceRhode
IslandUSA and  Carsten Eickhoff<EMAIL_ADDRESS>University of
TübingenTübingenGermany

###### Abstract.

Explainable Information Retrieval (XIR) is a growing research area focused on
enhancing transparency and trustworthiness of the complex decision-making
processes taking place in modern information retrieval systems. While there
has been progress in developing XIR systems, empirical evaluation tools to
assess the degree of explainability attained by such systems are lacking. To
close this gap and gain insights into the true merit of XIR systems, we extend
existing insights from a factor analysis of search explainability to introduce
SSE (Search System Explainability), an evaluation metric for XIR search
systems. Through a crowdsourced user study, we demonstrate SSE’s ability to
distinguish between explainable and non-explainable systems, showing that
systems with higher scores indeed indicate greater interpretability.
Additionally, we observe comparable perceived temporal demand and performance
levels between non-native and native English speakers. We hope that aside from
these concrete contributions to XIR, this line of work will serve as a
blueprint for similar explainability evaluation efforts in other domains of
machine learning and natural language processing.

explainability, search, evaluation, crowdsourcing

††ccs: Information systems Web search engines††ccs: Information systems
Crowdsourcing††ccs: General and reference Metrics

##  1\. Introduction

Explainable information retrieval (XIR) research aims to develop methods that
increase the transparency and reliability of information retrieval systems.
XIR systems are designed to provide end-users with a deeper understanding of
the rationale behind result ranking decisions. Besides casual Web search,
these systems hold promising potential for impactful real-world information
needs, such as matching patients to clinical trials, retrieving case law for
legal research, and detecting misinformation in news and media analysis.
Despite several advancements in their development, there is a lack of
empirical, standardized techniques for evaluating the efficacy of XIR systems.

Current approaches toward evaluating XIR systems are limited by a lack of
consensus in the broader explainable artificial intelligence (XAI) community
on a definition of explainability. Explainability has often been considered a
singular concept although recent literature suggests it to be an amalgamation
of several sub-factors (Lipton, 2018; Doshi-Velez and Kim, 2018; Nauta et al.,
2022). As a result, evaluation has occurred on a binary scale, considering
systems as either explainable or black-box, hindering direct system
comparison. Furthermore, explainability is often declared with anecdotal
evidence rather than measured quantitatively.

Chen and Eickhoff (2022) take a step towards addressing these limitations by
introducing a multidimensional definition of explainability in search systems.
In this paper, we extend their two-factor hierarchical definition to propose a
continuous-scale evaluation metric for quantifying explainability in Web
search systems. To validate this metric and gain practical insights into
implementing human-centered evaluation tools, we conduct a crowdsourced user
study and assess the impact such tools have on human annotators.111 We make
all data and analysis code publicly available at
https://github.com/catherineschen/sse-metric. Specifically, we make the
following contributions:

  * •

We introduce SSE222Pronounced ‘es-es-e’, a metric to evaluate Search System
Explainability, extending the search system explainability factor analysis by
Chen and Eickhoff (2022).

  * •

We conduct a crowdsourced user study and validate the ability of SSE to
distinguish between non-explainable and explainable systems, demonstrating
that higher SSE scores indicate greater explainability.

  * •

We evaluate the questionnaire workload on crowdworkers and find that while
non-native English speakers report slightly higher levels of perceived Mental
Demand and Effort, their perceived Temporal Demand and Performance are
comparable to those of native English speakers.

The remainder of the paper is structured as follows: In Section 2, we present
related work in XIR system evaluation. Section 3 provides an overview search
system explainability evaluation. In Section 4, we detail our study design and
data collection setup. In Section 5, we discuss the evaluation results of SSE,
examining its effectiveness in evaluating search system explainability and the
workload it imposes on crowdworkers. Finally, in Section 6, we conclude with
an overview of future research directions.

##  2\. Related Work

Due to the lack of consensus and recognition of explainability as a multi-
facet concept, evaluation falls short in two ways. Firstly, current methods do
not consider a multidimensional definition. Authors of frameworks such as LIME
and SHAP demonstrate the utility of their methods through user evaluations but
fail to quantify the degree of explainability provided by their methods
(Ribeiro et al., 2016; Lundberg and Lee, 2017). Relying solely on declarations
of explainability without measurement hinders targeted system improvements and
a more holistic definition is needed for robustness.

Secondly, while there are many ML evaluation metrics for system performance
comparison, there is no standard metric for evaluating explainability.
Although several authors acknowledge the importance of quantitative evaluation
metrics (Nauta et al., 2022; Doshi-Velez and Kim, 2017; Adadi and Berrada,
2018; Das and Rad, 2020), existing approaches often focus on a singular aspect
of explainability or lack evaluation of explanation quality, relying on
anecdotal evidence (Singh and Anand, 2019; Qu et al., 2020; Yu et al., 2022;
Polley et al., 2021; Colin et al., 2022). While Nguyen and Martínez (2020)
come close by introducing a suite of metrics to quantify interpretability,
they fail to quantify interactions between facets and measure the importance
of each individual facet.

Chen and Eickhoff (2022) address the first limitation by introducing an
empirically derived multidimensional definition of explainability through
factor analysis. In this paper, we address the second limitation by extending
the insights of their work and introducing an all-encompassing metric that
takes into account multiple desiderata of explainability, which will help us
gain more insight into the strengths and weaknesses of a system.

##  3\. Search System Explainability (SSE)

The search explainability factor model introduced by Chen and Eickhoff (2022)
consists of two factor groups, utility and roadblocks, where the first factor
broadly refers to a system’s usefulness within some context and the second
factor refers to qualities a system lacks in order to be fully explainable.
The factor model originates from a 19-item questionnaire (e.g., “I’m unable to
follow how the search engine ordered the results.”) scored on a 7-point Likert
scale (1 - Strongly Disagree, 7 - Strongly Agree). Administered following user
interaction with a specific search system (e.g., via a search task), the
questionnaire assesses the extent to which system features facilitate or
impede explainability.

(1)
$SSE=MinMaxNorm\bigg{(}\sum_{f\in{F}}w_{f}\sum_{i\in{I_{f}}}w_{i}r_{i}\bigg{)}$

We extend this factor analysis to introduce Search System Explainability
(SSE), a metric for evaluating search system explainability. The formula for
SSE is shown in Equation 1, where $F$ is the set of all factors $f$ and
$I_{f}$ is the set of survey items $i$ that correspond to factor $f$. The
user’s response $r_{i}$ to item $i$ is weighted by the corresponding loading
coefficient $w_{i}$ and the accumulated response scores for each factor are
weighted by the corresponding loading coefficient $w_{f}$ determined by Chen
and Eickhoff (2022). To make SSE score values more interpretable, min-max
normalization with the smallest and largest theoretically possible SSE scores
is applied to normalize overall values between 0 and 1.

##  4\. Study Design

In this paper, we aim to test the hypothesis that an explainable system will
attain a higher SSE score than a non-explainable system.

###  4.1. Task Setup

We employed a between-subjects design and asked participants to evaluate one
of two search systems: (A) BASELINE: a minimalist search engine resembling
existing commercial search systems with no explainable capabilities and (B)
BARS: a search system with visual explanations representing query term
importance (Figure 1) developed by Ramos and Eickhoff (2020). To guide their
evaluation, we provided users access to a collection of documents and asked
them to answer an open-ended question on one of 10 possible topics333List of
topics: compost pile, Antarctica exploration, heroic acts, Northern Ireland
industry, new hydroelectric projects, health and computer terminals, quilts
and income, wildlife extinction, illegal technology transfer, Salvation Army
benefits using the search engine. Documents, questions, and topics were
selected from the TREC 2004 Robust Track Dataset (Voorhees, 2005).

Figure 1. BARS search system based on the system presented by Ramos and
Eickhoff (2020). Stacked bar graphs, located alongside each search result,
depict the BM25 scores assigned to individual keywords in the search query.
Hovering over a bar reveals the corresponding score for the respective term.

Both search systems utilized the same BM25 retrieval model to rank relevant
documents. We followed the document selection procedure described in Chen and
Eickhoff (2022) to sample 50 relevant documents from each of the 10 topics,
resulting in a collection of 500 documents for users to search from. We used
PyTerrier (Macdonald and Tonellotto, 2020) to index the documents and wrapped
the index in a Flask server, hosting it on Fly.io. We hosted the front-end
search system on Netlify and used MongoDB for additional data collection.

After completing the search task, participants were asked to fill an
evaluation questionnaire. We incorporated two attention check items (“I swim
across the Atlantic Ocean to get to work every day.” and “I think search
engines are cool. Regardless of your opinion, select ‘Strongly Agree’ as your
response below. This is an attention check.”) at random points in the
questionnaire for quality control.

Following the evaluation questionnaire, we administered the NASA Task Load
Index (NASA-TLX) (Hart, 2006) to measure the perceived workload of the
evaluation questionnaire. The NASA-TLX is a 6-item questionnaire measuring
perceived workload on 6 dimensions (Mental Demand, Physical Demand, Temporal
Demand, Performance, Effort, Frustration) with responses recorded on a scale
of 0-20, where lower scores are more desirable and indicate a low workload
(except for Performance, where lower scores indicate high success). Since the
original scoring method involves a two-step weighting scheme through numerous
pairwise comparisons, we followed the common practice of analyzing raw TLX
(RTLX) responses to reduce the amount of time needed to administer the survey
(Hart, 2006).

###  4.2. Data Collection

Figure 2. Loadings for individual questionnaire items. Items are labeled by
their original aspect as determined by Chen and Eickhoff (2022) and ordered by
score (response multiplied by item loading) difference between the two
systems. Aspects with bars increasing toward the right indicate the practical
usefulness of each system, while aspects with bars increasing toward the left
represent areas requiring improvement to achieve full explainability.

Prior to full-scale data collection, we conducted a think-aloud study with 6
graduate students to identify and address bugs in the study design and system.
We used the concurrent think-aloud method due to its effectiveness in
usability testing and time efficiency (Olmsted-Hawala et al., 2010).
Observations from this period led to several changes. Specifically, we made
slight modifications to the NASA-TLX assessment: (1) we replaced the term
‘task’ with ‘survey’ to align responses with the evaluation questionnaire
rather than the search task and (2) we inverted the scale on the Performance
dimension to have higher responses equate to ‘Perfect’ rather than ‘Failure’
and align with a more intuitive understanding of performance. Additionally, we
added more detailed descriptions to certain questions to accurately anchor
neutral responses of low effort to 0 instead of 10, which would indicate a
medium effort.

We recruited 100 participants aged 18-65 for our study on Prolific444IRB
approval was judged unnecessary by an IRB member due to the classification of
our study as a system evaluation. and paid them £5.25 for 30 minutes of work
(estimated from a 15-participant pilot study), equivalent to the minimum wage
in our location. Additionally, we required participants to be fluent in
English and have a minimum prior approval rate of 95%. On average, users
completed the task more quickly than our initial estimate, with the median
time being approximately 22 minutes. 15 countries were represented in our
sample, with participants from Australia (1), Africa (4), Europe (81), North
America (12), South America (1), and 1 unknown location.

##  5\. Results and Discussion

###  5.1. Search Efficiency

Despite users in the BARS group spending a slightly longer duration on the
overall search task (11.97 min) compared to the BASELINE group (11.41 min), an
analysis of search time per query and the number of viewed documents per query
indicates that the explainable BARS system exhibited higher efficiency than
the non-transparent system (Table 1). Users issued more queries per task in
the explainable system (3.45 vs 2.54). This may be attributed to query
modifications aimed at targeting specific keywords to leverage query-term
matching for calculating BM25 scores. These results suggest that the BARS
system allowed users to scan search engine result pages (SERP) more quickly.
This observation aligns with previous research by Ramos and Eickhoff (2020),
which demonstrated the positive impact of explainable systems on search
efficiency.

Table 1. Task engagement statistics (M $\pm$ SD). Although participants in the BARS group spent more time overall compared to the BASELINE group, they were more efficient per SERP. | BASELINE | BARS
---|---|---
| n=56 | n=44
Task time (in min) | 11.41 $\pm$ 8.81 | 11.97 $\pm$ 6.42
# of queries | 2.54 $\pm$ 1.95 | 3.45 $\pm$ 2.90
# of documents viewed | 6.30 $\pm$ 4.82 | 6.89 $\pm$ 5.56
Time per query | 6.79 $\pm$ 8.22 | 5.36 $\pm$ 4.50
Documents viewed per query | 3.84 $\pm$ 4.65 | 2.91 $\pm$ 2.85
Table 2. NASA-RTLX responses (lower is better) for perceived evaluation questionnaire workload indicate that the BARS system caused less Mental and Physical Demand, higher perceived Performance, and less perceived Effort and Frustration than the non-transparent BASELINE system. | Mental Demand | Physical Demand | Temporal Demand | Performance | Effort | Frustration
---|---|---|---|---|---|---
Overall | 7.53 | 2.26 | 3.44 | 4.12 | 10.01 | 3.28
BASELINE | 7.79 | 2.89 | 3.02 | 4.02 | 10.18 | 4.00
BARS | 7.20 | 1.45 | 3.98 | 4.24 | 9.80 | 2.36

###  5.2. System Explainability

Congruent with our initial hypothesis, we find that users assigned higher
explainability scores to the BARS system in comparison to the non-transparent
BASELINE system (Figure 3). The BARS system exhibited a mean SSE score
(M=0.67) greater than that of the BASELINE system (M=0.44). The results from a
Wilcoxon signed rank test on a random sample of the maximum number of samples
in the minority class (n=44) reveal a statistically significant difference
(T=98.0, p¡0.001), confirming that SSE is able to accurately discern between
explainable and non-explainable systems.

Figure 3. Distribution of SSE scores. Results from a Wilcoxon signed rank test
(T = 98.0, p ¡ 0.001) indicate there is a statistically significant difference
in scores between systems.

The aspect scores offer a comprehensive evaluation of individual questionnaire
items and specific areas where the BARS system outperformed the BASELINE
(Figure 2). Notably, items are grouped by their representation of attributes
relating to utility (right) and areas for improvement (left). Additionally,
the aspects positioned closer to the top of the graph highlight the most
significant disparities in average item scores between the two systems. For
instance, the widest margin between the two systems is on lack of local
interpretability, indicating that the BARS system has more local
interpretability than the BASELINE system, but still has some room for
improvement. Overall, the BARS system outperformed the BASELINE system across
all questionnaire items, thus indicating higher readiness for deployment and a
reduced need for extensive feature modifications to achieve full explainable
capability.

###  5.3. Evaluation Questionnaire Task Load

Table 2 reports the average NASA-RTLX scores for the 6 workload dimensions.
Participants evaluating the BARS system reported a slightly heightened sense
of urgency (i.e., Temporal Demand), potentially due to the nature of the
search task rather than the survey itself, as users in this group took
slightly longer to complete the initial search task compared to the BASELINE
group (Table 1). Overall, participants from both groups considered the
evaluation instrument moderately demanding in terms of Mental Demand, moderate
in Effort, low in Physical Demand, Temporal Demand, and Frustration, and high
success in Performance.

Figure 4 presents the NASA-RTLX scores grouped by native and non-native
English speakers. On average, non-native English speakers demonstrated higher
perceived levels of Mental Demand, Physical Demand, and Frustration compared
to native English speakers. However, their perceived Performance was similar
to that of native English speakers while reporting less Temporal Demand and
Effort. Nevertheless, these findings are encouraging since the overall task
load for all participants ranged from low to medium, suggesting the
suitability of the explainability evaluation questionnaire for both native and
non-native English speakers.

Figure 4. NASA-RTLX responses between native (n=51) and non-native (n=49)
English speakers. While non-native English speakers have a higher perceived
task load than native English speakers on 3 dimensions, they had a lower
perceived Temporal Demand and Effort for a higher perceived Performance.

##  6\. Conclusion and Future Work

In this paper, we proposed a new metric for evaluating search system
explainability on a continuous scale. Through a crowdsourced user study, we
demonstrate that an XIR search system exhibited a significantly higher SSE
score than a baseline non-transparent search system, indicating greater
explainable capability. Additionally, a task load analysis reveals that both
native and non-native English speakers found the evaluation instrument to have
a low-to-medium workload. It is important to note that explainability can
differ between domains and thus, the findings in this paper are limited to Web
search. Future work should examine how SSE performs in other contexts.
Overall, our study offers valuable insights for future researchers in the
field of XIR, providing guidance on utilizing this evaluation instrument to
conduct their own assessments.

## References

  * (1)
  * Adadi and Berrada (2018) Amina Adadi and Mohammed Berrada. 2018.  Peeking inside the black-box: a survey on explainable artificial intelligence (XAI).  _IEEE access_ 6 (2018), 52138–52160.
  * Chen and Eickhoff (2022) Catherine Chen and Carsten Eickhoff. 2022.  Evaluating Search Explainability with Psychometrics and Crowdsourcing.  _arXiv preprint arXiv:2210.09430_ (2022).
  * Colin et al. (2022) Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. 2022\.  What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods.  _Advances in Neural Information Processing Systems_ 35 (2022), 2832–2845.
  * Das and Rad (2020) Arun Das and Paul Rad. 2020\.  Opportunities and challenges in explainable artificial intelligence (xai): A survey.  _arXiv preprint arXiv:2006.11371_ (2020).
  * Doshi-Velez and Kim (2017) Finale Doshi-Velez and Been Kim. 2017.  Towards a rigorous science of interpretable machine learning.  _arXiv preprint arXiv:1702.08608_ (2017).
  * Doshi-Velez and Kim (2018) Finale Doshi-Velez and Been Kim. 2018.  Considerations for evaluation and generalization in interpretable machine learning.  In _Explainable and interpretable models in computer vision and machine learning_. Springer, 3–17.
  * Hart (2006) Sandra G Hart. 2006\.  NASA-task load index (NASA-TLX); 20 years later. In _Proceedings of the human factors and ergonomics society annual meeting_ , Vol. 50. Sage publications Sage CA: Los Angeles, CA, 904–908.
  * Lipton (2018) Zachary C Lipton. 2018\.  The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ 16, 3 (2018), 31–57.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ 30 (2017).
  * Macdonald and Tonellotto (2020) Craig Macdonald and Nicola Tonellotto. 2020.  Declarative Experimentation inInformation Retrieval using PyTerrier. In _Proceedings of ICTIR 2020_.
  * Nauta et al. (2022) Meike Nauta, Jan Trienes, Shreyasi Pathak, Elisa Nguyen, Michelle Peters, Yasmin Schmitt, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. 2022\.  From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.  _arXiv preprint arXiv:2201.08164_ (2022).
  * Nguyen and Martínez (2020) An-phi Nguyen and María Rodríguez Martínez. 2020\.  On quantitative aspects of model interpretability.  _arXiv preprint arXiv:2007.07584_ (2020).
  * Olmsted-Hawala et al. (2010) Erica L Olmsted-Hawala, Elizabeth D Murphy, Sam Hawala, and Kathleen T Ashenfelter. 2010.  Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability. In _Proceedings of the SIGCHI conference on human factors in computing systems_. 2381–2390.
  * Polley et al. (2021) Sayantan Polley, Rashmi Raju Koparde, Akshaya Bindu Gowri, Maneendra Perera, and Andreas Nuernberger. 2021.  Towards trustworthiness in the context of explainable search. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2580–2584.
  * Qu et al. (2020) Jiaming Qu, Jaime Arguello, and Yue Wang. 2020\.  Towards explainable retrieval models for precision medicine literature search. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1593–1596.
  * Ramos and Eickhoff (2020) Jerome Ramos and Carsten Eickhoff. 2020.  Search result explanations improve efficiency and trust. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1597–1600.
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Singh and Anand (2019) Jaspreet Singh and Avishek Anand. 2019.  Exs: Explainable search using local model agnostic interpretability. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 770–773.
  * Voorhees (2005) Ellen Voorhees. 2005\.  Overview of the TREC 2004 Robust Retrieval Track.  https://doi.org/10.6028/NIST.SP.500-261
  * Yu et al. (2022) Puxuan Yu, Razieh Rahimi, and James Allan. 2022.  Towards Explainable Search Results: A Listwise Explanation Generator. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 669–680.