# Deep Neural Network Uncertainty Quantification for LArTPC Reconstruction

D. Koh,11footnotetext: Corresponding author. A. Mishra  and K. Terao

###### Abstract

We evaluate uncertainty quantification (UQ) methods for deep learning applied
to liquid argon time projection chamber (LArTPC) physics analysis tasks. As
deep learning applications enter widespread usage among physics data analysis,
neural networks with reliable estimates of prediction uncertainty and robust
performance against overconfidence and out-of-distribution (OOD) samples are
critical for their full deployment in analyzing experimental data. While
numerous UQ methods have been tested on simple datasets, performance
evaluations for more complex tasks and datasets are scarce. We assess the
application of selected deep learning UQ methods on the task of particle
classification using the PiLArNet [1] monte carlo 3D LArTPC point cloud
dataset. We observe that UQ methods not only allow for better rejection of
prediction mistakes and OOD detection, but also generally achieve higher
overall accuracy across different task settings. We assess the precision of
uncertainty quantification using different evaluation metrics, such as
distributional separation of prediction entropy across correctly and
incorrectly identified samples, receiver operating characteristic curves
(ROCs), and expected calibration error from observed empirical accuracy. We
conclude that ensembling methods can obtain well calibrated classification
probabilities and generally perform better than other existing methods in deep
learning UQ literature.

##  1 Introduction

Deep learning has largely established itself as a dominant method in machine
learning applications, in part due to its competence at a variety of well-
known tasks such as image recognition, natural language processing, and
automated control applications. As such, scientists in both artificial
intelligence and the physical sciences have been investigating ways to realize
deep learning’s success in more complex domains of fundamental research. The
trend for integrating deep learning for physics data reconstruction has been
particularly notable in experimental particle physics, where large data
generation from particle detectors such as liquid argon time projection
chambers (LArTPCs) and the Large Hadron Collider (LHC) naturally prepare
fertile grounds for deep learning models.

Using deep learning for fundamental research, however, presents complications
that are often omitted in many common industrial use cases, where
practitioners generally attend to achieving state-of-the-art results with
respect to a family of conventional performance metrics. In particular, one of
the most pressing issues with using deep neural networks for fundamental
research is developing robust and consistent methods for quantifying
uncertainties of their predictions. Deep neural networks are unable to
recognize out-of-distribution examples and habitually make incorrect
predictions with high confidence for such cases [2, 3]. Uncertainty in
predictions has had serious consequences while applying deep learning to high-
regret and safety-critical applications such as automated driving [4, 5, 6],
law enforcement [7], medical sciences[8], etc. Overconfidence for out-of-
distribution examples also emphasizes the need for deep learning models to
indicate whether a given prediction is to be trusted or not. Undoubtedly, for
deep neural nets to be integrated into the physics measurement process, such
characteristics of deterministic neural networks must be addressed by an
effective method for uncertainty quantification (UQ).

As the need for UQ gradually escalated in domains such as autonomous driving
and medicine, UQ methods diversified into a variety of different approaches
under the name of Bayesian Deep Learning (BDL) [9], but with scarce
substantial application in the physical sciences. Moreover, most BDL methods
have been benchmarked on simplified datasets (MNIST[10], CIFAR10[11]), which
are not representative of the complexity of physics data reconstruction
process. Modern accelerator neutrino experiments such as ICARUS[12] and
DUNE[13, 14] offer ideal grounds for testing the efficacy of BDL in UQ, due to
its recent adaptation and the moderate success of deep learning based
reconstruction techniques. The benefit derived from a detailed assessment of
different UQ algorithms on a complex, multi-objective task such as LArTPC data
reconstruction is two-fold: allow practitioners in machine learning to
evaluate BDL’s applicability in a real-world setting [15] and enable
physicists to design neural network that produce well justified uncertainty
estimates for rejecting erroneous predictions and detecting out-of-
distribution instances[16, 17].

Practitioners of deep learning in LArTPC reconstruction agree on the need for
calibrated uncertainty bounds for deep learning model predictions along with
OOD robustness. However, numerous different uncertainty quantification
algorithms have been proposed for deep learning. These range from empirical
approaches (such as bootstrapped ensembles[18, 19]), Bayesian approaches (such
as EDL[20], HMC[21]) and hybrid approaches (such as MC Dropout[22]). None of
these have been tested for complex applications such as LArTPC reconstruction.
In this investigation, we select the most promising uncertainty quantification
approaches from each of these categories, test and evaluate them with respect
to critical intermediate reconstruction tasks: particle classification and
particle shape (semantic) segmentation. For a given 3D LArTPC image, particle
classification is defined to be the task of predicting the type–the Particle
Data Group (PDG) code–of the particle that has created the energy depositions
recorded by the detector. Particle shape segmentation, which we will refer to
as semantic segmentation[23, 24], is the process of predicting a category for
every non-zero pixel of the LArTPC image. For instance, in a study by the
MicroBooNE collaboration [25], a neural network was trained to differentiate
electromagnetic showers and tracks for each pixel. We briefly summarize the
different methodologies and discuss the apparent advantages and disadvantages
of using each of the proposed models in the following section. We describe in
detail the monte-carlo generated 3D LArTPC particle image dataset and state
any assumptions or additional information that was used to train and evaluate
each model. In Section IV, we present quantitative performance evaluation of
different UQ models on three different settings of single particle
classification, multi-particle classification, and semantic segmentation,
using a variety of quantitative metrics to measure UQ fidelity.

While quantifying the predictive uncertainty for machine learning models,
there are two broad classes of uncertainty that need to be considered:
epistemic uncertainty and aleatoric uncertainty. Epistemic uncertainty [26]
arises due to a lack of knowledge regarding the dynamics of the system under
consideration, or an inability to express the underlying dynamics using
models. It is also referred to as reducible uncertainty, as one could in
principle reduce it by collecting more data to further constrain the model.
Quantifying epistemic uncertainties is essential for safety critical
applications, for instance to understand and identify examples that are
significantly different from the training data [27]. Epistemic uncertainties
are important while learning from small datasets where the training data is
sparse, globally or locally. Epistemic uncertainty can be further divided into
parameter and structural uncertainty. Parameter uncertainty[28] captures our
ignorance about the exact combination of model coefficients that generated the
training data, for example the weights and biases of a neural network model,
and may be minimized with larger corpora of training data. Structural
uncertainty in deep learning [29] captures our inability to represent the
underlying process via a limited model. In the context of deep learning, it
represents the uncertainties in the model architecture and hyperparameters
like the activation functions utilized.

Figure 1: Schematic outlining the types of uncertainty in Deep Learning model
predictions.

Aleatoric uncertainty [26] is often referred to as irreducible uncertainty or
stochastic uncertainty. In scientific applications, this arises due to noise
in the training data, projection of data onto a lower dimensional subspace,
absence of important input features in the data, etc. While large datasets can
ameliorate epistemic uncertainty, aleatoric uncertainties are immune to such
remedies and thus often named as irreducible uncertainties. Aleatoric
uncertainties[27] can be divided into homoscedastic and heteroscedastic
uncertainty. Homoscedastic uncertainties are independent of the input data,
but depend on the application. In contrast, heteroscedastic uncertainties
strongly depend on the model input. These exhibit variability across examples
with some inputs having significantly higher noise in the predictions for some
samples than others.

These two classes of uncertainty, epistemic and aleatoric, are significantly
different in their nature and in the manner that they are dealt with in
uncertainty quantification. For instance, Aleatoric uncertainties arise due to
random variability and thus, introduce random variance in the model
predictions. Epistemic uncertainties on the other hand arise due to incomplete
representation of underlying physics and add a bias to the simulation results.
This broad division regarding the sources and nature of predictive
uncertainties is schematically outlined in Figure 1.

In physical sciences, the divisions between different kinds of uncertainty and
their definitions are different. By precedent, uncertainty in physical
sciences is often demarcated into statistical and systematic uncertainties.
Statistical uncertainties refer to errors that may be quantified by
statistical analysis over a series of measurements. Systematic uncertainties
occur due to limitations of models and theory, and thus cannot be treated the
same as statistical. As a mapping between the nomenclature utilized in the
different fields, epistemic uncertainties are always systematic uncertainties,
and statistical uncertainties are always aleatoric.

##  2 Methods of Uncertainty Quantification in Deep Learning

In machine learning, a given dataset is divided into training and test sets.
The training set is used to find the optimal model that best describes the
data, with respect to some numerical criterion. The test set is later used to
estimate the generalization error and select the best performing trained
model, which has an inherent bias towards the training set. The generalization
error is the expected error of the model on data that it has not been trained
on.

We first introduce mathematical notations used throughout the paper. Let
$X=\\{x_{1},x_{2},...,x_{N}\\}$ and $Y=\\{y_{1},y_{2},...,y_{N}\\}$ be the
input data and labels in the training set, and let
$\hat{X}=\\{{\hat{x}}_{1},{\hat{x}}_{2},...,{\hat{x}}_{M}\\}$ and
$\hat{Y}=\\{{\hat{y}}_{1},{\hat{y}}_{2},...,{\hat{y}}_{M}\\}$ denote the
corresponding quantities in the test set. Uncertainty quantification for
$C$-class classification may be defined as the process of determining a
posterior distribution $\mathbf{p}(\hat{x}\mid X,Y)\in S_{C}$, where $S_{C}$
is the $C$-dimensional unit simplex. Among numerous models and studies
focusing on the uncertainty quantification of neural networks [30, 31], we
focus on methods designed for multi-class classification tasks that require
minimal changes to popular neural network architectures. In this paper, we
consider three specific UQ methods: model ensembling [19], Monte Carlo Dropout
(MCD) [32], and Evidential Deep Learning (EDL) [20, 33].

###  2.1 Deterministic Neural Networks

Let $f_{\theta}$ refer to a neural network with trainable parameters $\theta$.
Since each prediction of the network depends on the the training set data and
label pair $(X,Y)$, we denote the model predictions as
$\mathbf{f}_{\theta}(\hat{x};X,Y)\in\mathbb{R}^{C}$ with
$j\in\\{1,2,...,M\\}$, where $\hat{x}$ is an example in the test set and $C$
is the number of distinct labels in our dataset.

$\mathbf{f}_{\theta}(\hat{x};X,Y)\equiv\begin{bmatrix}f_{\theta}(\hat{x};X,Y)_{1}\\\
f_{\theta}(\hat{x};X,Y)_{2}\\\ \vdots\\\
f_{\theta}(\hat{x};X,Y)_{C}\end{bmatrix}\equiv\begin{bmatrix}\hat{z}_{1}\\\
\hat{z}_{2}\\\ \vdots\\\ \hat{z}_{C}\end{bmatrix}\in\mathbb{R}^{C}.$ (2.1)

Here, it is typical to call each predicted score value $\hat{z}_{c}\equiv
f_{\theta}(\hat{x};X,Y)_{c}\in\mathbb{R}$ as logits. The logit value of each
class may be interpreted as the log-odds value of predicting the given example
$\hat{x}$ as class $c\in C$. In mathematical terms, the logit function, $L(p)$
maps probabilities to their log-odds, $L(p)=ln(\frac{p}{1-p})$. The predicted
label $\hat{y}$ corresponding to $\hat{x}$ is then given as:

$\hat{y}\equiv\underset{{c\in C}}{\text{argmax}}[\hat{z}_{c}].$ (2.2)

We may also apply the softmax function to the logit value to obtain the
predicted probabilities $\hat{p}_{c}$:

$\mathbf{p}(\hat{x}\mid
X,Y)\equiv\begin{bmatrix}\frac{e^{\hat{z}_{1}}}{\underset{c\in
C}{\sum}e^{\hat{z}_{c}}}&\frac{e^{\hat{z}_{2}}}{\underset{c\in
C}{\sum}e^{\hat{z}_{c}}}&\cdots&\frac{e^{\hat{z}_{C}}}{\underset{c\in
C}{\sum}e^{\hat{z}_{c}}}\par\end{bmatrix}\equiv\begin{bmatrix}\hat{p}_{1}&\hat{p}_{2}&\cdots&\hat{p}_{C}\end{bmatrix}.$
(2.3)

For deterministic networks, the posterior is simply given by the predicted
probabilities $\hat{p}_{1},...,\hat{p}_{C}$. It is well known that for
deterministic neural networks, the probability values $\hat{p}_{c}\in[0,1]$
cannot be interpreted as valid estimate of the empirical accuracy of
predicting a given example as class $c$ and are often denoted as being
uncalibrated. For example, in a binary classification task with input
$\hat{x}$, $\mathbf{p}(\hat{x}\mid X,Y)\in[0,1]$, and $\hat{y}\in\\{0,1\\}$,
we may ask if the neural network’s prediction
$\mathbf{\hat{p}}\equiv\mathbf{p}(\hat{x}\mid X,Y)=0.7$ corresponds to a 70%
chance of classifying $\hat{x}$ as $\hat{y}=1$. UQ methods for neural networks
may be summarized as attempts to estimate the true empirical accuracy from
neural network outputs such as (but not limited to) $\mathbf{p}(\hat{x}\mid
X,Y)$.

###  2.2 Ensembling Methods

In the context of deep learning, model ensembling refers to the method of
training multiple instances of the same architecture with different random
initialization seeds. In Naive Ensembling (NE), one trains each member of the
ensemble on the same training dataset, resulting in $N$ networks with
identical architecture but different parameter values. The posterior
distribution for ensembling methods is obtained by computing the average of
deterministic posteriors $\mathbf{p}_{t}(\hat{x}\mid X,Y)$:

$\mathbf{p}(\hat{x}\mid
X,Y)\approx\frac{1}{N}\sum_{i=1}^{N}\mathbf{p}_{i}(\hat{x}\mid
X,Y),\quad\mathbf{p}_{i}(\hat{x}\mid
X,Y)\equiv\begin{bmatrix}f^{(i)}_{\theta}(\hat{x};X,Y)_{1}\\\
f^{(i)}_{\theta}(\hat{x};X,Y)_{2}\\\ \vdots\\\
f^{(i)}_{\theta}(\hat{x};X,Y)_{C}\end{bmatrix}.$ (2.4)

Often, to achieve better generalization and stability, Bootstrapped [34]
Ensembling (BE) (or bagging) is preferred over naive ensembling. This is done
by training each ensemble member on a dataset reorganized by sampling $n$
examples from the full training set with replacement. If the size of the
resampled dataset is equal to that of the original training set, each ensemble
member is expected to see approximately 63% of the original training set. For
classification, it is standard to use the most common label among the ensemble
members as the final prediction, while for regression one usually computes the
empirical mean. When an ensemble consists of a collection of neural networks
trained with respect to a proper scoring rule [35] and often coupled with an
optional adversarial training routine, the ensemble is termed as adeep
ensemble [19].

Ensemble methods are the one of the simplest UQ methods that require no
additional changes to the underlying model architecture, although the high
computational cost in training $M$ architecturally identical models and
performing $M$ forward passes for each prediction often renders them
impractical for some memory or time consuming tasks.

###  2.3 Monte Carlo Dropout

Monte Carlo Dropout is a Bayesian technique introduced in [32], where one
approximates the network’s posterior distribution of class predictions by
collecting samples obtained from multiple forward passes of dropout
regularized networks. Dropout regularization [36] involves random omissions of
hidden features during train time, which is equivalent to masking rows of
weight matrices. Inclusion of dropout layers mitigates model overfitting and
is empirically known to improve model accuracy [36]. A key observation of [32]
is that under suitable assumptions on the bayesian neural network prior and
training procedure, sampling $T$ predictions from the BNN’s posterior is
equivalent to performing $T$ stochastic forward passes with dropout layers
fully activated. In this manner, the full posterior distribution may be
approximated by monte-carlo integration of the posterior softmax probability
vector $\mathbf{p}(x^{*};X,Y)$:

$\mathbf{p}(\hat{x}\mid
X,Y)\approx\frac{1}{T}\sum_{t=1}^{T}\mathbf{p}_{t}(\hat{x}\mid
X,Y),\quad\mathbf{p}_{t}(\hat{x}\mid
X,Y)\equiv\begin{bmatrix}f^{(t)}_{\theta}(\hat{x}\mid X,Y)^{1}\\\
f^{(t)}_{\theta}(\hat{x}\mid X,Y)^{2}\\\ \vdots\\\
f^{(t)}_{\theta}(\hat{x}\mid X,Y)^{C}\end{bmatrix}.$ (2.5)

where $T$ denotes the number of stochastic forward passes. In MCDropout, each
output $f_{\theta}^{(t)}$ from different forward passes $t$ share the same
model weights $\theta$ but have different hidden features of the network
omitted. This way, each prediction from separate stochastic forward passes
effectively uses a slightly different set of hidden features for computing the
final probability values $\mathbf{p}_{t}$. As with ensembling methods, the
final prediction of MCDropout for classification is given by the majority vote
among all stochastic forward passes. For regression, we again compute the
empirical mean. As evident from the apparent similarities, MCDropout networks
may also be interpreted as a form of ensemble learning [36], where each
stochastic forward pass corresponds to a different realization of a trained
neural network.

Implementing MCDropout requires one to modify the underlying neural network
architecture to include dropout layers and configuring them to behave
stochastically during test time. Often the location of dropout layers can
critically affect prediction performance, and for convolutional neural
networks the decision is made via trial-and-error [27]. Also, for memory
intensive tasks such as semantic segmentation, sample collection by multiple
forward passes can accumulate rapidly towards high computational cost, similar
to ensembling methods.

###  2.4 Evidential Deep Learning

Evidential Deep Learning (EDL) [20, 33], refers to a class of deep neural
networks that exploit conjugate prior relationships to model the posterior
distribution analytically. For multi-class classification, the distribution
over the space of all probability vectors $\mathbf{p}=(p_{1},...,p_{c})$ is
modeled by a Dirichlet distribution with $c$ concentration parameters
$\bm{\alpha}=\alpha_{1},...,\alpha_{c}$:

$D(\mathbf{p}\mid\bm{\alpha})=\frac{1}{B(\bm{\alpha})}\prod_{i=1}^{c}p_{i}^{\alpha_{i}-1},$
(2.6)

where $\alpha_{i}\geq 1$ for all $i$, $B(\cdot)$ denotes the $c$-dimensional
multivariate Beta function, and $\mathbf{p}$ is in the $c$-unit simplex
$\mathcal{S}_{c}$:

$\mathcal{S}_{c}=\\{\mathbf{v}\in\mathbb{R}^{c}:\sum_{i=1}^{c}v_{i}=1\\}.$
(2.7)

In contrast to deterministic classification neural networks that minimize the
cross-entropy loss by predicting the class logits, evidential classification
networks predict the concentration parameters $\bm{\alpha}$. The expected
value of the $k$-th class probability under the distribution
$D(\mathbf{p}\mid\bm{\alpha})$ is then given analytically as

$\hat{p}_{k}=\frac{\alpha_{k}}{\sum_{i=1}^{c}\alpha_{i}}.$ (2.8)

To estimate the concentration parameters, several distinct loss functions are
available. Since in practice the concentration parameters are obtained from
neural network predictions, we denote $\bm{\alpha}_{\theta}$ as the predicted
concentration parameters of an EDL model. The marginal likelihood loss (MLL)
is given by:

$\mathcal{L}_{MLL}(\theta)=-\log{\left(\int\prod_{i=1}^{c}p_{i}^{y_{i}}D(\mathbf{p}\mid\bm{\alpha}_{\theta})\
d\mathbf{p}\right)}.$ (2.9)

The Bayes risk (posterior expectation of the risk) of the log-likelihood
(BR-L) formulation yields:

$\mathcal{L}_{BR}(\theta)=\int\left[\sum_{i=1}^{c}-y\log{(p_{i})}\right]D(\mathbf{p}\mid\bm{\alpha}_{\theta})\
d\mathbf{p}.$ (2.10)

The Bayes risk of the Brier score (BR-B) may also be used as an alternative
optimization objective:

$\mathcal{L}_{BS}(\theta)=\int\left\lVert\mathbf{y}-\mathbf{p}\right\rVert_{2}^{2}\
D(\mathbf{p}\mid\bm{\alpha}_{\theta})\ d\mathbf{p}.$ (2.11)

From Sensoy et. al. [20], analytic integration of the aforementioned loss
functions gives closed form expressions that are suited for gradient based
optimization of the parameters $\theta$.

EDL methods have the immediate advantage of requiring only one single pass to
access the full posterior distribution, at a price of restricting the space of
posterior functions onto the appropriate conjugate prior forms. Also, EDL
methods only require one to modify the loss function and the final layer of
its deterministic baseline (if necessary), which allows flexible integration
with complex, hierarchical deep neural architectures similar to the full
LArTPC reconstruction chain. However, as we later observe, EDL methods
generally fall short on various UQ evaluation metrics compared to ensembling
and MCDropout, depending on task specifics.

##  3 Evaluating Uncertainty Quantification Methods

As stated in [19], the goal for uncertainty quantification in deep learning is
two-fold: to achieve better alignment of estimated probability with their
long-run empirical accuracy and to serve as mis-classification or out-of-
distribution alarms that could be used for rejecting erroneous predictions.
The first condition, which we term calibration fidelity, may be evaluated by
plotting the reliability diagrams [37], which are constructed by binning the
predicted probabilities (often termed confidence) into equal sized bins and
plotting the bin centers in the $x$-axis and the empirical accuracy of the bin
members in the $y$-axis.

As calibration fidelity measurements using reliability diagrams are originally
designed for binary classifiers, there have been numerous proposals for their
extensions to multi-class classifiers [38, 39, 40]. We consider two relatively
simple methods; the first is a standard used in Guo et. al. [38], where only
the predicted probability for the most confident prediction of each sample is
used to plot the reliability diagram. We refer to this mode of assessment as
max-confidence calibration fidelity. An alternative method is to evaluate
calibration for each of the $K$ classes separately, as in B. Zadrozny and C.
Elkan [40]. We refer to this mode as marginal calibration fidelity.

The closer the reliability diagram curve is to the diagonal, the better
calibrated a given classifier is, in the sense of calibration fidelity. The
deviation of a given classifier from the diagonal could be summarized by
computing the adaptive calibration error (ACE) [41] for each reliability
diagram:

$ACE=\frac{1}{C}\frac{1}{R}\sum_{c=1}^{C}\sum_{r=1}^{R}|\text{acc}(r,c)-\text{conf}(r,c)|.$
(3.1)

, where

$\text{acc}(r,c)=\frac{\text{\\# of samples in bin $r$, classified as class
$c$}}{\text{\\# of samples in bin $r$}}$ (3.2)

and

$\text{conf}(r,c)=\frac{\text{sum of predicted probabilities $p_{j}^{c}$ in
bin $r$}}{\text{\\# of samples in bin $r$}}.$ (3.3)

In short, $\text{acc}(r,c)$ is the restricted empirical accuracy of samples
with predicted probabilities between the boundaries of bin $r$, and
$\text{conf}(r,c)$ is the average predicted probability for class $c$ within
that same bin $r$. Here, $C$ is the number of unique classes and $R$ denotes
the number of equal-sample bins used to plot the reliability diagram of class
$c$. In contrast to the expected calibration error (ECE) [42] that uses equal-
range bins to compute the summary value, the ACE uses bins with equal number
of samples. As such, ACE is better suited for comparing calibration fidelity
for models with highly skewed probability distributions. Note that the ACE
metric depends on how the reliability diagram is constructed: for definitions
(3.2) and (3.3), we use the marginal reliability diagram. For max-confidence
diagrams, we define $\text{acc}(r)$ to be the empirical accuracy of the most
confidence prediction and $\text{conf}(r)$ is the average maximum predicted
probability of samples in bin $r$.

Another metric of uncertainty quantification measures the model’s
discriminative capacity to mis-classified or out-of-distribution samples. In
practice, uncertainty quantification models have the capacity to reject
predictions based on a numerical estimate of the trustworthiness of the
prediction in question. For example, in a classification setting the entropy
of the predicted softmax probability distribution (predictive entropy) could
be used as a measure of confusion, as entropy is maximized if the predictive
distribution reduces to a uniform distribution over $K$ classes. In this
construction, it is desirable to have the predicted entropy distributions of
correctly and incorrectly classified samples to be as separated as possible.
To compute the extent of distributional separation, we may use the first
Wasserstein distance [43] between the predictive entropy distributions:

$W_{1}(u,v)=\inf_{\pi\in\Gamma(u,v)}\int_{\mathbb{R}\times\mathbb{R}}|x-y|\
d\pi(x,y).$ (3.4)

where $u$ and $v$ are two probability distributions, $\Gamma(u,v)$ is the set
of all joint probability measures in $\mathbb{R}^{2}$. We use the Wasserstein
distance with the $L_{1}$ metric due to its simple computational
implementation [43].

Sensitivity may also be measured by computing the area under the receiver
operating characteristic curve (AUROC), also known as the concordance
statistic (c-statistic) [44]. Using predictive entropy as the threshold value,
the ROC curve is constructed by plotting the false positive rate (incorrect
predictions) in the $x$-axis and the true positive rate (correct predictions)
in the $y$-axis at different threshold levels. In this setting, the AUROC is
the probability that a randomly chosen correct prediction will have a lower
predictive entropy than that of a randomly chosen incorrect prediction [45].

##  4 Datasets and Network Architectures

Dataset: As previously stated, we use the PiLArNet [1] public dataset for
semantic segmentation and multi-particle classification. PiLArNet[1] is a
collection of simulated 2D/3D images prepared for developing data analysis
techniques for LArTPC experiments. In the first step, two custom
algorithms–MultiParticleVertex (MPV) and MultiParticleRain (MPR)–generate a
list of particles to be produced in each image. The MPV generator produces $N$
particles from a common 3D spatial point, while for the MPR generator, the
particles may originate from different spatial points in the image. The MPV
and MPR generator is designed to emulate neutrino interactions (MPV) and
cosmic rays (MPR). Once each particle has been produced by MPV and MPR
generators, we use the GEANT4 [46] particle tracking simulation to produce the
energy depositions.

For semantic segmentation and multi-particle classification, we use the 768px
cubic PiLArNet dataset to train and test our models. For single-particle
classification, we use a 1024px cubic dataset, which is generated using the
same algorithm and parameters (apart from the difference in total resolution)
as those of the PiLArNet 768px data.

Figure 2: Sparse-CNN layer definitions for architecture reference. Figure 3:
Sparse-CNN block definitions for architecture reference. Figure 4: Sparse-CNN
architecture for single particle classifiers. Figure 5: Sparse-CNN
architecture for semantic segmentation networks. Figure 6: Architecture
outline of multi-particle classification network. The geometric node encoder
extracts hand-engineered features relevant to particle classification. The
colors in the right-most image represent the predicted particle type (blue-
photon, red-electron, orange-muon, green-pion, magenta-proton).

Network Architecture: The LArTPC images consist of 3D point clouds, in which
after voxelization less than approximately 1% of the pixels are non-zero. In
such a scenario, using standard 3D convolutions over the full grid is not only
computationally expensive but also does not process useful information during
operation. As such, in 3D applications it is much more common to build
convolutional neural network architectures using sparse convolutions [47] that
operate solely on active pixels of the full image. Unless stated otherwise, we
use MinkowskiEngine’s [47] sparse 3D convolutional layers to build our 3D
image processing architectures.

Figure 2 defines the color coding used to represent different architecture
components used to describe different models we use to compare UQ methods.
Figure 3 illustrates definitions for composite blocks that consists of two or
more elementary layers in 2. Figures 4, 5, and 6 show schematic diagrams of
architectures used to evaluate three tasks: single particle classification,
particle shape semantic segmentation, and multi particle classification,
respectively.

Single Particle Classification: In the first task, we implement and assess the
different UQ models on the simpler task of single particle classification. The
single particle dataset consists of 1024 $\times$ 1024 $\times$ 1024 3D images
each containing only one particle, where all voxels in the given image belong
to the same particle ID. The 3D images have one feature dimension
corresponding to the amount of energy deposited in a one-voxel equivalent
region of the detector. We use a ResNet [48] type encoder with dropout [36]
regularization, where convolution operations are substituted by sparse
convolutions implemented in the MinkowskiEngine library [47]. For standard
deterministic models, ensembles, and MCDropout, the final prediction
probabilities are given by softmax activations, whereas for evidential models
the concentration parameters $\alpha$ are computed from Softplus [49]
activations. The single particle dataset contains five particle classes:
photon showers ($\gamma$), electron showers ($e$), muons ($\mu$), pions
($\pi$), and protons ($p$). A diagram of the neural network architecture is
shown in Figure 2.

Semantic Segmentation As segmentation is a classification task on individual
pixels, the details of the implementation are mostly identical to those of
single particle classification. We employ a Sparse-UResNet [23] with dropout
layers in the deeper half of the network as the base architecture for semantic
segmentation networks and use the 768px resolution PILArNet [1] MultiPartRain
(MPR) and MultiPartVertex (MPV) datasets for multiple particle datasets. The
five semantic labels provided by PiLArNet consists of the following:

  * •

Shower Fragments: connected components of electromagnetic showers that are
above a set voxel count and energy deposition threshold.

  * •

Tracks: particle trajectories that resemble straight lines, mostly originating
from muon, pion, and protons.

  * •

Michel Electrons: an electron produced by muon decay at rest.

  * •

Delta Rays: electrons produced from muon tracks via hard scattering

  * •

Low Energy Depositions: clouds of low energy depositions of electromagnetic
showers which are not labeled as shower fragments.

Multi Particle Reconstruction The MPV/MPR dataset also contains particle type
labels for each particle instance in a given image. For multi particle
classification, we take each cluster of voxels that belong to the same
particle and reduce the resulting groups of point clouds into 1-dimensional
feature vectors. The node embeddings of each particle consists of geometric
features such as the trajectory start point, initial direction vector, and
mean energy deposition. These feature vectors are then given as input node
features to a graph neural network, which performs three message passing
operations to incorporate inter-particle relational information.

##  5 Results

In the ensuing discussions, we use the following abbreviations: Det
(Deterministic), NE (Naive Ensembles), BE (Bootstrapped Ensembles), MCD
(MCDropout), EDL-MLL (Evidential Deep Learning with Marginal Log-Likelihood),
EDL-BR (EDL with Bayes Risk), and EDL-B (EDL with Brier Score) to label each
model in Figures 7 to 17.

###  5.1 Training Details

The training set consists of 80k images, and the original test set was
separated into a 2k validation set used for model selection and a 18k test set
used for selected model evaluation with high statistics. All models were
trained until the validation accuracy plateaued, and the network weights that
achieved the highest validation accuracy were selected for further evaluation
on a separate test set. To fully account for possible variations in model
accuracy and uncertainty quantification quality due to randomized factors such
as parameter initialization, the model selection procedure was repeated for
five different random seeds for each model, except for ensemble methods. This
results in five independently trained models that share the same architecture
but differing in parameter values. For ensembling methods (NE and BE), the
results in Figures 7 to 17 represent predictions from a single ensemble model
with five independently trained ensemble members. As such, we mark the results
for NE and BE with an asterisk. We used the Adam optimizer [50] with decoupled
weight decay [51].

Single Particle Classification: Figure 7 shows the predictive entropy
distribution, accuracy, and the $W_{1}$ distance for the single particle
classification models. We observe that the distributional separation as
measured in $W_{1}$ is largest for the ensemble methods, while the Evidential
model trained on the Brier score is also competitive. In general, ensemble
methods achieve highest accuracy with better distributional separation
compared to Monte Carlo Dropout and Evidential models. The AUROC values in
figure 8 also reflect the superior discriminative capacity of ensembling.

The calibration curves for single particle classification are shown in the top
row of figure 16, and figure 9 illustrates the adaptive calibration error
(ACE) values across different subsets of the test set partitioned by true
particle ID labels. While all UQ models with the possible exception of EDL-
BR-B achieve better calibration compared to standard deterministic neural
networks, ensembling methods have the least max-confidence and marginal ACE
values.

Figure 7: Predictive entropy distribution, accuracy, and 1-Wasserstein
distance for single particle classification. Figure 8: Single particle ROC and
percentage rejection curves. Figure 9: Single particle classification adaptive
calibration errors (ACEs) for each model and class.

Semantic Segmentation: For segmentation, the best distributional separation is
achieved by Evidential models, which is exhibited in Figure 10. The ensemble
methods have the highest accuracy and AUROC scores as is shown in figure 11.
It is interesting to note that while distributional separation measured in
$W_{1}$ is greatest for Evidential models, the calibration fidelity falls
short even with respect to standard deterministic models. As with single
particle classification, the best calibration fidelity is realized by ensemble
methods.

Figure 10: Predictive entropy distribution, accuracy, and 1-Wasserstein
distance for semantic segmentation. Figure 11: Semantic segmentation adaptive
calibration errors (ACEs) for each model and class. Figure 12: Semantic
segmentation ROC and percentage rejection curves.

Multi Particle Reconstruction: Since the contextual information which is
useful in determining a given particle’s ID can only be used in a multi-
particle setting, we expect a gain in accuracy from the single particle
datasets. This approach leads to an overall approximate $5\%$ increase in
classification accuracy in all models. Again, ensemble methods provide the
highest $W_{1}$ distance, overall accuracy, and AUROC values (figures 13, 15)
and the best calibration fidelity (figure 14).

Figure 13: Predictive entropy distribution, accuracy, and 1-Wasserstein
distance for multi particle classification. Figure 14: Multi particle
classification adaptive calibration errors (ACEs) for each model and class.

The full reliability plots used to calculate ACE values are provided in
figures 16 and 17. A tabular summary of results is available in Table 1, where
each row corresponds to a different UQ approach and each column corresponds to
a different performance metric. We utilize bold font to highlight the best
performing approach for a specific performance metric (that is, best
performance for the entire column). Across different tasks and evaluation
metrics, we see that ensembling methods provide the best performance. Of
these, we find that Bootstrapped Ensembles exhibit superior performance in
most cases. Additionally, we highlight the fact that for some approaches such
as EDL-BR, the performance of the uncertainty enabled model is inferior to the
baseline deterministic model. For instance, EDL-BR is not as accurate as the
deterministic model for all three test cases of Semantic Segmentation, Single-
Particle and Multi-Particle Classification. Similarly, both EDL-Brier and EDL-
BR exhibit inferior AUROC scores than the deterministic approach across the
test cases.

Figure 15: Multi particle ROC and percentage rejection curves. Figure 16:
Reliability plots for single and multi-particle classification. Figure 17:
Reliability plots for semantic segmentation.

Table 1: Uncertainty Quantification Evaluation Metrics. | Single Particle Classification | Semantic Segmentation | Multi-Particle Classification
---|---|---|---
| Accuracy | AUROC | $W_{1}$ | ECE |  | Accuracy | AUROC | $W_{1}$ | ECE |  | Accuracy | AUROC | $W_{1}$ | ECE |
Deterministic | 0.8656 | 0.753 | 0.118 | 15.55% |  | 0.9821 | 0.954 | 0.451 | 0.59% |  | 0.9604 | 0.938 | 0.379 | 1.53% |
Naive Ensembles | 0.8844 | 0.827 | 0.328 | 5.02% |  | 0.9839 | 0.962 | 0.520 | 0.09% |  | 0.9640 | 0.944 | 0.420 | 0.56% |
Bootstrap Ensembles | 0.9014 | 0.842 | 0.373 | 4.35% |  | 0.9837 | 0.962 | 0.521 | 0.10% |  | 0.9644 | 0.942 | 0.418 | 0.51% |
MC Dropout | 0.8734 | 0.795 | 0.250 | 8.35% |  | 0.9821 | 0.957 | 0.465 | 0.41% |  | – | – | – | – |
EDL-MLL | 0.8622 | 0.762 | 0.301 | 10.36% |  | 0.9803 | 0.950 | 0.961 | 2.37% |  | 0.9604 | 0.935 | 0.394 | 1.13% |
EDL-BR | 0.8253 | 0.701 | 0.214 | 14.49% |  | 0.9818 | 0.948 | 1.049 | 2.59% |  | 0.9223 | 0.900 | 0.363 | 4.15% |
EDL-Brier | 0.8751 | 0.748 | 0.325 | 7.36% |  | 0.9806 | 0.953 | 1.028 | 4.70% |  | 0.9596 | 0.911 | 0.366 | 1.40% |

Table 2: Average time spent per iteration (seconds). | Single Particle Classification |  | Semantic Segmentation |  | Multi-Particle Classification
---|---|---|---|---|---
| Train (256) | Test (100) |  | Train (64) | Test (100) |  | Train (128) | Test (100)
Deterministic | 0.873 | 0.098 |  | 1.047 | 0.211 |  | 5.056 | 3.284
Naive Ensembles | 0.873 ($\times 5$) | 0.098 ($\times 5$) |  | 1.047 ($\times 5$) | 0.211 ($\times 5$) |  | 5.056 ($\times 5$) | 3.284 ($\times 5$)
Bootstrap Ensembles | 4.84 ($\times 5$) | 0.096 ($\times 5$) |  | 1.386 ($\times 5$) | 0.176 ($\times 5$) |  | 6.587 ($\times 5$) | 2.993 ($\times 5$)
MC Dropout | 0.873 | 2.581 |  | 1.047 | 2.049 |  | – | –
EDL-MLL | 0.776 | 0.067 |  | 0.964 | 0.235 |  | 5.650 | 2.899
EDL-BR | 0.410 | 0.088 |  | 1.058 | 0.280 |  | 5.280 | 3.610
EDL-Brier | 0.790 | 0.089 |  | 1.114 | 0.276 |  | 5.918 | 3.602

Table 3: Average memory consumption per iteration (GBs). | Single Particle Classification |  | Semantic Segmentation |  | Multi-Particle Classification
---|---|---|---|---|---
| Train (256) | Test (100) |  | Train (64) | Test (100) |  | Train (128) | Test (100)
Deterministic | 4.245 | 0.66 |  | 3.26 | 0.90 |  | 0.65 | 0.11
Naive Ensembles | 4.245 ($\times 5$) | 0.66 ($\times 5$) |  | 3.26 ($\times 5$) | 0.90 ($\times 5$) |  | 0.65 ($\times 5$) | 0.11 ($\times 5$)
Bootstrap Ensembles | 4.454 ($\times 5$) | 0.66 ($\times 5$) |  | 3.42 ($\times 5$) | 0.90 ($\times 5$) |  | 0.65 ($\times 5$) | 0.11 ($\times 5$)
MC Dropout | 4.245 | 0.66 |  | 3.26 | 0.95 |  | - | -
EDL-MLL | 4.250 | 0.66 |  | 3.31 | 0.90 |  | 0.65 | 0.11
EDL-BR | 4.260 | 0.66 |  | 3.31 | 0.90 |  | 0.65 | 0.11
EDL-Brier | 4.249 | 0.66 |  | 3.33 | 0.90 |  | 0.65 | 0.11

We include a brief summary of the required time complexity and GPU memory
requirements of each model in tables 2 and 3 respectively. The batch size is
denoted in the Train/Test sub-column. The time information corresponds to the
CPU time it takes for the model to run one iteration of training or evaluation
routine with the denoted batch size. For training, this includes the time
required for both model forwarding and gradient backpropagation, while for
inference we compute the sum of evaluation mode model forwarding time and
other post-processing operations (for example, in MC Dropout we have a sample
averaging procedure needed to obtain probability values). The memory value is
computed by taking the average of the maximum required GPU memory across 5
different samples. Note that the values for deterministic and naive ensembles
are identical, since naive ensembles were constructed from trained
deterministic models.

Table 3 shows that memory consumption for each model is roughly uniform across
different models, with the exception of ensembling methods which naturally
require multiple independent forward operations ($N=5$) for each prediction.
While it is possible to run the ensembling methods as fast as single network
(non-ensembling) models by parallelization, for computationally lightweight
systems this may not be feasible. Similarly, the average run times shown in
Table 2 also have the same order of magnitude across different models except
for naive and bootstrapped ensembles that scale up with the number of
constituent models in the ensemble.

Calibration fidelity cannot be examined in a single image basis, as
calibration is a collective property that must be measured by appropriate
binning of the test set predictions. However, it is possible to assess the
discriminative capacity by observing samples with antipodal entropy values.
With predictive entropy values in hand, the class predictions may be divided
into four categories: 1) confident (low entropy) correct predictions, 2)
uncertain (high entropy) correct predictions, 3) confident errors, and 4)
uncertain errors. Among the four groups, confident errors are most problematic
for robust design of deep learning models. Some representative examples are
shown in figures 18, 19, and 20. Figure 18 is a high-entropy misclassification
example, in which the network cannot confidently decide whether the set of
voxels circled in red is a muon or a pion, in contrast with the confident
predictions it gives for the pair of photons in and the vertex-attached
proton. It is likely that the model cannot determine the circled track as a
muon or a pion with certainty, since no inter-particle information such as
michel electrons or pion secondary interactions are available for the circled
track. In figures 19 and 20, the network predicts the vertex-attached shower
as an electron with high probability, while for the two $\mu$ and $\pi$ pair
it retains some level of uncertainty. Hence, we observe that the assessment of
the network on mis-identifying the shower as an electron is partly justified,
as it is difficult to distinguish a photon shower attached to an interaction
vertex from an electron shower.

Figure 18: Example high-entropy error from a multi-particle evidential GNN
(EDL-BR). Figure 19: Example low-entropy prediction from a multi-particle
evidential GNN (EDL-BR). (blue-photon, red-electron, orange-muon, green-pion,
magenta-proton)  Figure 20: Example low-entropy prediction from a multi-
particle evidential GNN (EDL-BR). Green and red indicate high and low
probability values, respectively.

##  6 Discussion

In this investigation, we evaluated three different uncertainty quantification
methods for deep neural networks on the task of single particle
classification, multi-particle classification, and semantic segmentation using
high resolution 3D LArTPC energy deposition images. The various metrics
evaluating calibration fidelity and discriminative capacity lead to a notable
conclusion: simple ensembling of few independently trained neural networks
generally achieves highest accuracy and best calibration of output probability
values. Also, we observe that the quality of uncertainty quantification
depends greatly on the type of the classifier’s task, and it is possible for
Bayesian models to perform worse than deterministic networks in calibration.

Often, the choices made in hyperparameters and neural network architecture
significantly affect the classifier’s capacity to achieve the desired
performance. It is important to note that the UQ methods presented in this
paper do not consider the effects of the structure and hyperparameters of our
models. Extant deep learning uncertainty quantification approaches can only
account for aleatoric uncertainty and the parameter uncertainty component of
epistemic uncertainty. Thus, these methods are unable to account for
structural (or model form) uncertainty, that is a component of epistemic
uncertainty. While a complete description of epistemic uncertainty is often
intractable in practice, it is desirable to assess how much of the variability
in a deep classifier’s predictions could be attributed to hyperparameter and
structural diversity.

Additionally, most deep learning uncertainty quantification approaches
introduce hyperparameters regarding their usage that need to be selected. The
performance of the uncertainty estimation is sensitive to these choices. For
instance, in the application of MCDropout, the hyperparameters include the
locations of the dropout layers in the network and the dropout probabilities
of each layer. Prior investigations have observed that the uncertainty
quantification capabilities of this approach are highly affected by these
choices, where for instance, choosing too many dropout layers or too strong a
dropout probability can lead to overly conservative estimates [52]. Similarly,
for ensembling approaches, number of constituent models in the ensemble needs
to be ascertained; in variation Bayes approaches the scaling between the
negative-log-likelihood and the KL-divergence losses needs to be estimated for
best performance [17], etc. In this investigation, we carried out such
hyperparameter selection using manual exploration over a validation dataset.

While out-of-distribution and mis-classification resilience of uncertainty
quantifying neural networks may be used for rejecting unreliable predictions,
obtaining calibrated probability estimates would provide further credibility
in using deep learning techniques for physical sciences. Post-hoc calibration
methods such as temperature scaling [38] train a calibration model (for
temperature scaling, a single parameter calibration model) after training to
obtain calibrated probabilities from a deterministic neural network. As post-
hoc methods do not require the classifier to be re-modeled and trained from
their initial state, such methods may be better suited for ensuring proper
calibration of classifiers with lower computational budget. Future work will
include evaluation of uncertainty quantifying neural networks and post-hoc
calibration methods for a full neutrino physics signal/background classifier,
which is built on top of the separate tasks of particle classification and
segmentation.

## Acknowledgment

This work was supported in part by funding from Zoox, Inc. This work is
supported by the U.S. Department of Energy, Office of Science, Office of High
Energy Physics, and Early Career Research Program under Contract DE-
AC02-76SF00515.

## References

  * [1] C. Adams, K. Terao, and T. Wongjirad, “Pilarnet: Public dataset for particle imaging liquid argon detectors in high energy physics,” ArXiv, vol. abs/2006.01993, 2020.
  * [2] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (Los Alamitos, CA, USA), pp. 427–436, IEEE Computer Society, jun 2015.
  * [3] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” ArXiv, vol. abs/1610.02136, 2016.
  * [4] N. T. S. B. (NTSB), “Tesla Crash Preliminary Evaluation Report ODI Resume. Investigation: PE 16-007,” tech. rep., NATIONAL TRANSPORTATION SAFETY BOARD, 2017.
  * [5] N. T. S. B. (NTSB), “Collision between a Car Operating with Automated Vehicle Control Systems and a Tractor-Semitrailer Truck, NTSB/HAR-17-XX,” tech. rep., NATIONAL TRANSPORTATION SAFETY BOARD, 2016.
  * [6] N. T. S. B. (NTSB), “Collision Between a Sport Utility Vehicle Operating With Partial Driving Automation and a Crash Attenuator, NTSB/HAR-20/01,” tech. rep., NATIONAL TRANSPORTATION SAFETY BOARD, 2020.
  * [7] L. Dodds, “Chinese businesswoman accused of jaywalking after ai camera spots her face on an advert,” The Telegraph, 2018.
  * [8] M. Sundararajan, A. Taly, and Q. Yan, “Axiomatic attribution for deep networks,” in Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 3319–3328, JMLR.org, 2017.
  * [9] A. G. Wilson and P. Izmailov, “Bayesian deep learning and a probabilistic perspective of generalization,” Advances in neural information processing systems, vol. 33, pp. 4697–4708, 2020.
  * [10] L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
  * [11] A. Krizhevsky, “Learning multiple layers of features from tiny images,” tech. rep., 2009.
  * [12] C. Rubbia, M. Antonello, P. Aprili, B. Baibussinov, M. B. Ceolin, L. Barze, P. Benetti, E. Calligarich, N. Canci, F. Carbonara, et al., “Underground operation of the icarus t600 lar-tpc: first results,” Journal of Instrumentation, vol. 6, no. 07, p. P07011, 2011.
  * [13] B. Abi, R. Acciarri, M. Acero, M. Adamowski, C. Adams, D. Adams, P. Adamson, M. Adinolfi, Z. Ahmad, C. Albright, et al., “The dune far detector interim design report volume 1: Physics, technology and strategies,” 2018.
  * [14] D. Collaboration et al., “Long-baseline neutrino facility (lbnf) and deep underground neutrino experiment (dune) conceptual design report volume 1: The lbnf and dune projects,” arxiv. org, 2016.
  * [15] Z. Nado, N. Band, M. Collier, J. Djolonga, M. W. Dusenberry, S. Farquhar, Q. Feng, A. Filos, M. Havasi, R. Jenatton, et al., “Uncertainty baselines: Benchmarks for uncertainty & robustness in deep learning,” arXiv preprint arXiv:2106.04015, 2021.
  * [16] A. F. Psaros, X. Meng, Z. Zou, L. Guo, and G. E. Karniadakis, “Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons,” Journal of Computational Physics, vol. 477, p. 111902, 2023\.
  * [17] A. A. Mishra, A. Edelen, A. Hanuka, and C. Mayes, “Uncertainty quantification for deep learning in particle accelerator applications,” Physical Review Accelerators and Beams, vol. 24, no. 11, p. 114601, 2021.
  * [18] B. Efron and R. J. Tibshirani, An introduction to the bootstrap.  CRC press, 1994.
  * [19] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, (Red Hook, NY, USA), p. 6405–6416, Curran Associates Inc., 2017.
  * [20] M. Sensoy, L. Kaplan, and M. Kandemir, “Evidential deep learning to quantify classification uncertainty,” in Advances in Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), vol. 31, Curran Associates, Inc., 2018\.
  * [21] R. M. Neal, Bayesian learning for neural networks, vol. 118.  Springer Science & Business Media, 2012.
  * [22] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, p. 1050–1059, JMLR.org, 2016.
  * [23] L. Dominé and K. Terao, “Scalable deep convolutional neural networks for sparse, locally dense liquid argon time projection chamber data,” Phys. Rev. D, vol. 102, p. 012005, Jul 2020.
  * [24] Y. Guo, Y. Liu, T. Georgiou, and M. S. Lew, “A review of semantic segmentation using deep neural networks,” International journal of multimedia information retrieval, vol. 7, pp. 87–93, 2018.
  * [25] P. Abratenko, M. Alrashed, R. An, J. Anthony, J. Asaadi, A. Ashkenazi, S. Balasubramanian, B. Baller, C. Barnes, G. Barr, V. Basque, L. Bathe-Peters, O. Benevides Rodrigues, S. Berkman, A. Bhanderi, A. Bhat, M. Bishai, A. Blake, T. Bolton, L. Camilleri, D. Caratelli, I. Caro Terrazas, R. Castillo Fernandez, F. Cavanna, G. Cerati, Y. Chen, E. Church, D. Cianci, J. M. Conrad, M. Convery, L. Cooper-Troendle, J. I. Crespo-Anadón, M. Del Tutto, S. R. Dennis, A. Devitt, R. Diurba, R. Dorrill, K. Duffy, S. Dytman, B. Eberly, A. Ereditato, J. J. Evans, G. A. Fiorentini Aguirre, R. S. Fitzpatrick, B. T. Fleming, N. Foppiani, D. Franco, A. P. Furmanski, D. Garcia-Gamez, S. Gardiner, G. Ge, S. Gollapinni, O. Goodwin, E. Gramellini, P. Green, H. Greenlee, W. Gu, R. Guenette, P. Guzowski, L. Hagaman, E. Hall, P. Hamilton, O. Hen, G. A. Horton-Smith, A. Hourlier, R. Itay, C. James, J. Jan de Vries, X. Ji, L. Jiang, J. H. Jo, R. A. Johnson, Y.-J. Jwa, N. Kamp, N. Kaneshige, G. Karagiorgi, W. Ketchum, B. Kirby, M. Kirby, T. Kobilarcik, I. Kreslo, R. LaZur, I. Lepetic, K. Li, Y. Li, B. R. Littlejohn, W. C. Louis, X. Luo, A. Marchionni, C. Mariani, D. Marsden, J. Marshall, J. Martin-Albo, D. A. Martinez Caicedo, K. Mason, A. Mastbaum, N. McConkey, V. Meddage, T. Mettler, K. Miller, J. Mills, K. Mistry, A. Mogan, T. Mohayai, J. Moon, M. Mooney, A. F. Moor, C. D. Moore, L. Mora Lepin, J. Mousseau, M. Murphy, D. Naples, A. Navrer-Agasson, R. K. Neely, P. Nienaber, J. Nowak, O. Palamara, V. Paolone, A. Papadopoulou, V. Papavassiliou, S. F. Pate, A. Paudel, Z. Pavlovic, E. Piasetzky, I. D. Ponce-Pinto, S. Prince, X. Qian, J. L. Raaf, V. Radeka, A. Rafique, M. Reggiani-Guzzo, L. Ren, L. Rochester, J. Rodriguez Rondon, H. E. Rogers, M. Rosenberg, M. Ross-Lonergan, B. Russell, G. Scanavini, D. W. Schmitz, A. Schukraft, W. Seligman, M. H. Shaevitz, R. Sharankova, J. Sinclair, A. Smith, E. L. Snider, M. Soderberg, S. Söldner-Rembold, S. R. Soleti, P. Spentzouris, J. Spitz, M. Stancari, J. S. John, T. Strauss, K. Sutton, S. Sword-Fehlberg, A. M. Szelc, N. Tagg, W. Tang, K. Terao, C. Thorpe, M. Toups, Y.-T. Tsai, M. A. Uchida, T. Usher, W. Van De Pontseele, B. Viren, M. Weber, H. Wei, Z. Williams, S. Wolbers, T. Wongjirad, M. Wospakrik, W. Wu, E. Yandel, T. Yang, G. Yarbrough, L. E. Yates, G. P. Zeller, J. Zennamo, and C. Zhang, “Semantic segmentation with a sparse convolutional neural network for event reconstruction in microboone,” Phys. Rev. D, vol. 103, p. 052012, Mar 2021.
  * [26] R. C. Smith, Uncertainty quantification: theory, implementation, and applications, vol. 12.  Siam, 2013.
  * [27] A. Kendall and Y. Gal, “What uncertainties do we need in bayesian deep learning for computer vision?,” in Advances in neural information processing systems, pp. 5574–5584, 2017.
  * [28] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra, “Weight uncertainty in neural network,” in International conference on machine learning, pp. 1613–1622, PMLR, 2015.
  * [29] J. Antorán, J. Allingham, and J. M. Hernández-Lobato, “Depth uncertainty in neural networks,” Advances in neural information processing systems, vol. 33, pp. 10620–10634, 2020.
  * [30] M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi, “A review of uncertainty quantification in deep learning: Techniques, applications and challenges,” Information Fusion, vol. 76, pp. 243–297, 2021.
  * [31] H. Wang and D.-Y. Yeung, “A survey on bayesian deep learning,” ACM Comput. Surv., vol. 53, sep 2020.
  * [32] Y. Gal and Z. Ghahramani, “Dropout as a bayesian approximation: Representing model uncertainty in deep learning,” in Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, p. 1050–1059, JMLR.org, 2016.
  * [33] A. Amini, W. Schwarting, A. Soleimany, and D. Rus, “Deep evidential regression,” ArXiv, vol. abs/1910.02600, 2020.
  * [34] C. Z. Mooney, R. D. Duval, and R. Duvall, Bootstrapping: A nonparametric approach to statistical inference.  No. 95, sage, 1993.
  * [35] T. Gneiting and A. E. Raftery, “Strictly proper scoring rules, prediction, and estimation,” Journal of the American Statistical Association, vol. 102, no. 477, pp. 359–378, 2007.
  * [36] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” Journal of Machine Learning Research, vol. 15, no. 56, pp. 1929–1958, 2014.
  * [37] H. C. Hartmann, T. C. Pagano, S. Sorooshian, and R. Bales, “Confidence builders: Evaluating seasonal climate forecasts from user perspectives,” Bulletin of the American Meteorological Society, vol. 83, no. 5, pp. 683 – 698, 2002.
  * [38] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 1321–1330, 2017.
  * [39] D. Widmann, F. Lindsten, and D. Zachariah, “Calibration tests in multi-class classification: A unifying framework,” in Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019.
  * [40] B. Zadrozny and C. Elkan, “Transforming classifier scores into accurate multiclass probability estimates,” in Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’02, (New York, NY, USA), p. 694–699, Association for Computing Machinery, 2002\.
  * [41] J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, “Measuring calibration in deep learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019\.
  * [42] M. P. Naeini, G. F. Cooper, and M. Hauskrecht, “Obtaining well calibrated probabilities using bayesian binning,” Proceedings of the … AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, vol. 2015, pp. 2901–2907, Jan 2015.
  * [43] A. Ramdas, N. G. Trillos, and M. Cuturi, “On wasserstein two-sample testing and related families of nonparametric tests,” Entropy, vol. 19, no. 2, 2017\.
  * [44] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning.  Springer Series in Statistics, New York, NY, USA: Springer New York Inc., 2001.
  * [45] T. Fawcett, “An introduction to roc analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006.
  * [46] S. Agostinelli et al., “GEANT4: A Simulation toolkit,” Nucl. Instrum. Meth., vol. A506, pp. 250–303, 2003.
  * [47] C. Choy, J. Gwak, and S. Savarese, “4d spatio-temporal convnets: Minkowski convolutional neural networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075–3084, 2019.
  * [48] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [49] H. Zheng, Z. Yang, W. Liu, J. Liang, and Y. Li, “Improving deep neural networks using softplus units,” in 2015 International Joint Conference on Neural Networks (IJCNN), pp. 1–4, 2015.
  * [50] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [51] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in International Conference on Learning Representations, 2019.
  * [52] A. Kendall, V. Badrinarayanan, and R. Cipolla, “Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding,” arXiv preprint arXiv:1511.02680, 2015.