# Anomaly detection with flow-based fast calorimeter simulators

Claudius Krause<EMAIL_ADDRESS>Institut für Theoretische Physik,
Universität Heidelberg, Philosophenweg 12, 69120 Heidelberg, Germany
Institute of High Energy Physics (HEPHY), Austrian Academy of Sciences (OeAW),
Georg-Coch-Platz 2, A-1010 Vienna, Austria  Benjamin Nachman
<EMAIL_ADDRESS>Physics Division, Lawrence Berkeley National Laboratory,
Berkeley, CA 94720, USA  Berkeley Institute for Data Science, University of
California, Berkeley, CA 94720, USA  Ian Pang<EMAIL_ADDRESS>NHETC, Department of Physics and Astronomy, Rutgers University, Piscataway, NJ
08854, USA  David Shih<EMAIL_ADDRESS>NHETC, Department of Physics
and Astronomy, Rutgers University, Piscataway, NJ 08854, USA  Yunhao Zhu
<EMAIL_ADDRESS>NHETC, Department of Physics and Astronomy, Rutgers
University, Piscataway, NJ 08854, USA  Department of Physics, Northeastern
University, Boston, MA 02115, USA

###### Abstract

Recently, several normalizing flow-based deep generative models have been
proposed to accelerate the simulation of calorimeter showers. Using CaloFlow
as an example, we show that these models can simultaneously perform
unsupervised anomaly detection with no additional training cost. As a
demonstration, we consider electromagnetic showers initiated by one
(background) or multiple (signal) photons. The CaloFlow model is designed to
generate single photon showers, but it also provides access to the shower
likelihood. We use this likelihood as an anomaly score and study the showers
tagged as being unlikely. As expected, the tagger struggles when the signal
photons are nearly collinear, but is otherwise effective. This approach is
complementary to a supervised classifier trained on only specific signal
models using the same low-level calorimeter inputs. While the supervised
classifier is also highly effective at unseen signal models, the unsupervised
method is more sensitive in certain regions and thus we expect that the
ultimate performance will require a combination of approaches.

##  I Introduction

In 2012, the final piece of the Standard Model, the Higgs boson, was
discovered at the Large Hadron Collider (LHC) by the ATLAS [1] and CMS [2]
collaborations. Despite this milestone, compelling theoretical and
experimental reasons continue to drive the search for physics Beyond the
Standard Model (BSM). Regrettably, the extensive search programs conducted by
ATLAS [3, 4, 5], CMS [6, 7, 8], and LHCb [9] at the LHC have yet to yield
conclusive evidence for new BSM physics. Given the impossibility of conducting
dedicated searches for every conceivable BSM scenario, most LHC searches
target specific signals derived from theoretical priors, leaving substantial
portions of the LHC phase space unexplored. This limitation has spurred the
development of more model-agnostic search strategies, with the hope of
detecting BSM physics at the LHC. The advent of deep learning has given rise
to various model-agnostic anomaly detection methods designed to explore
uncharted territories of the LHC phase space — for reviews and original
references, see e.g. [10, 11, 12, 13, 14].

Normalizing flows [15, 16, 17, 18, 19, 20] represent a class of deep learning
models particularly valuable for generative modeling and density estimation
tasks. A normalizing flow is characterized by a parametric diffeomorphism
$f_{\theta}$ mapping between a latent space, with a known distribution
$\pi(z)$, and a data space of interest with an analytically unknown
distribution $p(x)$. In the context of a conditional normalizing flow, this
transformation becomes $f_{\theta}(x|c)$, where $c$ denotes the conditional
inputs to the flow. It is defined through a series of invertible functions,
parameterized by $\theta$, that can be trained by maximizing the log-
likelihood of the data following the change of variables formula:

$\log\bigl(p(x|c)\bigr{missing})=\log(\pi(f_{\theta}(x|c)))+\log|\det\left(\mathcal{J}\bigl{(}f_{\theta}(x|c)\bigr{)}\right)\Bigr{|},$

where $\mathcal{J}\bigl{(}f_{\theta}(x|c)\bigr{)}$ represents the Jacobian of
the transformation $f_{\theta}(x|c)$. The allowable transformations must also
have a computationally tractable Jacobian, ideally efficient to compute, and
the probability density of the base distribution $\pi(z)$ must be known. A
common choice for $\pi(z)$ is the standard normal distribution.

Recently, normalizing flows have found successful applications in fast
calorimeter simulation tasks [39, 40, 43, 42, 44, 51]. In this paper, we
demonstrate the utility of these flow-based generative models as unsupervised
anomaly detectors for identifying BSM physics in calorimeter shower data.
Specifically, we apply CaloFlow [39], a flow-based fast calorimeter simulation
model, to single-photon showers from a new sampling calorimeter version [52]
of the CaloGAN dataset [53, 21, 22]. In this context, single-photon showers
serve as the background events, while the signal events consist of photon
showers originating from a generic BSM particle $\chi$ that undergoes the
decay $\chi\to\gamma\gamma$. The $\chi$ particle is taken to be invisible and
only interacts indirectly with the calorimeter through its decay products. By
training CaloFlow to maximize the log-likelihood when evaluated on background
events, we are able to detect the signal events based on a cut on the log-
likelihood. We focus on achieving signal sensitivity, but the approach could
be combined with a number of background estimation strategies [54, 11].

While we believe this is the first unification of simulation and anomaly
detection, both subjects have been well-studied with machine learning. Many
deep generative models have been studied for calorimeter simulation [21, 22,
23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 35, 37, 38, 39, 40,
41, 42, 43, 44, 45, 46, 47, 48, 49, 50], including a number of proposals
developed on the CaloChallenge datasets [55], and they are also now being
integrated into experimental workflows [36]. We focus on normalizing flows
since they give direct access to the likelihood. This information can also be
extracted from diffusion models [56] and it would be interesting to compare
approaches in the future. For anomaly detection, unsupervised methods have
been extensively studied (e.g. Refs. [57, 58, 59] and many others) and also
include density-based approaches [12]. Like the density-based methods, we use
the likelihood directly as the anomaly score.111 Note that this is not unique
and is sensitive to how the data are represented/preprocessed [60, 61, 62].

There may be other ways of reusing the generative model for BSM searches,
including fine-tuning supervised models based on particular signal hypotheses.

This paper is organized as follows. In Sec. II, we describe calorimeter setup
and the datasets that were used during training/evaluation. In Sec. III, we
explain how CaloFlow is used as an unsupervised anomaly detector by placing
cuts on the log-likelihood of background and signal showers evaluated using
CaloFlow. In Sec. IV, we include the results of performing unsupervised
anomaly detection with CaloFlow. Finally, we conclude in Sec. V.

##  II Dataset

For this study, we decided to create a new, more realistic sampling
calorimeter version [52] of the CaloGAN dataset. The original dataset included
energy contributions from both active and inactive calorimeter layers, whereas
our new dataset only includes energy contributions from the active layers as
would be available in practice. The sampling fraction of our new calorimeter
setup is $\sim 20\%$. The simple detector is a three-layer, liquid argon (LAr)
sampling calorimeter cube with 480mm side-length that is inspired by the ATLAS
LAr electromagnetic calorimeter [63]. It is simulated as follows: Geant4 [64,
65, 66] is used to generate particles and simulate their interaction with our
calorimeter using the Ftfp_Bert physics list based on the Fritiof [67, 68, 69,
70] and Bertini intra-nuclear cascade [71, 72, 73] models with the standard
electromagnetic physics package [74]. While we use this new simulator to
create a dataset for anomaly detection, we expect it should be more generally
useful for a broad variety of tasks.

The calorimeter showers are represented as three-dimensional images that are
binned in position space. In this representation, the calorimeter shower
geometry is made up of voxels (volumetric pixels) and the details of the
calorimeter voxel dimensions are included in Table 1. Figure 1 (taken from
[21, 75]) shows the three-dimensional representation of a shower in the
CaloGAN calorimeter. The three longitudinal layers are separated in the figure
for visualization purposes. In this work, the center of the detector is taken
to be at $z=0$ m, while the front face of layer 0 is positioned at $z=1$ m
which is consistent with an ATLAS-like configuration.

| Layer
---
index
|  $z$ length
---
(mm)
|  $\eta$ length
---
(mm)
|  $\phi$ length
---
(mm)
| Number
---
of voxels
0 | 90 | 5 | 160 | $3\times 96$
1 | 347 | 40 | 40 | $12\times 12$
2 | 43 | 80 | 40 | $12\times 6$
Table 1: Dimensions of a calorimeter voxel. The positive $z-$axis (radial
direction in full detector) is the direction of particle propagation, the
$\eta$ direction is along the proton beam axis, and $\phi$ is perpendicular to
$z$ and $\eta$. For the number of voxels, the first (second) number is the
number of bins in the $\phi$ ($\eta$) direction (e.g., $12\times 6$ refers to
12 $\phi$ bins and 6 $\eta$ bins). Figure 1: Three-dimensional representation
of the shower in the CaloGAN calorimeter, figure taken from [21, 75]. Not-to-
scale separation among the longitudinal layers is added for visualization
purposes.

For the background dataset (single-photon calorimeter showers), we generate
100k showers with incident energies $E_{\text{inc}}$ uniformly distributed in
the range $[1,100]$ GeV. This dataset was used as the training dataset for
CaloFlow where we used a train/validation split of 70%/30%. A second
independent dataset of 100k showers with the same range of $E_{\text{inc}}$
was generated and used for evaluation.

To obtain the signal datasets, we defined the hypothetical $\chi$ particle in
Geant4 which has the same properties as the $\eta^{0}$ particle except having
a different mass and being invisible to the detector. Next, we generate
multiple sets of 100k showers that originate from $\chi\to\gamma\gamma$ decay
at 10 chosen fixed displacements from the center of detector along the
$z-$axis. The chosen fixed displacements are
$z_{i}\in\\{0.33,0.66,1.00,1.04,1.08,1.16,1.24,1.32,1.40,1.44\\}$ m. Note that
the first two displacements are located in front of the calorimeter, while the
last eight displacements are located at distinct positions within the
calorimeter. The energy of the $\chi$ particle was fixed at 50 GeV. Such a
hypothetical scenario might arise from the decay of a 100 GeV particle (close
to e.g. the Higgs boson mass), which is at rest, to a pair of $\chi$
particles.

To study how the mass $m_{\chi}$ affects the anomaly detection performance, we
generated signal datasets with different $m_{\chi}\in\\{5\times 10^{-3},\
5\times 10^{-2},\ 5\times 10^{-1},\ 5\\}$ GeV. For each choice of $m_{\chi}$,
we generate 100k showers at each of the 10 fixed displacements. The results
for these fixed displacement signal datasets are shown in Section IV.1.

To consider particles with fixed lifetimes, we construct new signal datasets
using the fixed displacement samples. In particular, we use the probability
that the $\chi$ particle decays at various positions along the $z-$axis to
determine the proportion of showers originating from decays at each of the 10
fixed displacements. Generating directly with fixed lifetimes would have been
too inefficient, since we are discarding decays after the calorimeter volume,
which can happen often for these lifetimes that we consider. More details of
the fixed lifetime signal datasets are included in Section IV.2.

We also generated 100k signal showers with kinetic energy222We distinguish
between kinetic energy and incident energy for massive particles. The incident
energy is the sum of the kinetic energy and rest mass energy of the particle.
This distinction only becomes noticeable in the case of $m_{\chi}=5$ GeV.
uniformly distributed in the range [1,100] GeV for a $\chi$ particle with
$m_{\chi}=5\times 10^{-3}$ GeV and lifetime $\tau=1.00$ ns. A second set of
100k signal showers was generated with the same range of kinetic energies, but
for a $\chi$ particle with $m_{\chi}=5$ GeV and lifetime $\tau=1.00$ ns. Each
of these two datasets (together with the background dataset) was used to train
a supervised classifier described in Section IV.3. Using a range of kinetic
energies ensured that we obtain showers with a range of $E_{\text{inc}}$ such
that the $E_{\text{inc}}$ would not be used as a discriminating factor by the
supervised classifier. This allows for a fairer comparison between performance
of the supervised classifier and our unsupervised method since CaloFlow learns
the likelihood conditioned on $E_{\text{inc}}$.

##  III CaloFlow

CaloFlow [39, 40] is an approach to fast calorimeter simulation based on
conditional normalizing flows. In the context of fast calorimeter simulation,
CaloFlow uses a two-flow method to learn to generate the voxel level shower
energies $\vec{\mathcal{I}}$ conditioned on the corresponding incident
energies of the showers $E_{\text{inc}}$ denoted by
$p(\vec{\mathcal{I}}|E_{\text{inc}})$. Flow-I is constructed to learn the
probability density of calorimeter layer energies333The layer energy of a
given calorimeter layer is the sum of all the voxel energies in that layer.
$E_{i}$ conditioned on the incident energy
$p_{1}(E_{0},E_{1},E_{2}|E_{\text{inc}})$, while Flow-II is designed to learn
the probability density of the normalized voxel level shower energies
conditioned on the calorimeter layer energies
$p_{2}\left(\hat{\mathcal{I}}|E_{0},E_{1},E_{2},E_{\text{inc}}\right)$. By
normalized, we mean that the voxel energies in each layer are made to sum to
unity. The dimensions of the conditional inputs and outputs of the two flows
are shown in Table 2. Importantly, these flows were trained only using single
photon showers.

One important difference with original CaloFlow is that in this application to
anomaly detection we do not have direct empirical access to $E_{\text{inc}}$.
For a given shower, we do not know a priori what the corresponding
$E_{\text{inc}}$ is and would instead have to use a reconstructed estimate of
$E_{\text{inc}}$ which we denote as $E^{(\text{rec})}_{\text{inc}}$. In this
work, we use a simple regression method to reconstruct $E_{\text{inc}}$ given
the total deposited energy in the calorimeter $E_{\text{dep}}$. The
reconstructed incident energy is defined as
$E^{(\text{rec})}_{\text{inc}}=\lambda E_{\text{dep}}$, where $\lambda$ is the
mean of $E_{\text{inc}}/E_{\text{dep}}$ in the single photon training dataset.
Note that the true $E_{\text{inc}}$ is still used for training CaloFlow, while
$E^{(\text{rec})}_{\text{inc}}$ is used when performing anomaly detection.

The architecture and training of CaloFlow are outlined in Appendix A. Some
modifications were made to the original CaloFlow, while most of the main
algorithm remains unchanged.

| Conditionals |  | Dim of
---
conditional
Output |  | Dim of
---
output
Flow-I | $E_{\rm inc}$ | 1 | $E_{0},E_{1},E_{2}$ | 3
Flow-II | $E_{0},E_{1},E_{2},E_{\rm inc}$ | 4 | $\hat{\mathcal{I}}$ | 504
Table 2: The conditional inputs for each flow, and the features whose
probability distributions are the output of each flow.

###  III.1 Unsupervised anomaly detection with CaloFlow

After training CaloFlow on background single-photon showers, we evaluate the
log-likelihood $\log
p\left(\hat{\mathcal{I}},E_{i}|E^{(\text{rec})}_{\text{inc}}\right)$ of the
background and signal showers by using the trained flows. Using both Flow-I
and Flow-II, we are able to obtain the full log-likelihood:

$\displaystyle\log
p\left(\hat{\mathcal{I}},E_{i}|E^{(\text{rec})}_{\text{inc}}\right)=\underbrace{\log
p_{1}(E_{i}|E^{(\text{rec})}_{\text{inc}})}_{\text{Flow-I}}+\underbrace{\log
p_{2}\left(\hat{\mathcal{I}}|E_{i},E^{(\text{rec})}_{\text{inc}}\right)}_{\text{Flow-
II}}\,.$ (1)

Figure 2: Plot of log-likelihood comparison between signal showers from $\chi$
decays at the 10 fixed displacements (lines) and background photon showers
(filled). This example is for a $\chi$ particle with $m_{\chi}=5$ GeV.

Figure 2 shows an example of a plot of the full log-likelihood of the signal
showers from $\chi$ decays at the 10 fixed displacements and background photon
showers.

We see from Figure 2 that the signal showers generally have log-likelihoods
that are distinguishable from that of the background showers. Hence, we can
use CaloFlow as an unsupervised anomaly detector by placing cuts on the full
log-likelihood to discriminate signal from background showers. The results are
detailed in Section IV.

##  IV Results

###  IV.1 Decay at fixed displacement

In this section, we study the effect of varying the displacement from the
center of the detector at which the decay occurs on the evaluated likelihood.
In reality, particles do not decay at fixed positions, but instead the
probability of a particle decaying at a given displacement from where it was
created is related to its lifetime $\tau$. Nevertheless, studying the showers
produced by the particle $\chi$ at fixed decay positions is interesting from
an experimental perspective, and doing so also helps us interpret the more
physical results in Section IV.2.

Figure 3: Heatmap of maximum significance improvement
$\left(\text{tpr}/\sqrt{\text{fpr}}\right)$ for different masses of the $\chi$
particle $m_{\chi}$ and the 10 fixed displacements where the decay occurs.

Our performance metric for anomaly detection will be the significance
improvement which is defined as the signal efficiency (i.e. true positive rate
or tpr) divided by the square root of the background efficiency (i.e. false
positive rate or fpr). The maximum significance improvement corresponds to the
best possible444This is signal model-dependent, but still provides a useful
bound on the achievable performance. multiplicative factor by which the signal
significance can be improved with a cut on the anomaly score. Figure 3 shows a
heatmap of maximum significance improvement for each of the four different
particle masses $m_{\chi}$, and the 10 fixed displacements where the decay
occurs. In general, we find that showers from decays at larger $z$ are more
anomalous. However, there is clear exception in the case of $m_{\chi}=5$ GeV,
where the showers originating from decays that occur before the $\chi$
particle reaches the calorimeter (e.g., $z=$ 0.33 m, 0.66 m) are more
anomalous than those from decays occuring at the front face of the calorimeter
(i.e., $z=1.0$ m). This is due to the fact that the 5 GeV particle is less
boosted compared to the other lighter particles that we consider in this
study. As a result, the decay of the 5 GeV particle often results in a wider
angle between the produced pair of photons which CaloFlow is better able to
distinguish from the background single-photon showers. See Figure 4 for an
example of such a shower and comparison with a regular photon shower. On the
other hand, if a decay occurs right at the front face of the calorimeter,
there is insufficient time for the pair of photons to propagate and create two
distinct blobs of energy in the calorimeter.

Figure 4: Top row: Example of shower with two distinct blobs of energy
originating from a 5 GeV $\chi$ particle. Bottom row: Example of typical
photon shower with centralized energy core. The energy deposition in each of
the three layers are shown here with layer 0 on the left, layer 1 in the
center, and layer 2 on the right.

For the three lowest $m_{\chi}$, we find that CaloFlow struggles to detect
signal showers that originate from decays occurring before the front face of
the calorimeter (i.e., $z\leq 1.0$ m). These lower mass particles are highly
boosted which results in showers from highly collimated photon pairs that are
more similar to the background photon showers.

###  IV.2 Varying lifetime

To assess the performance of our unsupervised anomaly detector on realistic
scenarios, we construct new datasets consisting of showers from $\chi$
particles with fixed rest frame lifetimes. In particular, for each chosen mass
$m_{\chi}$ and lifetime $\tau$, we have a total of 100k events that are made
up of showers from $\chi$ decays at fixed displacements. The proportion of
showers associated to decays at each fixed displacement is determined based on
the lifetime $\tau$. Particles that decay after the calorimeter volume are not
included in the events since they are not detected within the calorimeter.

The probability for a particle to survive for time $t$ before decaying is
given by $p_{s}(t)=\exp(-\frac{t}{\gamma\tau})$, where the relativistic boost
factor $\gamma=E_{\text{particle}}/M_{\text{particle}}$. Equivalently, the
probability that a particle decays before reaching displacement $z$ is given
by $p_{d}(z)=1-\exp(-\frac{z}{c\tau\sqrt{\gamma^{2}-1}})$. Since we consider
10 fixed displacements indexed by $i\in\\{1,2,...,10\\}$, the fraction of the
total showers that originates from $\chi$ decay at displacement $z_{i}$ is set
to be $\hat{w}_{i}$ which is defined by

$w_{i}=\left\\{\begin{array}[]{ll}p_{d}(z_{1}),\quad i=1\\\
p_{d}(z_{i})-p_{d}(z_{i-1}),\quad\text{otherwise}\end{array}\right.$
$\text{and}\quad\hat{w}_{i}=\frac{w_{i}}{\sum_{j=1}^{10}w_{j}}$

In other words, the number of showers originating from the decay at $z=z_{i}$
is equal to $10^{5}\times\hat{w}_{i}$555In some cases, we had to round
$\hat{w}_{i}$ to ensure that the total number of showers is equal to 100k.. In
this work, we consider three possible lifetimes $\tau\in\\{0.01,0.1,1\\}$ ns
for each choice of $m_{\chi}$.

Figure 5: Plots of significance improvement of our unsupervised (unsup)
CaloFlow anomaly detector as a function of background rejection (1/false
positive rate) for various types of signal showers originating from $\chi$
particles with different $m_{\chi}$ and $\tau$. Top left: $m_{\chi}=5\times
10^{-3}$ GeV, top right: $m_{\chi}=0.05$ GeV, bottom left: $m_{\chi}=0.5$ GeV,
bottom right: $m_{\chi}=5$ GeV. The performance of a random classifier is
drawn with black dashed lines to serve as a baseline.

Figure 5 shows the significance improvement of our CaloFlow anomaly detector
as a function of the background rejection (1/false positive rate) for various
types of signal showers originating from $\chi$ particles with different
$m_{\chi}$ and $\tau$.

Overall, we find that CaloFlow is able to achieve significantly better
performance compared to a random classifier which we take to be the baseline.
One exception is for $(m_{\chi},\tau)=(0.5\ \text{GeV},0.01\ \text{ns})$ where
the performance closely matches that of the random baseline. The poorer
performance here is due to the lower boost factor $\gamma$ for the $0.5$ GeV
particle compared to the two lower particles masses which results in the
majority of the decays occurring close to the center of the detector ($z=0$
m). Meanwhile, the mass is still low enough that the photons are not widely
separated by the time they reach the calorimeter. As we have seen in Figure 3,
such showers are mostly not detected as anomalous by CaloFlow. At longer
lifetimes, the showers are more anomalous as a larger proportion of them
originate from decays occuring within the calorimeter.

For the three lowest $m_{\chi}$, the significance improvement generally
increases with background rejection. Also, we find that significance
improvement curves are similar across lifetimes for each of these three lowest
$m_{\chi}$. At fixed $m_{\chi}$, the large boost of these particles results in
a similar proportion of particles decaying at a given fixed displacement
before/within the calorimeter for different lifetimes $\tau$.

The best performance among all the (fixed mass and lifetime) signal models we
considered in this study was achieved in the cases with the largest mass of
$m_{\chi}=5$ GeV. As explained in Section IV.1, showers from early decay of
these more massive particles are quite anomalous according to CaloFlow due to
the wider angle between the produced pair of photons. In this case, going to
higher lifetimes actually makes these showers slightly less anomalous (which
is opposite to the trend seen at lower masses) since it gives the photons less
time to separate before interacting with the detector.

###  IV.3 Comparison with supervised anomaly detection

The previous section showed that the unsupervised anomaly detector has broad
coverage across the various model parameters. An important question to ask is
how this compares to a dedicated search for the $\chi\rightarrow\gamma\gamma$
signal. For a particular signal model, we would expect that model to
outperform the unsupervised approach. However, it is not possible to have a
dedicated search for every possible signal and so the key question to ask is
how well a supervised model trained on one signal would perform on other
signals not seen during training.

In this section, we compare the performance of our method against two
supervised classifiers. Each supervised classifier was trained on a combined
dataset with 100k signal showers and 100k background showers. The showers
originate from particles with kinetic energy uniformly distributed in the
range [1,100] GeV. The signal showers originate from $\chi$ particles with
lifetime $\tau=1.00$ ns. The first (second) supervised classifier was trained
on the a dataset with $m_{\chi}=5\times 10^{-3}\ (5)$ GeV. These models were
chosen because they are sufficiently different that they would likely be
covered by different dedicated searches. It is thus interesting to ask if a
search optimized for one of the models would still be sensitive to the other,
since the unsupervised approach has some sensitivity to both models.

The supervised classifier is a fully-connected neural network with two hidden
layer with 512 nodes each. We have a 505-dimensional input consisting of the
voxel energies normalized by the reconstructed incident energy
$\vec{\mathcal{I}}/E^{(\text{rec})}_{\text{inc}}$ (504-dim) and the
reconstructed incident energy $E^{(\text{rec})}_{\text{inc}}$ (1-dim). The
output layer returns a single number which is passed through a sigmoid
activation function. All other activation functions are ReLUs. The supervised
classifier was trained for a total of 50 epochs with a train/test/validation
split of 60%/20%/20%. These parameters were not extensively optimized, but we
found little gain from small variations in the setup.

Figure 6: Comparison plots of significance improvement of our CaloFlow anomaly
detector as a function of background rejection (1/false positive rate). The
plots for the performance for the $m_{\chi}=5\times 10^{-3}$ GeV and
$m_{\chi}=5$ GeV cases are shown on the left and right respectively. Top row:
supervised classifier was trained on signal showers originating from $\chi$
particles with $m_{\chi}=5\times 10^{-3}$ GeV and lifetime $\tau=1.00$ ns.
Bottom row: supervised classifier was trained on signal showers originating
from $\chi$ particles with $m_{\chi}=5$ GeV and lifetime $\tau=1.00$ ns. The
performance of our unsupervised CaloFlow anomaly detector (unsup) is shown as
solid lines, while that of the supervised classifier (sup) is shown as dotted
lines. The performance of a random classifier is drawn with black dashed lines
to serve as a baseline.

After training the supervised classifiers, they were evaluated on the signal
showers discussed in Section IV.2. Figure 6 shows the comparison plots between
the performance of our CaloFlow anomaly detector (unsup) and the supervised
classifiers (sup). The top (bottom) row shows the performance of the
supervised classifier that was trained on signal showers originating from
$\chi$ particles with $m_{\chi}=5\times 10^{-3}\ (5)$ GeV and lifetime
$\tau=1.00$ ns.

In the top left plot, we see that the supervised classifier clearly performs
better than CaloFlow on the signal showers originating from $\chi$ particles
that have same mass ($5\times 10^{-3}$ GeV) as what it was trained on,
achieving a larger significance improvement for various background rejection.
We found that the supervised classifier still performs well on signal showers
originating from $\chi$ particles with higher masses as long as the $\chi$
particles they originate from are still highly boosted.

However, as shown in the top right plot, the performance of the supervised
classifier decreases significantly for signal showers associated to
$m_{\chi}=5$ GeV that are less highly boosted. We find that CaloFlow achieves
a much higher significance improvement for large range of background rejection
in the cases of $\tau=0.01,0.1$ ns. For the case of $\tau=1$ ns, the
supervised classifier and CaloFlow appear to have comparable performance
across different background rejection.

In the bottom right plot, we again see that the supervised classifier
outperforms CaloFlow on the signal showers originating from $\chi$ particles
that have same mass (5 GeV) as what it was trained on. Not surprisingly, we
see that this supervised classifier achieves much higher significance
improvement compared to the other supervised classifier that was trained on
signal showers associated to $m_{\chi}=5\times 10^{-3}$ GeV (top right plot)
when evaluated on signal showers associated to $m_{\chi}=5$ GeV.

Similarly, this supervised classifier evaluated on signal showers associated
to $m_{\chi}=5\times 10^{-3}$ GeV (bottom left plot) achieves lower
significance improvement compared to the supervised classifier that was
trained on signal showers associated to $m_{\chi}=5\times 10^{-3}$ GeV (top
left plot). Despite the drop in performance, we see that the supervised
classifier still outperforms CaloFlow when evaluated on signal showers it was
not trained on. This is likely due to CaloFlow not being able to fully
discriminate against signal showers that only decay after the first
longitudinal layer (i.e., $E_{0}=0$ GeV), whereas such showers are usually
perfectly distinguished by the supervised classifier. As noise is added to the
voxel energies when using CaloFlow, this artificially causes $E_{0}>0$
GeV666We checked that a different treatment of the layers with zero energy
deposition does not improve the performance. and prevents such showers from
being perfectly distinguished by CaloFlow.

This comparison of our unsupervised anomaly detection method against a
supervised classifier highlights the potential limitation of model-specific
anomaly detection as the supervised model is unable to generalize its
excellent performance to signal that is too different from what it was trained
on. We note that it is possible to train a supervised classifier on all the
types of signal showers we have considered here. Doing so would likely result
in the supervised classifier outperforming CaloFlow when evaluated on all the
signal showers. However, the point is that even if one is to train a
supervised classifier on a large number of signal types, it is impossible to
exhaust the space of all possible signals. Hence, there may be an advantage in
using model-agnostic, unsupervised anomaly detection methods such as the one
we proposed in this work. This is especially true when using flow-based fast
calorimeter simulators as no additional training has to be performed to use
them as unsupervised anomaly detectors.

##  V Conclusion and Outlook

Using CaloFlow as an example, we demonstrated how fast calorimeter surrogate
models with access to the data likelihood can double up as unsupervised
anomaly detectors.

By studying the anomaly detection performance of CaloFlow on showers from
$\chi$ particle decays occurring at fixed displacements in the detector, we
found that CaloFlow is generally more sensitive to signal showers from decays
that occur deeper in the calorimeter. However, in the case of more massive,
less highly boosted $\chi$ particles, CaloFlow still has significant
discriminative power for showers from decays occurring in front of the
calorimeter.

By reweighting the proportion of showers originating from decays at each fixed
displacement, we constructed signal datasets corresponding to fixed particle
lifetimes. We found that CaloFlow has discriminative power for most of the
models we tested. In particular, CaloFlow achieves the best performance in the
case with $m_{\chi}=5$ GeV and $\tau=0.01$ ns where the particle is less
highly boosted and majority of the particles decay close to the center of the
detector.

Finally, we compared to performance of our unsupervised CaloFlow anomaly
detector against a supervised classifier. We found that a supervised
classifier trained on signal showers from highly boosted $\chi$ particles
performed significantly poorer on showers from more massive, less highly
boosted particles compared to our unsupervised method. On more boosted
signals, the supervised classifier is still highly effective. This highlights
the complementarity of different approaches and reaffirms the need to have a
diversity of methods in order to achieve broad sensitivity.

## Data and code availability

The datasets used in this study can be found at [52] and the software to
generate these datasets are located at https://github.com/Ian-
Pang/CaloGAN/tree/anomaly_detection. The machine learning software is at
https://github.com/Ian-Pang/AD_with_CF.

## Acknowledgements

CK would like to thank the Baden-Württemberg-Stiftung for financing through
the program Internationale Spitzenforschung, project Uncertainties – Teaching
AI its Limits (BWST_IF2020-010). IP and DS are supported by the U.S.
Department of Energy (DOE), Office of Science grant DOE-SC0010008 and BN is
supported by the DOE under contract DE-AC02-05CH11231.

##  Appendix A Architecture and training

Here we briefly describe the architecture and training procedure used for
CaloFlow (see [39, 40] for more details). There are some differences compared
to the implementation in the original CaloFlow papers [39, 40], but most of
main algorithm remains the same.

Both Flow-I and Flow-II are Masked Autoregressive Flows (MAFs) [76] with
compositions of Rational Quadratic Splines (RQS) [77] as transformations. The
RQS transformations are parameterized using neural networks known as MADE
blocks [78]. The details of the architecture of Flow-I and Flow-II are
summarized in Table 3.

| dim of | number of | layer sizes | number of | RQS
---|---|---|---|---|---
Model | base distribution | MADE blocks | input | hidden | output | RQS bins | tail bound
Flow-I | 3 | 6 | 64 | $2\times 64$ | 69 | 8 | 14
Flow-II | 504 | 8 | 378 | $1\times 378$ | 11592 | 8 | 14
Table 3: Summary of architecture of the Flow-I and Flow-II. For the hidden
layer sizes, the first number is the number of hidden layers in each MADE
block and the second number is the number of nodes in each hidden layer (e.g.,
$2\times 64$ refers to 2 hidden layers per MADE block with 64 nodes per hidden
layer).

The incident energy of the incoming photon is preprocessed as

$\displaystyle E_{\text{inc}}\to\log_{10}(E_{\text{inc}}/10\ \text{GeV})\,.$
(2)

The inputs to the flows are preprocessed as follows:

  * •

Flow-I: $E_{i}\to 2\left(\log_{10}(E_{i}+1\ \text{keV})-1\right)$

  * •

Flow-II:

$E_{i}\to\log_{10}(E_{i}+1\ \text{keV})-2$,

$\mathcal{I}_{ia}\to u_{\text{logit},ia}(\mathcal{I}_{ia}/E_{i})$,

where $u_{\text{logit},ia}=\log\frac{\tilde{u}_{ia}}{1-\tilde{u}_{ia}}$,

$\tilde{u}_{ia}=\alpha+(1-2\alpha)u_{ia}$ and $\alpha=10^{-6}$ .

The index $i$ denotes the layer number, while the index $a$ specifies the
voxel within the given layer. In the original CaloFlow, a different
preprocessing was used for the layer energies $E_{i}$ in Flow-I where $E_{i}$
were transformed to unit-space (see [39]).

As in Ref. [39, 40], uniform noise in the range [0,1] keV was applied to the
voxel energies during training and evaluation. The addition of noise was found
to prevent the flow from fitting unimportant features. The training of both
flows in this work is optimized using independent Adam optimizers [79]. Flow-I
was trained by minimizing $-\log p(E_{0},E_{1},E_{2}|E_{\text{inc}})$ for 75
epochs with a batch size of 200. Flow-II was trained by minimizing $-\log
p\left(\hat{\mathcal{I}}|E_{0},E_{1},E_{2},E_{\text{inc}}\right)$ for 100
epochs with a batch size of 200. The initial learning of $1\times 10^{-4}$ was
chosen for the two flows and a multi-step learning schedule was used when
training the flows which halves the learning rate after each selected epoch
milestone during the training.

## References

  * [1] ATLAS collaboration, _Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC_ , _Phys. Lett. B_ 716 (2012) 1 [1207.7214].
  * [2] CMS collaboration, _Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC_ , _Phys. Lett. B_ 716 (2012) 30 [1207.7235].
  * [3] ATLAS Collaboration, _Exotic Physics Searches_ , 2023.
  * [4] ATLAS Collaboration, _Supersymmetry searches_ , 2023.
  * [5] ATLAS Collaboration, _Higgs and Diboson Searches_ , 2023.
  * [6] CMS Collaboration, _CMS Exotica Public Physics Results_ , 2023.
  * [7] CMS Collaboration, _CMS Supersymmetry Physics Results_ , 2023.
  * [8] CMS Collaboration, _CMS Beyond-two-generations (B2G) Public Physics Results_ , 2023.
  * [9] LHCb Collaboration, _Publications of the QCD, Electroweak and Exotica Working Group_ , 2023.
  * [10] HEP ML Community, “A Living Review of Machine Learning for Particle Physics.”
  * [11] G. Kasieczka et al., _The LHC Olympics 2020: A Community Challenge for Anomaly Detection in High Energy Physics_ , _Rept.Prog.Phys._ 84 (2021) 124201 [2101.08320].
  * [12] T. Aarrestad et al., _The Dark Machines Anomaly Score Challenge: Benchmark Data and Model Independent Event Classification for the Large Hadron Collider_ , _SciPost Phys._ 12 (2021) 043 [2105.14027].
  * [13] G. Karagiorgi, G. Kasieczka, S. Kravitz, B. Nachman and D. Shih, _Machine Learning in the Search for New Fundamental Physics_ , 2112.03769.
  * [14] M. Feickert and B. Nachman, _A Living Review of Machine Learning for Particle Physics_ , 2102.02770.
  * [15] E.G. Tabak and C.V. Turner, _A family of nonparametric density estimation algorithms_ , _Communications on Pure and Applied Mathematics_ 66 (2013) 145.
  * [16] L. Dinh, D. Krueger and Y. Bengio, _Nice: Non-linear independent components estimation_ , _arXiv preprint arXiv:1410.8516_ (2014) .
  * [17] D.J. Rezende and S. Mohamed, _Variational inference with normalizing flows_ , 2015.  10.48550/ARXIV.1505.05770.
  * [18] L. Dinh, J. Sohl-Dickstein and S. Bengio, _Density estimation using real nvp_ , _arXiv preprint arXiv:1605.08803_ (2016) .
  * [19] G. Papamakarios, E. Nalisnick, D.J. Rezende, S. Mohamed and B. Lakshminarayanan, _Normalizing flows for probabilistic modeling and inference_ , .
  * [20] C. Winkler, D.E. Worrall, E. Hoogeboom and M. Welling, _Learning likelihoods with conditional normalizing flows_ , _CoRR_ abs/1912.00042 (2019) [1912.00042].
  * [21] M. Paganini, L. de Oliveira and B. Nachman, _Accelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multilayer Calorimeters_ , _Phys. Rev. Lett._ 120 (2018) 042003 [1705.02355].
  * [22] M. Paganini, L. de Oliveira and B. Nachman, _CaloGAN : Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks_ , _Phys. Rev._ D97 (2018) 014021 [1712.10321].
  * [23] L. de Oliveira, M. Paganini and B. Nachman, _Controlling Physical Attributes in GAN-Accelerated Simulation of Electromagnetic Calorimeters_ , _J. Phys. Conf. Ser._ 1085 (2018) 042017 [1711.08813].
  * [24] M. Erdmann, L. Geiger, J. Glombitza and D. Schmidt, _Generating and refining particle detector simulations using the Wasserstein distance in adversarial networks_ , _Comput. Softw. Big Sci._ 2 (2018) 4 [1802.03325].
  * [25] M. Erdmann, J. Glombitza and T. Quast, _Precise simulation of electromagnetic calorimeter showers using a Wasserstein Generative Adversarial Network_ , _Comput. Softw. Big Sci._ 3 (2019) 4 [1807.01954].
  * [26] ATLAS Collaboration, _Deep generative models for fast shower simulation in ATLAS_ , _ATL-SOFT-PUB-2018-001_ (2018) .
  * [27] D. Belayneh et al., _Calorimetry with Deep Learning: Particle Simulation and Reconstruction for Collider Physics_ , 1912.06794.
  * [28] S. Vallecorsa, F. Carminati and G. Khattak, _3D convolutional GAN for fast simulation_ , _Proceedings, 23rd International Conference on Computing in High Energy and Nuclear Physics (CHEP 2018): Sofia, Bulgaria, July 9-13, 2018; EPJ Web Conf._ 214 (2019) 02010.
  * [29] SHiP Collaboration, _Fast simulation of muons produced at the SHiP experiment using Generative Adversarial Networks_ , 1909.04451.
  * [30] V. Chekalina, E. Orlova, F. Ratnikov, D. Ulyanov, A. Ustyuzhanin and E. Zakharov, _Generative Models for Fast Calorimeter Simulation.LHCb case_ , _CHEP 2018_ (2018)  [1812.01319].
  * [31] F. Carminati, A. Gheata, G. Khattak, P. Mendez Lorenzo, S. Sharan and S. Vallecorsa, _Three dimensional Generative Adversarial Networks for fast simulation_ , _Proceedings, 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2017): Seattle, WA, USA, August 21-25, 2017; J. Phys. Conf. Ser._ 1085 (2018) 032016.
  * [32] S. Vallecorsa, _Generative models for fast simulation_ , _Proceedings, 18th International Workshop on Advanced Computing and Analysis Techniques in Physics Research (ACAT 2017): Seattle, WA, USA, August 21-25, 2017; J. Phys. Conf. Ser._ 1085 (2018) 022005.
  * [33] P. Musella and F. Pandolfi, _Fast and Accurate Simulation of Particle Detectors Using Generative Adversarial Networks_ , _Comput. Softw. Big Sci._ 2 (2018) 8 [1805.00850].
  * [34] K. Deja, T. Trzcinski and L. Graczykowski, _Generative models for fast cluster simulations in the TPC for the ALICE experiment_ , _Proceedings, 23rd International Conference on Computing in High Energy and Nuclear Physics (CHEP 2018): Sofia, Bulgaria, July 9-13, 2018; EPJ Web Conf._ 214 (2019) 06003.
  * [35] ATLAS Collaboration, _Deep generative models for fast photon shower simulation in ATLAS_ , 2210.06204.
  * [36] ATLAS collaboration, _AtlFast3: the next generation of fast simulation in ATLAS_ , _Comput. Softw. Big Sci._ 6 (2022) 7 [2109.02551].
  * [37] E. Buhmann, S. Diefenbacher, E. Eren, F. Gaede, G. Kasieczka, A. Korol et al., _Decoding Photons: Physics in the Latent Space of a BIB-AE Generative Network_ , _EPJ Web Conf._ 251 (2021) 03003 [2102.12491].
  * [38] E. Buhmann, S. Diefenbacher, E. Eren, F. Gaede, D. Hundhausen, G. Kasieczka et al., _Hadrons, Better, Faster, Stronger_ , _Mach.Learn.Sci.Tech._ 3 (2021) 025014 [2112.09709].
  * [39] C. Krause and D. Shih, _CaloFlow: Fast and Accurate Generation of Calorimeter Showers with Normalizing Flows_ , _Phys.Rev.D_ 107 (2021) 113003 [2106.05285].
  * [40] C. Krause and D. Shih, _CaloFlow II: Even Faster and Still Accurate Generation of Calorimeter Showers with Normalizing Flows_ , _Phys.Rev.D_ 107 (2021) 113004 [2110.11377].
  * [41] V. Mikuni and B. Nachman, _Score-based Generative Models for Calorimeter Shower Simulation_ , _Phys.Rev.D_ 106 (2022) 092009 [2206.11898].
  * [42] M.R. Buckley, C. Krause, I. Pang and D. Shih, _Inductive CaloFlow_ , 2305.11934.
  * [43] C. Krause, I. Pang and D. Shih, _CaloFlow for CaloChallenge Dataset 1_ , 2210.14245.
  * [44] S. Diefenbacher, E. Eren, F. Gaede, G. Kasieczka, C. Krause, I. Shekhzadeh et al., _L2LFlows: Generating High-Fidelity 3D Calorimeter Images_ , _JINST_ 18 (2023) P10017 [2302.11594].
  * [45] J.C. Cresswell, B.L. Ross, G. Loaiza-Ganem, H. Reyes-Gonzalez, M. Letizia and A.L. Caterini, _CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds_ , in _36th Conference on Neural Information Processing Systems_ , 11, 2022 [2211.15380].
  * [46] J. Liu, A. Ghosh, D. Smith, P. Baldi and D. Whiteson, _Generalizing to new geometries with Geometry-Aware Autoregressive Models (GAAMs) for fast calorimeter simulation_ , 2305.11531.
  * [47] S. Diefenbacher, E. Eren, F. Gaede, G. Kasieczka, A. Korol, K. Krüger et al., _New Angles on Fast Calorimeter Shower Simulation_ , _Mach.Learn.Sci.Tech._ 4 (2023) 035044 [2303.18150].
  * [48] E. Buhmann, S. Diefenbacher, E. Eren, F. Gaede, G. Kasieczka, A. Korol et al., _CaloClouds: Fast Geometry-Independent Highly-Granular Calorimeter Simulation_ , _JINST_ 18 (2023) P11025 [2305.04847].
  * [49] V. Mikuni and B. Nachman, _CaloScore v2: Single-shot Calorimeter Shower Simulation with Diffusion Models_ , 2308.03847.
  * [50] F.T. Acosta, V. Mikuni, B. Nachman, M. Arratia, K. Barish, B. Karki et al., _Comparison of Point Cloud and Image-based Models for Calorimeter Fast Simulation_ , 2307.04780.
  * [51] I. Pang, J.A. Raine and D. Shih, _SuperCalo: Calorimeter shower super-resolution_ , 2308.11700.
  * [52] C. Krause, B. Nachman, I. Pang, D. Shih and Y. Zhu, _Electromagnetic sampling calorimeter shower images_ , Dec., 2023.  10.5281/zenodo.10393540.
  * [53] B. Nachman, L. de Oliveira and M. Paganini, _Electromagnetic calorimeter shower images_ , _Mendeley Data_ (2017) .
  * [54] B. Nachman and D. Shih, _Anomaly Detection with Density Estimation_ , _Phys. Rev. D_ 101 (2020) 075042 [2001.04990].
  * [55] “Fast calorimeter simulation challenge 2022.”
  * [56] V. Mikuni and B. Nachman, _High-dimensional and Permutation Invariant Anomaly Detection_ , 2306.03933.
  * [57] J. Hajer, Y.-Y. Li, T. Liu and H. Wang, _Novelty Detection Meets Collider Physics_ , 1807.10261.
  * [58] M. Farina, Y. Nakai and D. Shih, _Searching for New Physics with Deep Autoencoders_ , 1808.08992.
  * [59] T. Heimel, G. Kasieczka, T. Plehn and J.M. Thompson, _QCD or What?_ , _SciPost Phys._ 6 (2019) 030 [1808.08979].
  * [60] P. Kirichenko, P. Izmailov and A.G. Wilson, _Why Normalizing Flows Fail to Detect Out-of-Distribution Data_ , _arXiv e-prints_ (2020) arXiv:2006.08545 [2006.08545].
  * [61] C. Le Lan and L. Dinh, _Perfect density models cannot guarantee anomaly detection_ , _Entropy_ 23 (2021) 1690.
  * [62] G. Kasieczka, R. Mastandrea, V. Mikuni, B. Nachman, M. Pettee and D. Shih, _Anomaly Detection under Coordinate Transformations_ , _Phys.Rev.D_ 107 (2022) 015009 [2209.06225].
  * [63] ATLAS collaboration, _ATLAS liquid-argon calorimeter: Technical Design Report_ , Technical design report. ATLAS, CERN, Geneva (1996), 10.17181/CERN.FWRW.FOOQ.
  * [64] GEANT4 collaboration, _GEANT4–a simulation toolkit_ , _Nucl. Instrum. Meth. A_ 506 (2003) 250.
  * [65] J. Allison, K. Amako, J. Apostolakis, H. Araujo, P. Arce Dubois, M. Asai et al., _Geant4 developments and applications_ , _IEEE Transactions on Nuclear Science_ 53 (2006) 270.
  * [66] J. Allison, K. Amako, J. Apostolakis, P. Arce, M. Asai, T. Aso et al., _Recent developments in geant4_ , _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ 835 (2016) 186.
  * [67] B. Ganhuyag and V. Uzhinsky, _Modified FRITIOF code: Negative charged particle production in high energy nucleus nucleus interactions_ , _Czech. J. Phys._ 47 (1997) 913.
  * [68] B. Nilsson-Almqvist and E. Stenlund, _Interactions Between Hadrons and Nuclei: The Lund Monte Carlo, Fritiof Version 1.6_ , _Comput. Phys. Commun._ 43 (1987) 387.
  * [69] B. Andersson, G. Gustafson and B. Nilsson-Almqvist, _A Model for Low p(t) Hadronic Reactions, with Generalizations to Hadron - Nucleus and Nucleus-Nucleus Collisions_ , _Nucl. Phys. B_ 281 (1987) 289.
  * [70] B. Andersson, A. Tai and B.-H. Sa, _Final state interactions in the (nuclear) FRITIOF string interaction scenario_ , _Z. Phys. C_ 70 (1996) 499.
  * [71] M.P. Guthrie, R.G. Alsmiller and H.W. Bertini, _Calculation of the capture of negative pions in light elements and comparison with experiments pertaining to cancer radiotherapy_ , _Nucl. Instrum. Meth._ 66 (1968) 29.
  * [72] H.W. Bertini and M.P. Guthrie, _News item results from medium-energy intranuclear-cascade calculation_ , _Nucl. Phys. A_ 169 (1971) 670.
  * [73] V.A. Karmanov, _LIGHT FRONT WAVE FUNCTION OF RELATIVISTIC COMPOSITE SYSTEM IN EXPLICITLY SOLVABLE MODEL_ , _Nucl. Phys. B_ 166 (1980) 378.
  * [74] H. Burkhardt, V. Grichine, P. Gumplinger, V. Ivanchenko, R. Kokoulin, M. Maire et al., _Geant4 standard electromagnetic package for hep applications_ , in _IEEE Symposium Conference Record Nuclear Science 2004._ , vol. 3, pp. 1907–1910 Vol. 3, 2004, DOI.
  * [75] M. Paganini, L. de Oliveira and B. Nachman, _CaloGAN: Simulating 3d high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks_ , _Physical Review D_ 97 (2018) .
  * [76] G. Papamakarios, T. Pavlakou and I. Murray, _Masked autoregressive flow for density estimation_ , _Advances in neural information processing systems_ 30 (2017) [1705.07057].
  * [77] C. Durkan, A. Bekasov, I. Murray and G. Papamakarios, _Neural spline flows_ , _Advances in Neural Information Processing Systems_ 32 (2019) 7511 [1906.04032].
  * [78] M. Germain, K. Gregor, I. Murray and H. Larochelle, _Made: Masked autoencoder for distribution estimation_ , in _International conference on machine learning_ , pp. 881–889, PMLR, 2015.
  * [79] D.P. Kingma and J. Ba, _Adam: A method for stochastic optimization_ , 2014.