abstractWe introduce Bootstrap Your Own Latent (BYOL), a new approach to self-supervised image representation learning. BYOL relies on two neural networks, referred to as online and target networks, that interact and learn from each other. From an augmented view of an image, we train the online network to predict the target network representation of the same image under a different augmented view. At the same time, we update the target network with a slow-moving average of the online network. While state-of-the art methods rely on negative pairs, BYOL achieves a new state of the art without them. BYOL reaches $74.3\%$ top-1 classification accuracy on ImageNet using a linear evaluation with a ResNet-50 architecture and $79.6\%$ with a larger ResNet. We show that BYOL performs on par or better than the current state of the art on both transfer and semi-supervised benchmarks. Our implementation and pretrained models are given on GitHub.
shorttitleBootstrap your own latent
titleBootstrap your own latent: A new approach to self-supervised Learning
Computer Science - Computer Vision and Pattern Recognition,Computer Science - Machine Learning,Statistics - Machine Learning

abstractSiamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.
titleExploring Simple Siamese Representation Learning
Computer Science - Computer Vision and Pattern Recognition,Computer Science - Machine Learning

abstractIn this paper, we question if self-supervised learning provides new properties to Vision Transformer (ViT) that stand out compared to convolutional networks (convnets). Beyond the fact that adapting self-supervised methods to this architecture works particularly well, we make the following observations: first, self-supervised ViT features contain explicit information about the semantic segmentation of an image, which does not emerge as clearly with supervised ViTs, nor with convnets. Second, these features are also excellent k-NN classifiers, reaching 78.3% top-1 on ImageNet with a small ViT. Our study also underlines the importance of momentum encoder, multi-crop training, and the use of small patches with ViTs. We implement our findings into a simple self-supervised method, called DINO, which we interpret as a form of self-distillation with no labels. We show the synergy between DINO and ViTs by achieving 80.1% top-1 on ImageNet in linear evaluation with ViT-Base.
titleEmerging Properties in Self-Supervised Vision Transformers
94 [cs]
abstractSummary (1) BYOL often performs no better than random when batch normalization is removed, and (2) the presence of batch normalization…
titleUnderstanding self-supervised and contrastive learning with "Bootstrap Your Own Latent" (BYOL)

abstractA number of recent self-supervised learning methods have shown impressive performance on image classification and other tasks. A somewhat bewildering variety of techniques have been used, not always with a clear understanding of the reasons for their benefits, especially when used in combination. Here we treat the embeddings of images as point particles and consider model optimization as a dynamic process on this system of particles. Our dynamic model combines an attractive force for similar images, a locally dispersive force to avoid local collapse, and a global dispersive force to achieve a globally-homogeneous distribution of particles. The dynamic perspective highlights the advantage of using a delayed-parameter image embedding (a la BYOL) together with multiple views of the same image. It also uses a purely-dynamic local dispersive force (Brownian motion) that shows improved performance over other methods and does not require knowledge of other particle coordinates. The method is called MSBReg which stands for (i) a Multiview centroid loss, which applies an attractive force to pull different image view embeddings toward their centroid, (ii) a Singular value loss, which pushes the particle system toward spatially homogeneous density, (iii) a Brownian diffusive loss. We evaluate downstream classification performance of MSBReg on ImageNet as well as transfer learning tasks including fine-grained classification, multi-class object classification, object detection, and instance segmentation. In addition, we also show that applying our regularization term to other methods further improves their performance and stabilize the training by preventing a mode collapse.
titleAn Embedding-Dynamic Approach to Self-supervised Learning
Computer Science - Computer Vision and Pattern Recognition,Computer Science - Machine Learning

abstractTo avoid collapse in self-supervised learning (SSL), a contrastive loss is widely used but often requires a large number of negative samples. Without negative samples yet achieving competitive performance, a recent work has attracted significant attention for providing a minimalist simple Siamese (SimSiam) method to avoid collapse. However, the reason for how it avoids collapse without negative samples remains not fully clear and our investigation starts by revisiting the explanatory claims in the original SimSiam. After refuting their claims, we introduce vector decomposition for analyzing the collapse based on the gradient analysis of the $l_2$-normalized representation vector. This yields a unified perspective on how negative samples and SimSiam alleviate collapse. Such a unified perspective comes timely for understanding the recent progress in SSL.
shorttitleHow Does SimSiam Avoid Collapse Without Negative Samples?
titleHow Does SimSiam Avoid Collapse Without Negative Samples? A Unified Understanding with Self-supervised Contrastive Learning
Computer Science - Artificial Intelligence,Computer Science - Machine Learning

abstractWhen a vision system creates an interpretation of some input datn, it assigns truth values or probabilities to intcrnal hypothcses about the world. We present a non-dctcrministic method for assigning truth values that avoids many of the problcms encountered by existing relaxation methods. Instead of rcprcscnting probabilitics with realnumbers, we usc a more dircct encoding in which thc probability \ associated with a hypotlmis is rcprcscntcd by the probability hat it is in one of two states, true or false. Wc give a particular nondeterministic operator, based on statistical mechanics, for updating the truth values of hypothcses. The operator ensures that the probability of discovering a particular combination of hypothcscs is a simplc function of how good that combination is. Wc show that thcrc is a simple relationship bctween this operator and Bayesian inference, and we describe a learning rule which allows a parallel system to converge on a set ofweights that optimizes its perccptt~al inferences.
titleOPTIMAL PERCEPTUAL INFERENCE

abstractIt is possible to combine multiple latent-variable models of the same data by multiplying their probability distributions together and then renormalizing. This way of combining individual "expert" models makes it hard to generate samples from the combined model but easy to infer the values of the latent variables of each expert, because the combination rule ensures that the latent variables of different experts are conditionally independent when given the data. A product of experts (PoE) is therefore an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary. Training a PoE by maximizing the likelihood of the data is difficult because it is hard even to approximate the derivatives of the renormalization term in the combination rule. Fortunately, a PoE can be trained using a different objective function called "contrastive divergence" whose derivatives with regard to the parameters can be approximated accurately and efficiently. Examples are presented of contrastive divergence learning using several types of expert on several types of data.
journaltitleNeural Computation
shortjournalNeural Comput
titleTraining products of experts by minimizing contrastive divergence

abstractWe show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
journaltitleNeural Computation
shortjournalNeural Comput
titleA fast learning algorithm for deep belief nets
Algorithms,Animals,Humans,Learning,Neural Networks,Computer,Neurons

abstractWe provide a bridge between generative modeling in the Machine Learning community and simulated physical processes in High Energy Particle Physics by applying a novel Generative Adversarial Network (GAN) architecture to the production of jet images – 2D representations of energy depositions from particles interacting with a calorimeter. We propose a simple architecture, the Location-Aware Generative Adversarial Network, that learns to produce realistic radiation patterns from simulated high energy particle collisions. The pixel intensities of GAN-generated images faithfully span over many orders of magnitude and exhibit the desired low-dimensional physical properties (i.e., jet mass, n-subjettiness, etc.). We shed light on limitations, and provide a novel empirical validation of image quality and validity of GAN-produced simulations of the natural world. This work provides a base for further explorations of GANs for use in faster simulation in High Energy Particle Physics.
issn2510-2036, 2510-2044
journaltitleComputing and Software for Big Science
shortjournalComput Softw Big Sci
shorttitleLearning Particle Physics by Example
titleLearning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis
.05927 [hep-ex, physics:physics, stat]
Energy Physics - Experiment,Physics - Data Analysis,Statistics and Probability,Statistics - Machine Learning

abstractA neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by "learning without a teacher", and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname "neocognitron". After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consists of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of "S-cells', which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of "C-cells" similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any "teacher" during the process of selforganization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cells of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern.
issn0340-1200, 1432-0770
journaltitleBiological Cybernetics
shortjournalBiol. Cybernetics
titleNeocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

abstractIn many ways, graphs are the main modality of data we receive from nature. This is due to the fact that most of the patterns we see, both in natural and artificial systems, are elegantly representable using the language of graph structures. Prominent examples include molecules (represented as graphs of atoms and bonds), social networks and transportation networks. This potential has already been seen by key scientific and industrial groups, with already-impacted application areas including traffic forecasting, drug discovery, social network analysis and recommender systems. Further, some of the most successful domains of application for machine learning in previous years – images, text and speech processing – can be seen as special cases of graph representation learning, and consequently there has been significant exchange of information between these areas. The main aim of this short survey is to enable the reader to assimilate the key concepts in the area, and position graph representation learning in a proper context with related fields.
journaltitleCurrent Opinion in Structural Biology
shortjournalCurrent Opinion in Structural Biology
shorttitleEverything is Connected
titleEverything is Connected: Graph Neural Networks
10 [cs, stat]
ine Learning,Computer Science - Social and Information Networks,Statistics - Machine Learning

abstractWe study the problem of designing models for machine learning tasks defined on \emph{sets}. In contrast to traditional approach of operating on fixed dimensional vectors, we consider objective functions defined on sets that are invariant to permutations. Such problems are widespread, ranging from estimation of population statistics \cite{poczos13aistats}, to anomaly detection in piezometer data of embankment dams \cite{Jung15Exploration}, to cosmology \cite{Ntampaka16Dynamical,Ravanbakhsh16ICML1}. Our main theorem characterizes the permutation invariant functions and provides a family of functions to which any permutation invariant objective function must belong. This family of functions has a special structure which enables us to design a deep network architecture that can operate on sets and which can be deployed on a variety of scenarios including both unsupervised and supervised learning tasks. We also derive the necessary and sufficient conditions for permutation equivariance in deep models. We demonstrate the applicability of our method on population statistic estimation, point cloud classification, set expansion, and outlier detection.
titleDeep Sets
14 [cs, stat]
ine Learning,Statistics - Machine Learning

abstractA sampling technique is defined as introducing control into the selection of n out of N sampling units when it increases the probabilities of selection for preferred combinations of units (and decreases the probabilities for non-preferred combinations). Methods used in the past have by no means exhausted the possibilities of controlled selection, however. Procedures are developed by which the probabilities of selection for preferred combinations are sharply increased and the theoretical basis for the methods is stated. The methods are applied to a specific problem and the procedures are described in detail. It it found that as a result the variances of estimates for several important items are reduced as compared with the corresponding variances for stratified random sampling.
journaltitleJournal of the American Statistical Association
notePublisher: [American Statistical Association, Taylor & Francis, Ltd.]
titleControlled Selection–A Technique in Probability Sampling

abstractTraining generative adversarial networks requires balancing of delicate adversarial dynamics. Even with careful tuning, training may diverge or end up in a bad equilibrium with dropped modes. In this work, we improve CS-GAN with natural gradient-based latent optimisation and show that it improves adversarial dynamics by enhancing interactions between the discriminator and the generator. Our experiments demonstrate that latent optimisation can significantly improve GAN training, obtaining state-of-the-art performance for the ImageNet ($128 \times 128$) dataset. Our model achieves an Inception Score (IS) of $148$ and an Fr\'echet Inception Distance (FID) of $3.4$, an improvement of $17\%$ and $32\%$ in IS and FID respectively, compared with the baseline BigGAN-deep model with the same architecture and number of parameters.
titleLOGAN: Latent Optimisation for Generative Adversarial Networks
2.00953 [cs, stat]
ine Learning,Statistics - Machine Learning

abstractThe need for large scale and high fidelity simulated samples for the ATLAS experiment motivates the development of new simulation techniques. Building on the recent success of deep learning algorithms at interpolation as well as image generation, Variational Auto-Encoders and Generative Adversarial Networks are investigated for modeling the response of the electromagnetic calorimeter for photons in a central calorimeter region over a range of energies. The synthesized showers are compared to showers from a full detector simulation using Geant4. This study demonstrates the potential of using such algorithms for fast calorimeter simulation for the ATLAS experiment in the future.
journaltitleJournal of Physics: Conference Series
notePublisher: IOP Publishing
shortjournalJ. Phys.: Conf. Ser.
titleDeep generative models for fast shower simulation in ATLAS

abstractGiven the increasing data collection capabilities and limited computing resources of future collider experiments, interest in using generative neural networks for the fast simulation of collider events is growing. In our previous study, the Bounded Information Bottleneck Autoencoder (BIB-AE) architecture for generating photon showers in a high-granularity calorimeter showed a high accuracy modeling of various global differential shower distributions. In this work, we investigate how the BIB-AE encodes this physics information in its latent space. Our understanding of this encoding allows us to propose methods to optimize the generation performance further, for example, by altering latent space sampling or by suggesting specific changes to hyperparameters. In particular, we improve the modeling of the shower shape along the particle incident axis.
journaltitleEPJ Web of Conferences
shortjournalEPJ Web Conf.
shorttitleDecoding Photons
titleDecoding Photons: Physics in the Latent Space of a BIB-AE Generative Network
.12491 [hep-ex, physics:hep-ph, physics:physics]
Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors

abstractAccurate simulation of physical processes is crucial for the success of modern particle physics. However, simulating the development and interaction of particle showers with calorimeter detectors is a time consuming process and drives the computing needs of large experiments at the LHC and future colliders. Recently, generative machine learning models based on deep neural networks have shown promise in speeding up this task by several orders of magnitude. We investigate the use of a new architecture – the Bounded Information Bottleneck Autoencoder – for modelling electromagnetic showers in the central region of the Silicon-Tungsten calorimeter of the proposed International Large Detector. Combined with a novel second post-processing network, this approach achieves an accurate simulation of differential distributions including for the first time the shape of the minimum-ionizing-particle peak compared to a full GEANT4 simulation for a high-granularity calorimeter with 27k simulated channels. The results are validated by comparing to established architectures. Our results further strengthen the case of using generative networks for fast simulation and demonstrate that physically relevant differential distributions can be described with high accuracy.
issn2510-2036, 2510-2044
journaltitleComputing and Software for Big Science
shortjournalComput Softw Big Sci
shorttitleGetting High
titleGetting High: High Fidelity Simulation of High Granularity Calorimeters with High Speed
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors

abstractInformation bottleneck (IB) principle [1] has become an important element in information-theoretic analysis of deep models. Many state-of-the-art generative models of both Variational Autoencoder (VAE) [2; 3] and Generative Adversarial Networks (GAN) [4] families use various bounds on mutual information terms to introduce certain regularization constraints [5; 6; 7; 8; 9; 10]. Accordingly, the main difference between these models consists in add regularization constraints and targeted objectives. In this work, we will consider the IB framework for three classes of models that include supervised, unsupervised and adversarial generative models. We will apply a variational decomposition leading a common structure and allowing easily establish connections between these models and analyze underlying assumptions. Based on these results, we focus our analysis on unsupervised setup and reconsider the VAE family. In particular, we present a new interpretation of VAE family based on the IB framework using a direct decomposition of mutual information terms and show some interesting connections to existing methods such as VAE [2; 3], beta-VAE [11], AAE [12], InfoVAE [5] and VAE/GAN [13]. Instead of adding regularization constraints to an evidence lower bound (ELBO) [2; 3], which itself is a lower bound, we show that many known methods can be considered as a product of variational decomposition of mutual information terms in the IB framework. The proposed decomposition might also contribute to the interpretability of generative models of both VAE and GAN families and create a new insights to a generative compression [14; 15; 16; 17]. It can also be of interest for the analysis of novelty detection based on one-class classifiers [18] with the IB based discriminators.
titleInformation bottleneck through variational glasses
Computer Science - Computer Vision and Pattern Recognition

abstractMotivated by the computational limitations of simulating interactions of particles in highly-granular detectors, there exists a concerted effort to build fast and exact machine-learning-based shower simulators. This work reports progress on two important fronts. First, the previously investigated WGAN and BIB-AE generative models are improved and successful learning of hadronic showers initiated by charged pions in a segment of the hadronic calorimeter of the International Large Detector (ILD) is demonstrated for the first time. Second, we consider how state-of-the-art reconstruction software applied to generated shower energies affects the obtainable energy response and resolution. While many challenges remain, these results constitute an important milestone in using generative models in a realistic setting.
titleHadrons, Better, Faster, Stronger
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors

abstractWe present a study for the generation of events from a physical process with deep generative models. The simulation of physical processes requires not only the production of physical events, but also to ensure these events occur with the correct frequencies. We investigate the feasibility of learning the event generation and the frequency of occurrence with Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) to produce events like Monte Carlo generators. We study three processes: a simple two-body decay, the processes $eˆ+eˆ-\to Z \to lˆ+lˆ-$ and $p p \to t\bar{t} $ including the decay of the top quarks and a simulation of the detector response. We find that the tested GAN architectures and the standard VAE are not able to learn the distributions precisely. By buffering density information of encoded Monte Carlo events given the encoder of a VAE we are able to construct a prior for the sampling of new events from the decoder that yields distributions that are in very good agreement with real Monte Carlo events and are generated several orders of magnitude faster. Applications of this work include generic density estimation and sampling, targeted event generation via a principal component analysis of encoded ground truth data, anomaly detection and more efficient importance sampling, e.g. for the phase space integration of matrix elements in quantum field theories.
titleEvent Generation and Statistical Sampling for Physics with Deep Generative Models and a Density Information Buffer
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability

journaltitleThe Annals of Mathematical Statistics
notePublisher: Institute of Mathematical Statistics
titleOn Estimation of a Probability Density Function and Mode

abstractIn this work, we propose a novel end-to-end sinkhorn autoencoder with noise generator for efficient data collection simulation. Simulating processes that aim at collecting experimental data is crucial for multiple real-life applications, including nuclear medicine, astronomy and high energy physics. Contemporary methods, such as Monte Carlo algorithms, provide high-fidelity results at a price of high computational cost. Multiple attempts are taken to reduce this burden, e.g. using generative approaches based on Generative Adversarial Networks or Variational Autoencoders. Although such methods are much faster, they are often unstable in training and do not allow sampling from an entire data distribution. To address these shortcomings, we introduce a novel method dubbed end-to-end Sinkhorn Autoencoder, that leverages sinkhorn algorithm to explicitly align distribution of encoded real data examples and generated noise. More precisely, we extend autoencoder architecture by adding a deterministic neural network trained to map noise from a known distribution onto autoencoder latent space representing data distribution. We optimise the entire model jointly. Our method outperforms competing approaches on a challenging dataset of simulation data from Zero Degree Calorimeters of ALICE experiment in LHC. as well as standard benchmarks, such as MNIST and CelebA.
titleEnd-to-end Sinkhorn Autoencoder with Noise Generator
Computer Science - Machine Learning,Statistics - Machine Learning

abstractOptimal transport offers an alternative to maximum likelihood for learning generative autoencoding models. We show that minimizing the p-Wasserstein distance between the generator and the true data distribution is equivalent to the unconstrained min-min optimization of the p-Wasserstein distance between the encoder aggregated posterior and the prior in latent space, plus a reconstruction error. We also identify the role of its trade-off hyperparameter as the capacity of the generator: its Lipschitz constant. Moreover, we prove that optimizing the encoder over any class of universal approximators, such as deterministic neural networks, is enough to come arbitrarily close to the optimum. We therefore advertise this framework, which holds for any metric space and prior, as a sweet-spot of current generative autoencoding objectives. We then introduce the Sinkhorn auto-encoder (SAE), which approximates and minimizes the p-Wasserstein distance in latent space via backprogation through the Sinkhorn algorithm. SAE directly works on samples, i.e. it models the aggregated posterior as an implicit distribution, with no need for a reparameterization trick for gradients estimations. SAE is thus able to work with different metric spaces and priors with minimal adaptations. We demonstrate the flexibility of SAE on latent spaces with different geometries and priors and compare with other methods on benchmark data sets.
titleSinkhorn AutoEncoders
Computer Science - Computer Vision and Pattern Recognition,Computer Science - Machine Learning,Statistics - Machine Learning

abstractThis paper proposes a new and efficient technique to regularize the neural network in the context of deep learning using correlations among features. Previous studies have shown that oversized deep neural network models tend to produce a lot of redundant features that are either the shifted version of one another or are very similar and show little or no variations, thus resulting in redundant filtering. We propose a way to address this problem and show that such redundancy can be avoided using regularization and adaptive feature dropout mechanism. We show that regularizing both negative and positive correlated features according to their differentiation and based on their relative cosine distances yields network extracting dissimilar features with less overfitting and better generalization. This concept is illustrated with deep multilayer perceptron, convolutional neural network, sparse autoencoder, gated recurrent unit, and long short-term memory on MNIST digits recognition, CIFAR-10, ImageNet, and Stanford Natural Language Inference data sets.
journaltitleIEEE Transactions on Neural Networks and Learning Systems
noteConference Name: IEEE Transactions on Neural Networks and Learning Systems
titleRegularizing Deep Neural Networks by Enhancing Diversity in Feature Extraction
Correlation,Cosine similarity,Decorrelation,Deep learning,Feature extraction,Neural networks,Redundancy,Training,deep learning,feature clustering,feature correlation,redundancy elimination,regularization

abstractImaging Cherenkov detectors are largely used for particle identification (PID) in nuclear and particle physics experiments, where developing fast reconstruction algorithms is becoming of paramount importance to allow for near real time calibration and data quality control, as well as to speed up offline analysis of large amount of data. In this paper we present DeepRICH, a novel deep learning algorithm for fast reconstruction which can be applied to different imaging Cherenkov detectors. The core of our architecture is a generative model which leverages on a custom Variational Auto-encoder (VAE) combined to Maximum Mean Discrepancy (MMD), with a Convolutional Neural Network (CNN) extracting features from the space of the latent variables for classification. A thorough comparison with the simulation/reconstruction package FastDIRC is discussed in the text. DeepRICH has the advantage to bypass low-level details needed to build a likelihood, allowing for a sensitive improvement in computation time at potentially the same reconstruction performance of other established reconstruction algorithms. In the conclusions, we address the implications and potentialities of this work, discussing possible future extensions and generalization.
journaltitleMachine Learning: Science and Technology
shortjournalMach. Learn.: Sci. Technol.
titleDeepRICH: Learning Deeply Cherenkov Detectors
1.11717 [hep-ex, physics:nucl-ex, physics:physics]
ine Learning,High Energy Physics - Experiment,Nuclear Experiment,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors

Curran Associates, Inc.

abstractSupervised deep learning has been successfully applied for many recognition problems in machine learning and computer vision. Although it can approximate a complex many-to-one function very well when large number of training data is provided, the lack of probabilistic inference of the current supervised deep learning methods makes it difficult to model a complex structured output representations. In this work, we develop a scalable deep conditional generative model for structured output variables using Gaussian latent variables. The model is trained efficiently in the framework of stochastic gradient variational Bayes, and allows a fast prediction using stochastic feed-forward inference. In addition, we provide novel strategies to build a robust structured prediction algorithms, such as recurrent prediction network architecture, input noise-injection and multi-scale prediction training methods. In experiments, we demonstrate the effectiveness of our proposed algorithm in comparison to the deterministic deep neural network counterparts in generating diverse but realistic output representations using stochastic inference. Furthermore, the proposed schemes in training methods and architecture design were complimentary, which leads to achieve strong pixel-level object segmentation and semantic labeling performance on Caltech-UCSD Birds 200 and the subset of Labeled Faces in the Wild dataset.
booktitleAdvances in Neural Information Processing Systems
titleLearning Structured Output Representation using Deep Conditional Generative Models

abstractA key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degrades the inference quality. In addition, it has been observed that variational autoencoders tend to ignore the latent variables when combined with a decoding distribution that is too flexible. We again identify the cause in existing training criteria and propose a new class of objectives (InfoVAE) that mitigate these problems. We show that our model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution. Through extensive qualitative and quantitative analyses, we demonstrate that our models outperform competing approaches on multiple performance metrics.
titleInfoVAE: Information Maximizing Variational Autoencoders
Computer Science - Artificial Intelligence,Computer Science - Machine Learning,Statistics - Machine Learning

Curran Associates, Inc.

abstractThe use of machine learning algorithms frequently involves careful tuning of learning parameters and model hyperparameters. Unfortunately, this tuning is often a “black art” requiring expert experience, rules of thumb, or sometimes brute-force search. There is therefore great appeal for automatic approaches that can optimize the performance of any given learning algorithm to the problem at hand. In this work, we consider this problem through the framework of Bayesian optimization, in which a learning algorithm’s generalization performance is modeled as a sample from a Gaussian process (GP). We show that certain choices for the nature of the GP, such as the type of kernel and the treatment of its hyperparameters, can play a crucial role in obtaining a good optimizer that can achieve expert-level performance. We describe new algorithms that take into account the variable cost (duration) of learning algorithm experiments and that can leverage the presence of multiple cores for parallel experimentation. We show that these proposed algorithms improve on previous automatic procedures and can reach or surpass human expert-level optimization for many algorithms including Latent Dirichlet Allocation, Structured SVMs and convolutional neural networks.
booktitleAdvances in Neural Information Processing Systems
titlePractical Bayesian Optimization of Machine Learning Algorithms

abstractAccurate and fast simulation of particle physics processes is crucial for the high-energy physics community. Simulating particle interactions with detectors is both time consuming and computationally expensive. With the proton-proton collision energy of 13 TeV, the Large Hadron Collider is uniquely positioned to detect and measure the rare phenomena that can shape our knowledge of new interactions. The High-Luminosity Large Hadron Collider (HL-LHC) upgrade will put a significant strain on the computing infrastructure due to increased event rate and levels of pile-up. Simulation of high-energy physics collisions needs to be significantly faster without sacrificing the physics accuracy. Machine learning approaches can offer faster solutions, while maintaining a high level of fidelity. We discuss a graph generative model that provides effective reconstruction of LHC events, paving the way for full detector level fast simulation for HL-LHC.
titleGraph Generative Models for Fast Detector Simulations in High Energy Physics
Computer Science - Machine Learning,High Energy Physics - Experiment

abstractWe develop a generative neural network for the generation of sparse data in particle physics using a permutation-invariant and physics-informed loss function. The input dataset used in this study consists of the particle constituents of hadronic jets due to its sparsity and the possibility of evaluating the network's ability to accurately describe the particles and jets properties. A variational autoencoder composed of convolutional layers in the encoder and decoder is used as the generator. The loss function consists of a reconstruction error term and the Kullback-Leibler divergence between the output of the encoder and the latent vector variables. The permutation-invariant loss on the particles' properties is combined with two mean-squared error terms that measure the difference between input and output jets mass and transverse momentum, which improves the network's generation capability as it imposes physics constraints, allowing the model to learn the kinematics of the jets.
titleSparse Data Generation for Particle-Based Simulation of Hadronic Jets in the LHC
High Energy Physics - Experiment,Physics - Data Analysis,Statistics and Probability

abstractWe study how to use Deep Variational Autoencoders for a fast simulation of jets of particles at the LHC. We represent jets as a list of constituents, characterized by their momenta. Starting from a simulation of the jet before detector effects, we train a Deep Variational Autoencoder to return the corresponding list of constituents after detection. Doing so, we bypass both the time-consuming detector simulation and the collision reconstruction steps of a traditional processing chain, speeding up significantly the events generation workflow. Through model optimization and hyperparameter tuning, we achieve state-of-the-art precision on the jet four-momentum, while providing an accurate description of the constituents momenta, and an inference time comparable to that of a rule-based fast simulation.
journaltitleMachine Learning: Science and Technology
shortjournalMach. Learn.: Sci. Technol.
titleParticle-based Fast Jet Simulation at the LHC with Variational Autoencoders
03.00520 [hep-ex, physics:hep-ph, physics:physics]
ine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Computational Physics

abstractAutoencoders have useful applications in high energy physics in anomaly detection, particularly for jets - collimated showers of particles produced in collisions such as those at the CERN Large Hadron Collider. We explore the use of graph-based autoencoders, which operate on jets in their "particle cloud" representations and can leverage the interdependencies among the particles within a jet, for such tasks. Additionally, we develop a differentiable approximation to the energy mover's distance via a graph neural network, which may subsequently be used as a reconstruction loss function for autoencoders.
titleParticle Graph Autoencoders and Differentiable, Learned Energy Mover's Distance
Computer Science - Machine Learning,High Energy Physics - Experiment,Physics - Data Analysis,Statistics and Probability

abstractPoint clouds provide a flexible geometric representation suitable for countless applications in computer graphics; they also comprise the raw output of most 3D data acquisition devices. While hand-designed features on point clouds have long been proposed in graphics and vision, however, the recent overwhelming success of convolutional neural networks (CNNs) for image analysis suggests the value of adapting insight from CNN to the point cloud world. Point clouds inherently lack topological information so designing a model to recover topology can enrich the representation power of point clouds. To this end, we propose a new neural network module dubbed EdgeConv suitable for CNN-based high-level tasks on point clouds including classification and segmentation. EdgeConv acts on graphs dynamically computed in each layer of the network. It is differentiable and can be plugged into existing architectures. Compared to existing modules operating in extrinsic space or treating each point independently, EdgeConv has several appealing properties: It incorporates local neighborhood information; it can be stacked applied to learn global shape properties; and in multi-layer systems affinity in feature space captures semantic characteristics over potentially long distances in the original embedding. We show the performance of our model on standard benchmarks including ModelNet40, ShapeNetPart, and S3DIS.
titleDynamic Graph CNN for Learning on Point Clouds
Computer Science - Computer Vision and Pattern Recognition

abstractCalorimeter simulation is the most computationally expensive part of Monte Carlo generation of samples necessary for analysis of experimental data at the Large Hadron Collider (LHC). The High-Luminosity upgrade of the LHC would require an even larger amount of such samples. We present a technique based on Discrete Variational Autoencoders (DVAEs) to simulate particle showers in Electromagnetic Calorimeters. We discuss how this work paves the way towards exploration of quantum annealing processors as sampling devices for generation of simulated High Energy Physics datasets.
titleCaloDVAE : Discrete Variational Autoencoders for Fast Calorimeter Shower Simulation
Computer Science - Machine Learning,High Energy Physics - Experiment,Physics - Instrumentation and Detectors,Statistics - Machine Learning

abstractProbabilistic models with discrete latent variables naturally capture datasets composed of discrete classes. However, they are difficult to train efficiently, since backpropagation through discrete variables is generally not possible. We present a novel method to train a class of probabilistic models with discrete latent variables using the variational autoencoder framework, including backpropagation through the discrete latent variables. The associated class of probabilistic models comprises an undirected discrete component and a directed hierarchical continuous component. The discrete component captures the distribution over the disconnected smooth manifolds induced by the continuous component. As a result, this class of models efficiently learns both the class of objects in an image, and their specific realization in pixels, from unsupervised data, and outperforms state-of-the-art methods on the permutation-invariant MNIST, Omniglot, and Caltech-101 Silhouettes datasets.
titleDiscrete Variational Autoencoders
Computer Science - Machine Learning,Statistics - Machine Learning

abstractTraining of discrete latent variable models remains challenging because passing gradient information through discrete units is difficult. We propose a new class of smoothing transformations based on a mixture of two overlapping distributions, and show that the proposed transformation can be used for training binary latent models with either directed or undirected priors. We derive a new variational bound to efficiently train with Boltzmann machine priors. Using this bound, we develop DVAE++, a generative model with a global discrete prior and a hierarchy of convolutional continuous variables. Experiments on several benchmarks show that overlapping transformations outperform other recent continuous relaxations of discrete latent variables including Gumbel-Softmax (Maddison et al., 2016; Jang et al., 2016), and discrete variational autoencoders (Rolfe 2016).
titleDVAE++: Discrete Variational Autoencoders with Overlapping Transformations
Computer Science - Machine Learning,Statistics - Machine Learning

abstractBoltzmann machines (BMs) are appealing candidates for powerful priors in variational autoencoders (VAEs), as they are capable of capturing nontrivial and multi-modal distributions over discrete variables. However, non-differentiability of the discrete units prohibits using the reparameterization trick, essential for low-noise back propagation. The Gumbel trick resolves this problem in a consistent way by relaxing the variables and distributions, but it is incompatible with BM priors. Here, we propose the GumBolt, a model that extends the Gumbel trick to BM priors in VAEs. GumBolt is significantly simpler than the recently proposed methods with BM prior and outperforms them by a considerable margin. It achieves state-of-the-art performance on permutation invariant MNIST and OMNIGLOT datasets in the scope of models with only discrete latent variables. Moreover, the performance can be further improved by allowing multi-sampled (importance-weighted) estimation of log-likelihood in training, which was not possible with previous models.
titleGumBolt: Extending Gumbel trick to Boltzmann priors
Computer Science - Artificial Intelligence,Computer Science - Machine Learning,Statistics - Machine Learning

abstractThe restricted Boltzmann machine is a network of stochastic units with undirected interactions between pairs of visible and hidden units. This model was popularized as a building block of deep learning architectures and has continued to play an important role in applied and theoretical machine learning. Restricted Boltzmann machines carry a rich structure, with connections to geometry, applied algebra, probability, statistics, machine learning, and other areas. The analysis of these models is attractive in its own right and also as a platform to combine and generalize mathematical tools for graphical models with hidden variables. This article gives an introduction to the mathematical analysis of restricted Boltzmann machines, reviews recent results on the geometry of the sets of probability distributions representable by these models, and suggests a few directions for further investigation.
shorttitleRestricted Boltzmann Machines
titleRestricted Boltzmann Machines: Introduction and Review
Computer Science - Information Theory,Computer Science - Machine Learning,Mathematics - Probability,Mathematics - Statistics Theory,Statistics - Machine Learning

abstractDeep generative models parametrised by neural networks have recently started to provide accurate results in modelling natural images. In particular, generative adversarial networks provide an unsupervised solution to this problem. In this work we apply this kind of technique to the simulation of particle-detector response to hadronic jets. We show that deep neural networks can achieve high-fidelity in this task, while attaining a speed increase of several orders of magnitude with respect to traditional algorithms.
issn2510-2036, 2510-2044
journaltitleComputing and Software for Big Science
shortjournalComput Softw Big Sci
titleFast and accurate simulation of particle detectors using generative adversarial networks
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability

abstractPrecision measurements and new physics searches at the Large Hadron Collider require efficient simulations of particle propagation and interactions within the detectors. The most computationally expensive simulations involve calorimeter showers. Advances in deep generative modelling - particularly in the realm of high-dimensional data - have opened the possibility of generating realistic calorimeter showers orders of magnitude more quickly than physics-based simulation. However, the high-dimensional representation of showers belies the relative simplicity and structure of the underlying physical laws. This phenomenon is yet another example of the manifold hypothesis from machine learning, which states that high-dimensional data is supported on low-dimensional manifolds. We thus propose modelling calorimeter showers first by learning their manifold structure, and then estimating the density of data across this manifold. Learning manifold structure reduces the dimensionality of the data, which enables fast training and generation when compared with competing methods.
titleCaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors
titleRepresentation Learning: A Review and New Perspectives | IEEE Journals & Magazine | IEEE Xplore

abstractWe introduce manifold-learning flows (M-flows), a new class of generative models that simultaneously learn the data manifold as well as a tractable probability density on that manifold. Combining aspects of normalizing flows, GANs, autoencoders, and energy-based models, they have the potential to represent datasets with a manifold structure more faithfully and provide handles on dimensionality reduction, denoising, and out-of-distribution detection. We argue why such models should not be trained by maximum likelihood alone and present a new training algorithm that separates manifold and density updates. In a range of experiments we demonstrate how M-flows learn the data manifold and allow for better inference than standard flows in the ambient data space.
titleFlows for simultaneous manifold learning and density estimation
Computer Science - Machine Learning,Statistics - Machine Learning

abstractDeep learning has had tremendous success at learning low-dimensional representations of high-dimensional data. This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis, which states that the data lies on an unknown manifold of low intrinsic dimension. In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in image data. Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. To address this deficiency, we consider the union of manifolds hypothesis, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. We empirically verify this hypothesis on commonly-used image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant. We also provide insights into the implications of the union of manifolds hypothesis in deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this structure improves performance across classification and generative modelling tasks. Our code is available at https://github.com/layer6ai-labs/UoMH.
titleVerifying the Union of Manifolds Hypothesis for Image Data
07.02862 [cs, stat]
ine Learning,Statistics - Machine Learning

abstractI present a Variational Autoencoder (VAE) trained on collider physics data (specifically boosted $W$ jets), with reconstruction error given by an approximation to the Earth Movers Distance (EMD) between input and output jets. This VAE learns a concrete representation of the data manifold, with semantically meaningful and interpretable latent space directions which are hierarchically organized in terms of their relation to physical EMD scales in the underlying physical generative process. The variation of the latent space structure with a resolution hyperparameter provides insight into scale dependent structure of the dataset and its information complexity. I introduce two measures of the dimensionality of the learnt representation that are calculated from this scaling.
titleAn Exploration of Learnt Representations of W Jets
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology

abstractIn collider-based particle and nuclear physics experiments, data are produced at such extreme rates that only a subset can be recorded for later analysis. Typically, algorithms select individual collision events for preservation and store the complete experimental response. A relatively new alternative strategy is to additionally save a partial record for a larger subset of events, allowing for later specific analysis of a larger fraction of events. We propose a strategy that bridges these paradigms by compressing entire events for generic offline analysis but at a lower fidelity. An optimal-transport-based $\beta$ Variational Autoencoder (VAE) is used to automate the compression and the hyperparameter $\beta$ controls the compression fidelity. We introduce a new approach for multi-objective learning functions by simultaneously learning a VAE appropriate for all values of $\beta$ through parameterization. We present an example use case, a di-muon resonance search at the Large Hadron Collider (LHC), where we show that simulated data compressed by our $\beta$-VAE has enough fidelity to distinguish distinct signal morphologies.
titleMachine-Learning Compression for Particle Physics Discoveries
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability

abstractA key question for machine learning approaches in particle physics is how to best represent and learn from collider events. As an event is intrinsically a variable-length unordered set of particles, we build upon recent machine learning efforts to learn directly from sets of features or "point clouds". Adapting and specializing the "Deep Sets" framework to particle physics, we introduce Energy Flow Networks, which respect infrared and collinear safety by construction. We also develop Particle Flow Networks, which allow for general energy dependence and the inclusion of additional particle-level information such as charge and flavor. These networks feature a per-particle internal (latent) representation, and summing over all particles yields an overall event-level latent representation. We show how this latent space decomposition unifies existing event representations based on detector images and radiation moments. To demonstrate the power and simplicity of this set-based approach, we apply these networks to the collider task of discriminating quark jets from gluon jets, finding similar or improved performance compared to existing methods. We also show how the learned event representation can be directly visualized, providing insight into the inner workings of the model. These architectures lend themselves to efficiently processing and analyzing events for a wide variety of tasks at the Large Hadron Collider. Implementations and examples of our architectures are available online in our EnergyFlow package.
journaltitleJournal of High Energy Physics
shortjournalJ. High Energ. Phys.
shorttitleEnergy Flow Networks
titleEnergy Flow Networks: Deep Sets for Particle Jets
0.05165 [hep-ex, physics:hep-ph, stat]
Energy Physics - Experiment,High Energy Physics - Phenomenology,Statistics - Machine Learning

abstractLearning an interpretable factorised representation of the independent data generative factors of the world without supervision is an important precursor for the development of artificial intelligence that is able to learn and reason in the same way that humans do. We introduce beta-VAE, a new state-of-the-art framework for automated discovery of interpretable factorised latent representations from raw image data in a completely unsupervised manner. Our approach is a modification of the variational autoencoder (VAE) framework. We introduce an adjustable hyperparameter beta that balances latent channel capacity and independence constraints with reconstruction accuracy. We demonstrate that beta-VAE with appropriately tuned beta > 1 qualitatively outperforms VAE (beta = 1), as well as state of the art unsupervised (InfoGAN) and semi-supervised (DC-IGN) approaches to disentangled factor learning on a variety of datasets (celebA, faces and chairs). Furthermore, we devise a protocol to quantitatively compare the degree of disentanglement learnt by different models, and show that our approach also significantly outperforms all baselines quantitatively. Unlike InfoGAN, beta-VAE is stable to train, makes few assumptions about the data and relies on tuning a single hyperparameter, which can be directly optimised through a hyper parameter search using weakly labelled data or through heuristic visual inspection for purely unsupervised data.
eventtitleInternational Conference on Learning Representations
titlebeta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

abstractWe present the first steps in the development of a new class of hadronization models utilizing machine learning techniques. We successfully implement, validate, and train a conditional sliced-Wasserstein autoencoder to replicate the Pythia generated kinematic distributions of first-hadron emissions, when the Lund string model of hadronization implemented in Pythia is restricted to the emissions of pions only. The trained models are then used to generate the full hadronization chains, with an IR cutoff energy imposed externally. The hadron multiplicities and cumulative kinematic distributions are shown to match the Pythia generated ones. We also discuss possible future generalizations of our results.
titleModeling hadronization using machine learning
High Energy Physics - Phenomenology

abstractIn this paper we study generative modeling via autoencoders while using the elegant geometric properties of the optimal transport (OT) problem and the Wasserstein distances. We introduce Sliced-Wasserstein Autoencoders (SWAE), which are generative models that enable one to shape the distribution of the latent space into any samplable probability distribution without the need for training an adversarial network or defining a closed-form for the distribution. In short, we regularize the autoencoder loss with the sliced-Wasserstein distance between the distribution of the encoded training samples and a predefined samplable distribution. We show that the proposed formulation has an efficient numerical solution that provides similar capabilities to Wasserstein Autoencoders (WAE) and Variational Autoencoders (VAE), while benefiting from an embarrassingly simple implementation.
shorttitleSliced-Wasserstein Autoencoder
titleSliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
Computer Science - Machine Learning,Statistics - Machine Learning

abstractThe precise modeling of subatomic particle interactions and propagation through matter is paramount for the advancement of nuclear and particle physics searches and precision measurements. The most computationally expensive step in the simulation pipeline of a typical experiment at the Large Hadron Collider (LHC) is the detailed modeling of the full complexity of physics processes that govern the motion and evolution of particle showers inside calorimeters. We introduce \textsc{CaloGAN}, a new fast simulation technique based on generative adversarial networks (GANs). We apply these neural networks to the modeling of electromagnetic showers in a longitudinally segmented calorimeter, and achieve speedup factors comparable to or better than existing full simulation techniques on CPU ($100\times$-$1000\times$) and even faster on GPU (up to $\sim10ˆ5\times$). There are still challenges for achieving precision across the entire phase space, but our solution can reproduce a variety of geometric shower shape properties of photons, positrons and charged pions. This represents a significant stepping stone toward a full neural network-based detector simulation that could save significant computing time and enable many analyses now and in the future.
issn2470-0010, 2470-0029
journaltitlePhysical Review D
shortjournalPhys. Rev. D
titleCaloGAN: Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks
2.10321 [hep-ex, physics:hep-ph, stat]
ine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Statistics - Machine Learning

abstractHigh-precision modeling of subatomic particle interactions is critical for many fields within the physical sciences, such as nuclear physics and high energy particle physics. Most simulation pipelines in the sciences are computationally intensive – in a variety of scientific fields, Generative Adversarial Networks have been suggested as a solution to speed up the forward component of simulation, with promising results. An important component of any simulation system for the sciences is the ability to condition on any number of physically meaningful latent characteristics that can effect the forward generation procedure. We introduce an auxiliary task to the training of a Generative Adversarial Network on particle showers in a multi-layer electromagnetic calorimeter, which allows our model to learn an attribute-aware conditioning mechanism.
issn1742-6588, 1742-6596
journaltitleJournal of Physics: Conference Series
shortjournalJ. Phys.: Conf. Ser.
titleControlling Physical Attributes in GAN-Accelerated Simulation of Electromagnetic Calorimeters
1.08813 [hep-ex, physics:physics]
ine Learning,High Energy Physics - Experiment,Physics - Data Analysis,Statistics and Probability

abstractPhysicists at the Large Hadron Collider (LHC) rely on detailed simulations of particle collisions to build expectations of what experimental data may look like under different theory modeling assumptions. Petabytes of simulated data are needed to develop analysis techniques, though they are expensive to generate using existing algorithms and computing resources. The modeling of detectors and the precise description of particle cascades as they interact with the material in the calorimeter are the most computationally demanding steps in the simulation pipeline. We therefore introduce a deep neural network-based generative model to enable high-fidelity, fast, electromagnetic calorimeter simulation. There are still challenges for achieving precision across the entire phase space, but our current solution can reproduce a variety of particle shower properties while achieving speed-up factors of up to 100,000$\times$. This opens the door to a new era of fast simulation that could save significant computing time and disk space, while extending the reach of physics searches and precision measurements at the LHC and beyond.
issn0031-9007, 1079-7114
journaltitlePhysical Review Letters
shortjournalPhys. Rev. Lett.
shorttitleAccelerating Science with Generative Adversarial Networks
titleAccelerating Science with Generative Adversarial Networks: An Application to 3D Particle Showers in Multi-Layer Calorimeters
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Statistics - Machine Learning

abstractHigh Energy Physics (HEP) simulations are traditionally based on the Monte Carlo approach and generally rely on time consuming calculations. The present work investigates the use of Generative Adversarial Networks (GANs) as a fast alternative. Our approach treats the energy deposited by a particle inside a calorimeter detector as a three-dimensional image. True three-dimensional convolutions can be employed to capture the spatio-temporal correlation of shower energy depositions. Three-dimensional images are generated, conditioned on the energy of the incoming particle and validated against Monte Carlo simulation. The results show an agreement to full Mote Carlo simulations well within 10% thus proving that GAN can be used as a fast alternative for simulation of HEP detector response.
booktitle2018 25th IEEE International Conference on Image Processing (ICIP)
eventtitle2018 25th IEEE International Conference on Image Processing (ICIP)
noteISSN: 2381-8549
titleThree Dimensional Energy Parametrized Generative Adversarial Networks for Electromagnetic Shower Simulation
Deep Neural Networks,Detectors,GAN,Gallium nitride,Generative adversarial networks,Generators,HEP,Monte Carlo methods,Physics,Simulation,Training

abstractMachine Learning techniques have been used in diﬀerent applications by the HEP community: in this talk, we discuss the case of detector simulation. The need for simulated events, expected in the future for LHC experiments and their High Luminosity upgrades, is increasing dramatically and requires new fast simulation solutions. We describe an R&D activity aimed at providing a conﬁgurable tool capable of training a neural network to reproduce the detector response and speed-up standard Monte Carlo simulation. This represents a generic approach in the sense that such a network could be designed and trained to simulate any kind of detector and, eventually, the whole data processing chain in order to get, directly in one step, the ﬁnal reconstructed quantities, in just a small fraction of time. We present the ﬁrst application of three-dimensional convolutional Generative Adversarial Networks to the simulation of high granularity electromagnetic calorimeters. We describe detailed validation studies comparing our results to Geant4 Monte Carlo simulation. Finally we show how this tool could be generalized to describe a whole class of calorimeters, opening the way to a generic machine learning based fast simulation approach.
journaltitleEPJ Web of Conferences
shortjournalEPJ Web Conf.
title3D convolutional GAN for fast simulation

abstractWe present the 3DGAN for the simulation of a future high granularity calorimeter output as three-dimensional images. We prove the efficacy of Generative Adversarial Networks (GANs) for generating scientific data while retaining a high level of accuracy for diverse metrics across a large range of input variables. We demonstrate a successful application of the transfer learning concept: we train the network to simulate showers for electrons from a reduced range of primary energies, we then train further for a five times larger range (the model could not train for the larger range directly). The same concept is extended to generate showers for other particles (photons and neutral pions) depositing most of their energies in electromagnetic interactions. In addition, the generation of charged pion showers is also explored, a more accurate effort would require additional data from other detectors not included in the scope of the current work. Our further contribution is a demonstration of using GAN-generated data for a practical application. We train a third-party network using GAN-generated data and prove that the response is similar to a network trained with data from the Monte Carlo simulation. The showers generated by GAN present accuracy within $10\%$ of Monte Carlo for a diverse range of physics features, with three orders of magnitude speedup. The speedup for both the training and inference can be further enhanced by distributed training.
titleFast Simulation of a High Granularity Calorimeter by Generative Adversarial Networks
High Energy Physics - Experiment,Physics - Instrumentation and Detectors

abstractUsing detailed simulations of calorimeter showers as training data, we investigate the use of deep learning algorithms for the simulation and reconstruction of single isolated particles produced in high-energy physics collisions. We train neural networks on single-particle shower data at the calorimeter-cell level, and show significant improvements for simulation and reconstruction when using these networks compared to methods which rely on currently-used state-of-the-art algorithms. We define two models: an end-to-end reconstruction network which performs simultaneous particle identification and energy regression of particles when given calorimeter shower data, and a generative network which can provide reasonable modeling of calorimeter showers for different particle types at specified angles and energies. We investigate the optimization of our models with hyperparameter scans. Furthermore, we demonstrate the applicability of the reconstruction model to shower inputs from other detector geometries, specifically ATLAS-like and CMS-like geometries. These networks can serve as fast and computationally light methods for particle shower simulation and reconstruction for current and future experiments at particle colliders.
journaltitleThe European Physical Journal C
shortjournalEur. Phys. J. C
shorttitleCalorimetry with deep learning
titleCalorimetry with deep learning: particle simulation and reconstruction for collider physics

abstractWe use adversarial network architectures together with the Wasserstein distance to generate or refine simulated detector data. The data reflect two-dimensional projections of spatially distributed signal patterns with a broad spectrum of applications. As an example, we use an observatory to detect cosmic ray-induced air showers with a ground-based array of particle detectors. First we investigate a method of generating detector patterns with variable signal strengths while constraining the primary particle energy. We then present a technique to refine simulated time traces of detectors to match corresponding data distributions. With this method we demonstrate that training a deep network with refined data-like signal traces leads to a more precise energy reconstruction of data events compared to training with the originally simulated traces.
titleGenerating and refining particle detector simulations using the Wasserstein distance in adversarial networks
Astrophysics - Instrumentation and Methods for Astrophysics,High Energy Physics - Experiment

Springer International Publishing

abstractThere is large consent that successful training of deep networks requires many thousand annotated training samples. In this paper, we present a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently. The architecture consists of a contracting path to capture context and a symmetric expanding path that enables precise localization. We show that such a network can be trained end-to-end from very few images and outperforms the prior best method (a sliding-window convolutional network) on the ISBI challenge for segmentation of neuronal structures in electron microscopic stacks. Using the same network trained on transmitted light microscopy images (phase contrast and DIC) we won the ISBI cell tracking challenge 2015 in these categories by a large margin. Moreover, the network is fast. Segmentation of a 512x512 image takes less than a second on a recent GPU. The full implementation (based on Caffe) and the trained networks are available at http://lmb.informatik.uni-freiburg.de/people/ronneber/u-net.
booktitleMedical Image Computing and Computer-Assisted Intervention – MICCAI 2015
seriesLecture Notes in Computer Science
titleU-Net: Convolutional Networks for Biomedical Image Segmentation
Convolutional Layer,Data Augmentation,Deep Network,Ground Truth Segmentation,Training Image

abstractTo match the statistical precision due to the large dataset that Belle II is expected to collect, simulations that accurately describe real data are required. The eﬀect of beam background must be considered and can be taken into account with overlaying random trigger data, but its large size is a technical challenge. This problem can be mitigated by generating beam background data with generative adversarial networks. A proof of principle is shown for the background data recorded by the pixel vertex detector.
journaltitleEPJ Web of Conferences
shortjournalEPJ Web Conf.
titleGeneration of Belle II Pixel Detector Background Data with a GAN

abstractGenerative Adversarial Networks (GANs) are powerful generative models, but suffer from training instability. The recently proposed Wasserstein GAN (WGAN) makes progress toward stable training of GANs, but sometimes can still generate only low-quality samples or fail to converge. We find that these problems are often due to the use of weight clipping in WGAN to enforce a Lipschitz constraint on the critic, which can lead to undesired behavior. We propose an alternative to clipping weights: penalize the norm of gradient of the critic with respect to its input. Our proposed method performs better than standard WGAN and enables stable training of a wide variety of GAN architectures with almost no hyperparameter tuning, including 101-layer ResNets and language models over discrete data. We also achieve high quality generations on CIFAR-10 and LSUN bedrooms.
titleImproved Training of Wasserstein GANs
Computer Science - Machine Learning,Statistics - Machine Learning

abstractSimulation is one of the key components in high energy physics. Historically it relies on the Monte Carlo methods which require a tremendous amount of computation resources. These methods may have difficulties with the expected High Luminosity Large Hadron Collider (HL LHC) need, so the experiment is in urgent need of new fast simulation techniques. We introduce a new Deep Learning framework based on Generative Adversarial Networks which can be faster than traditional simulation methods by 5 order of magnitude with reasonable simulation accuracy. This approach will allow physicists to produce a big enough amount of simulated data needed by the next HL LHC experiments using limited computing resources.
journaltitleEPJ Web of Conferences
shortjournalEPJ Web Conf.
titleGenerative Models for Fast Calorimeter Simulation.LHCb case
2.01319 [physics]
ine Learning,Physics - Data Analysis,Statistics and Probability

abstractWe propose and demonstrate the use of a model-assisted generative adversarial network (GAN) to produce fake images that accurately match true images through the variation of the parameters of the model that describes the features of the images. The generator learns the model parameter values that produce fake images that best match the true images. Two case studies show excellent agreement between the generated best match parameters and the true parameters. The best match model parameter values can be used to retune the default simulation to minimize any bias when applying image recognition techniques to fake and true images. In the case of a real-world experiment, the true images are experimental data with unknown true model parameter values, and the fake images are produced by a simulation that takes the model parameters as input. The model-assisted GAN uses a convolutional neural network to emulate the simulation for all parameter values that, when trained, can be used as a conditional generator for fast fake-image production.
issn2162-237X, 2162-2388
journaltitleIEEE Transactions on Neural Networks and Learning Systems
shortjournalIEEE Trans. Neural Netw. Learning Syst.
titleImage-based model parameter optimization using Model-Assisted Generative Adversarial Networks
2.00879 [hep-ex, stat]
ine Learning,High Energy Physics - Experiment,Statistics - Machine Learning

abstractThis paper describes an algorithm for verification of signatures written on a pen-input tablet. The algorithm is based on a novel, artificial neural network, called a "Siamese" neural network. This network consists of two identical sub-networks joined at their out(cid:173) puts. During training the two sub-networks extract features from two signatures, while the joining neuron measures the distance be(cid:173) tween the two feature vectors. Verification consists of comparing an extracted feature vector ~ith a stored feature vector for the signer. Signatures closer to this stored representation than a chosen thresh(cid:173) old are accepted, all other signatures are rejected as forgeries.
booktitleAdvances in Neural Information Processing Systems
titleSignature Verification using a "Siamese" Time Delay Neural Network

abstractWe present a method for training a similarity metric from data. The method can be used for recognition or verification applications where the number of categories is very large and not known during training, and where the number of training samples for a single category is very small. The idea is to learn a function that maps input patterns into a target space such that the L/sub 1/ norm in the target space approximates the "semantic" distance in the input space. The method is applied to a face verification task. The learning process minimizes a discriminative loss function that drives the similarity metric to be small for pairs of faces from the same person, and large for pairs from different persons. The mapping from raw to the target space is a convolutional network whose architecture is designed for robustness to geometric distortions. The system is tested on the Purdue/AR face database which has a very high degree of variability in the pose, lighting, expression, position, and artificial occlusions such as dark glasses and obscuring scarves.
booktitle2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)
eventtitle2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)
noteISSN: 1063-6919
titleLearning a similarity metric discriminatively, with application to face verification
pages539546 vol. 1
Artificial neural networks,Character generation,Drives,Face recognition,Glass,Robustness,Spatial databases,Support vector machine classification,Support vector machines,System testing

abstractThe process of learning good features for machine learning applications can be very computationally expensive and may prove difﬁcult in cases where little data is available. A prototypical example of this is the one-shot learning setting, in which we must correctly make predictions given only a single example of each new class. In this paper, we explore a method for learning siamese neural networks which employ a unique structure to naturally rank similarity between inputs. Once a network has been tuned, we can then capitalize on powerful discriminative features to generalize the predictive power of the network not just to new data, but to entirely new classes from unknown distributions. Using a convolutional architecture, we are able to achieve strong results which exceed those of other deep learning models with near state-of-the-art performance on one-shot classiﬁcation tasks.
titleSiamese Neural Networks for One-shot Image Recognition

abstractSignificant advances in deep learning have led to more widely used and precise neural network-based generative models such as Generative Adversarial Networks (GANs). We introduce a post-hoc correction to deep generative models to further improve their fidelity, based on the Deep neural networks using the Classification for Tuning and Reweighting (DCTR) protocol. The correction takes the form of a reweighting function that can be applied to generated examples when making predictions from the simulation. We illustrate this approach using GANs trained on standard multimodal probability densities as well as calorimeter simulations from high energy physics. We show that the weighted GAN examples significantly improve the accuracy of the generated samples without a large loss in statistical power. This approach could be applied to any generative model and is a promising refinement method for high energy physics applications and beyond.
journaltitleJournal of Instrumentation
shortjournalJ. Inst.
titleDCTRGAN: Improving the Precision of Generative Models with Reweighting
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors,Statistics - Machine Learning

abstractPrecise scientific analysis in collider-based particle physics is possible because of complex simulations that connect fundamental theories to observable quantities. The significant computational cost of these programs limits the scope, precision, and accuracy of Standard Model measurements and searches for new phenomena. We therefore introduce Deep neural networks using Classification for Tuning and Reweighting (DCTR), a neural network-based approach to reweight and fit simulations using all kinematic and flavor information – the full phase space. DCTR can perform tasks that are currently not possible with existing methods, such as estimating non-perturbative fragmentation uncertainties. The core idea behind the new approach is to exploit powerful high-dimensional classifiers to reweight phase space as well as to identify the best parameters for describing data. Numerical examples from $eˆ+eˆ-\rightarrow\text{jets}$ demonstrate the fidelity of these methods for simulation parameters that have a big and broad impact on phase space as well as those that have a minimal and/or localized impact. The high fidelity of the full phase-space reweighting enables a new paradigm for simulations, parameter tuning, and model systematic uncertainties across particle physics and possibly beyond.
issn2470-0010, 2470-0029
journaltitlePhysical Review D
shortjournalPhys. Rev. D
titleNeural Networks for Full Phase-space Reweighting and Parameter Tuning
High Energy Physics - Experiment,High Energy Physics - Phenomenology,Statistics - Machine Learning

abstractMultidimensional efficiency maps are commonly used in high energy physics experiments to mitigate the limitations in the generation of large samples of simulated events. Binned multidimensional efficiency maps are however strongly limited by statistics. We propose a neural network approach to learn ratios of local densities to estimate in an optimal fashion efficiencies as a function of a set of parameters. Graph neural network techniques are used to account for the high dimensional correlations between different physics objects in the event. We show in a specific toy model how this method is applicable to produce accurate multidimensional efficiency maps for heavy flavor tagging classifiers in HEP experiments, including for processes on which it was not trained.
titleEfficiency Parameterization with Neural Networks
High Energy Physics - Experiment,High Energy Physics - Phenomenology

abstractWe develop a graph generative adversarial network to generate sparse data sets like those produced at the CERN Large Hadron Collider (LHC). We demonstrate this approach by training on and generating sparse representations of MNIST handwritten digit images and jets of particles in proton-proton collisions like those at the LHC. We find the model successfully generates sparse MNIST digits and particle jet data. We quantify agreement between real and generated data with a graph-based Fr\'echet Inception distance, and the particle and jet feature-level 1-Wasserstein distance for the MNIST and jet datasets respectively.
titleGraph Generative Adversarial Networks for Sparse Data Generation in High Energy Physics
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Computational Physics,Physics - Data Analysis,Statistics and Probability

abstractIn high energy physics (HEP), jets are collections of correlated particles produced ubiquitously in particle collisions such as those at the CERN Large Hadron Collider (LHC). Machine learning (ML)-based generative models, such as generative adversarial networks (GANs), have the potential to significantly accelerate LHC jet simulations. However, despite jets having a natural representation as a set of particles in momentum-space, a.k.a. a particle cloud, there exist no generative models applied to such a dataset. In this work, we introduce a new particle cloud dataset (JetNet), and apply to it existing point cloud GANs. Results are evaluated using (1) 1-Wasserstein distances between high- and low-level feature distributions, (2) a newly developed Fr\'{e}chet ParticleNet Distance, and (3) the coverage and (4) minimum matching distance metrics. Existing GANs are found to be inadequate for physics applications, hence we develop a new message passing GAN (MPGAN), which outperforms existing point cloud GANs on virtually every metric and shows promise for use in HEP. We propose JetNet as a novel point-cloud-style dataset for the ML community to experiment with, and set MPGAN as a benchmark to improve upon for future generative models. Additionally, to facilitate research and improve accessibility and reproducibility in this area, we release the open-source JetNet Python package with interfaces for particle cloud datasets, implementations for evaluation and loss metrics, and more tools for ML in HEP development.
titleParticle Cloud Generation with Message Passing Generative Adversarial Networks
Computer Science - Machine Learning,High Energy Physics - Experiment

abstractSupervised learning on molecules has incredible potential to be useful in chemistry, drug discovery, and materials science. Luckily, several promising and closely related neural network models invariant to molecular symmetries have already been described in the literature. These models learn a message passing algorithm and aggregation procedure to compute a function of their entire input graph. At this point, the next step is to find a particularly effective variant of this general approach and apply it to chemical prediction benchmarks until we either solve them or reach the limits of the approach. In this paper, we reformulate existing models into a single common framework we call Message Passing Neural Networks (MPNNs) and explore additional novel variations within this framework. Using MPNNs we demonstrate state of the art results on an important molecular property prediction benchmark; these results are strong enough that we believe future work should focus on datasets with larger molecules or more accurate ground truth labels.
titleNeural Message Passing for Quantum Chemistry
Computer Science - Machine Learning,I.2.6

abstractDeep generative models are becoming widely used across science and industry for a variety of purposes. A common challenge is achieving a precise implicit or explicit representation of the data probability density. Recent proposals have suggested using classifier weights to refine the learned density of deep generative models. We extend this idea to all types of generative models and show how latent space refinement via iterated generative modeling can circumvent topological obstructions and improve precision. This methodology also applies to cases were the target model is non-differentiable and has many internal latent dimensions which must be marginalized over before refinement. We demonstrate our Latent Space Refinement (LaSeR) protocol on a variety of examples, focusing on the combinations of Normalizing Flows and Generative Adversarial Networks.
titleLatent Space Refinement for Deep Generative Models
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Statistics - Machine Learning

abstractWe propose a novel method for gradient-based optimization of black-box simulators using differentiable local surrogate models. In fields such as physics and engineering, many processes are modeled with non-differentiable simulators with intractable likelihoods. Optimization of these forward models is particularly challenging, especially when the simulator is stochastic. To address such cases, we introduce the use of deep generative models to iteratively approximate the simulator in local neighborhoods of the parameter space. We demonstrate that these local surrogates can be used to approximate the gradient of the simulator, and thus enable gradient-based optimization of simulator parameters. In cases where the dependence of the simulator on the parameter space is constrained to a low dimensional submanifold, we observe that our method attains minima faster than baseline methods, including Bayesian optimization, numerical optimization, and approaches using score function gradient estimators.
titleBlack-Box Optimization with Local Generative Surrogates
Computer Science - Machine Learning,High Energy Physics - Experiment,Physics - Data Analysis,Statistics and Probability,Statistics - Machine Learning

abstractForeseen increasing demand for simulations of particle transport through detectors in High Energy Physics motivated the search for faster alternatives to Monte Carlo-based simulations. Deep learning approaches provide promising results in terms of speed up and accuracy, among which generative adversarial networks (GANs) appear to be particularly successful in reproducing realistic detector data. However, the GANs tend to suffer from different issues such as not reproducing the full variability of the training data, missing modes problem, and unstable convergence. Various ensemble techniques applied to image generation proved that these issues can be moderated either by deploying multiple generators or multiple discriminators. This work follows a development of a GAN with two-dimensional convolutions that reproduces 3D images of an electromagnetic calorimeter. We build on top of this model and construct an ensemble of generators. With each new generator, the ensemble shows better agreement with the Monte Carlo images in terms of shower shapes and the sampling fraction.
journaltitleJournal of Physics: Conference Series
notePublisher: IOP Publishing
shortjournalJ. Phys.: Conf. Ser.
titleEnsemble Models for Calorimeter Simulations

abstractGenerative Adversarial Networks (GAN) (Goodfellow et al., 2014) are an effective method for training generative models of complex data such as natural images. However, they are notoriously hard to train and can suffer from the problem of missing modes where the model is not able to produce examples in certain regions of the space. We propose an iterative procedure, called AdaGAN, where at every step we add a new component into a mixture model by running a GAN algorithm on a reweighted sample. This is inspired by boosting algorithms, where many potentially weak individual predictors are greedily aggregated to form a strong composite predictor. We prove that such an incremental procedure leads to convergence to the true distribution in a finite number of steps if each step is optimal, and convergence at an exponential rate otherwise. We also illustrate experimentally that this procedure addresses the problem of missing modes.
titleAdaGAN: Boosting Generative Models
Computer Science - Machine Learning,Statistics - Machine Learning

abstractUsing generative adversarial networks (GANs), we investigate the possibility of creating large amounts of analysis-specific simulated LHC events at limited computing cost. This kind of generative model is analysis specific in the sense that it directly generates the high-level features used in the last stage of a given physics analyses, learning the N-dimensional distribution of relevant features in the context of a specific analysis selection. We apply this idea to the generation of muon four-momenta in $Z \to \mu\mu$ events at the LHC. We highlight how use-case specific issues emerge when the distributions of the considered quantities exhibit particular features. We show how substantial performance improvements and convergence speed-up can be obtained by including regression terms in the loss function of the generator. We develop an objective criterion to assess the geenrator performance in a quantitative way. With further development, a generalization of this approach could substantially reduce the needed amount of centrally produced fully simulated events in large particle physics experiments.
titleLHC analysis-specific datasets with Generative Adversarial Networks
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology

abstractA Generative-Adversarial Network (GAN) based on convolutional neural networks is used to simulate the production of pairs of jets at the LHC. The GAN is trained on events generated using MadGraph5 + Pythia8, and Delphes3 fast detector simulation. We demonstrate that a number of kinematic distributions both at Monte Carlo truth level and after the detector simulation can be reproduced by the generator network with a very good level of agreement. The code can be checked out or forked from the publicly accessible online repository https://gitlab.cern.ch/disipio/DiJetGAN .
journaltitleJournal of High Energy Physics
shortjournalJ. High Energ. Phys.
titleDijetGAN: A Generative-Adversarial Network Approach for the Simulation of QCD Dijet Events at the LHC
High Energy Physics - Experiment,High Energy Physics - Phenomenology

abstractEvent generation for the LHC can be supplemented by generative adversarial networks, which generate physical events and avoid highly inefficient event unweighting. For top pair production we show how such a network describes intermediate on-shell particles, phase space boundaries, and tails of distributions. In particular, we introduce the maximum mean discrepancy to resolve sharp local features. It can be extended in a straightforward manner to include for instance off-shell contributions, higher orders, or approximate detector effects.
journaltitleSciPost Physics
shortjournalSciPost Phys.
titleHow to GAN LHC Events
High Energy Physics - Phenomenology

abstractWe introduce a generative model to simulate radiation patterns within a jet using the Lund jet plane. We show that using an appropriate neural network architecture with a stochastic generation of images, it is possible to construct a generative model which retrieves the underlying two-dimensional distribution to within a few percent. We compare our model with several alternative state-of-the-art generative techniques. Finally, we show how a mapping can be created between different categories of jets, and use this method to retroactively change simulation settings or the underlying process on an existing sample. These results provide a framework for significantly reducing simulation times through fast inference of the neural network as well as for data augmentation of physical measurements.
issn1434-6044, 1434-6052
journaltitleThe European Physical Journal C
shortjournalEur. Phys. J. C
titleLund jet images from generative and cycle-consistent adversarial networks
359 [hep-ex, physics:hep-ph, stat]
ine Learning,Electrical Engineering and Systems Science - Image and Video Processing,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Statistics - Machine Learning

abstractImage-to-image translation is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image using a training set of aligned image pairs. However, for many tasks, paired training data will not be available. We present an approach for learning to translate an image from a source domain $X$ to a target domain $Y$ in the absence of paired examples. Our goal is to learn a mapping $G: X \rightarrow Y$ such that the distribution of images from $G(X)$ is indistinguishable from the distribution $Y$ using an adversarial loss. Because this mapping is highly under-constrained, we couple it with an inverse mapping $F: Y \rightarrow X$ and introduce a cycle consistency loss to push $F(G(X)) \approx X$ (and vice versa). Qualitative results are presented on several tasks where paired training data does not exist, including collection style transfer, object transfiguration, season transfer, photo enhancement, etc. Quantitative comparisons against several prior methods demonstrate the superiority of our approach.
titleUnpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Computer Science - Computer Vision and Pattern Recognition

abstractLund diagrams, a theoretical representation of the phase space within jets, have long been used in discussing parton showers and resummations. We point out that they can be created for individual jets through repeated Cambridge/Aachen declustering, providing a powerful visual representation of the radiation within any given jet. Concentrating here on the primary Lund plane, we outline some of its analytical properties, highlight its scope for constraining Monte Carlo simulations and comment on its relation with existing observables such as the zg variable and the iterated soft-drop multiplicity. We then examine its use for boosted electroweak boson tagging at high momenta. It provides good performance when used as an input to machine learning. Much of this performance can be reproduced also within a transparent log-likelihood method, whose underlying assumption is that different regions of the primary Lund plane are largely decorrelated. This suggests a potential for unique insight and experimental validation of the features being used by machine-learning approaches.
journaltitleJournal of High Energy Physics
shortjournalJ. High Energ. Phys.
titleThe Lund jet plane
Jets,QCD Phenomenology

abstractInitial studies have suggested generative adversarial networks (GANs) have promise as fast simulations within HEP. These studies, while promising, have been insufficiently precise and also, like GANs in general, suffer from stability issues.We apply GANs to to generate full particle physics events (not individual physics objects), explore conditioning of generated events based on physics theory parameters and evaluate the precision and generalization of the produced datasets. We apply this to SUSY mass parameter interpolation and pileup generation. We also discuss recent developments in convergence and representations that match the structure of the detector better than images.In addition we describe on-going work making use of large-scale distributed resources on the Cori supercomputer at NERSC, and developments to control distributed training via interactive jupyter notebook sessions. This will allow tackling high-resolution detector data; model selection and hyper-parameter tuning in a productive yet scalable deep learning environment.
journaltitleEPJ Web of Conferences
notePublisher: EDP Sciences
shortjournalEPJ Web Conf.
titleNext Generation Generative Neural Networks for HEP

abstractMeasuring the polarization fractions of the $Wˆ+Wˆ-$ scattering reveals the interactions of the Higgs boson as well as new neutral states that are related to the standard model electroweak symmetry breaking. The dileptonic channel has a relatively lower background rate, but the kinematics of its final states can not be fully reconstructed due to the presence of two neutrinos. We propose neural networks to establish maps between the distributions of measurable quantities and the distributions of the lepton angles in $W$ boson rest frames. New physics contributions and collision energy can largely affect the kinematic properties of the $Wˆ+Wˆ-$ scattering beside the lepton angles. To make the network in ignorance of that information, the loss function is modified in two different ways. We show that the networks are promising in reproducing the lepton angle distributions, and the precision of the fitted polarization fractions obtained from network predictions is comparable to that obtained with the truth lepton angle. Although the best-fit values of polarization fractions do not change much after including the background uncertainty, the precisions is substantially reduced. Our trained models are available at GitHub.
issn2470-0010, 2470-0029
journaltitlePhysical Review D
shortjournalPhys. Rev. D
titlePolarization measurement for the dileptonic channel of $Wˆ+ Wˆ-$ scattering using generative adversarial network
4 [hep-ex, physics:hep-ph]
Energy Physics - Experiment,High Energy Physics - Phenomenology

abstractWe propose an alternative generator architecture for generative adversarial networks, borrowing from style transfer literature. The new architecture leads to an automatically learned, unsupervised separation of high-level attributes (e.g., pose and identity when trained on human faces) and stochastic variation in the generated images (e.g., freckles, hair), and it enables intuitive, scale-specific control of the synthesis. The new generator improves the state-of-the-art in terms of traditional distribution quality metrics, leads to demonstrably better interpolation properties, and also better disentangles the latent factors of variation. To quantify interpolation quality and disentanglement, we propose two new, automated methods that are applicable to any generator architecture. Finally, we introduce a new, highly varied and high-quality dataset of human faces.
titleA Style-Based Generator Architecture for Generative Adversarial Networks
Computer Science - Machine Learning,Computer Science - Neural and Evolutionary Computing,Statistics - Machine Learning

abstractWe apply generative adversarial network (GAN) technology to build an event generator that simulates particle production in electron-proton scattering that is free of theoretical assumptions about underlying particle dynamics. The difficulty of efficiently training a GAN event simulator lies in learning the complicated patterns of the distributions of the particles physical properties. We develop a GAN that selects a set of transformed features from particle momenta that can be generated easily by the generator, and uses these to produce a set of augmented features that improve the sensitivity of the discriminator. The new Feature-Augmented and Transformed GAN (FAT-GAN) is able to faithfully reproduce the distribution of final state electron momenta in inclusive electron scattering, without the need for input derived from domain-based theoretical assumptions. The developed technology can play a significant role in boosting the science of existing and future accelerator facilities, such as the Electron-Ion Collider.
booktitleProceedings of the Thirtieth International Joint Conference on Artificial Intelligence
titleSimulation of electron-proton scattering events by a Feature-Augmented and Transformed Generative Adversarial Network (FAT-GAN)
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Statistics - Machine Learning

abstractVelasco, LuisaMcClellan, EvanSato, N.Ambrozewicz, PawelLiu, TianboMelnitchouk, W.Kuchera, MichelleAlanazi, YasirLi, YaohangThe recently proposed generative adversarial network (GAN)-based event generator, the Feature Augmented and Transformed GAN (FAT-GAN), has shown an impressive capability of reproducing inclusive electron–proton scattering events at given collision energy. In contrast, many practical applications require the event generator to have the flexibility of allowing users to specify the reaction energy as an input to produce the corresponding synthetic events. In this work, we extend the FAT-GAN framework by conditioning the component neural networks according to the given reaction energy. We demonstrate that this model, referred to as cFAT-GAN, can reliably produce inclusive event feature distributions and correlations for a continuous range of reaction energies by automatically interpolating and extrapolating from a set of trained energies. We employ a continuous energy feature representation to enable the networks to organically learn the distribution relationships between different reaction energies, laying the groundwork for accessing events at untrained energies. This continuous conditional energy provides a degree of versatility to the cFAT-GAN for its further development as a significant research tool in high-energy and nuclear physics.
booktitleDeep Learning Applications, Volume 3
seriesAdvances in Intelligent Systems and Computing
titlecFAT-GAN: Conditional Simulation of Electron–Proton Scattering Events with Variate Beam Energies by a Feature Augmented and Transformed Generative Adversarial Network

abstractWe propose and assess an alternative quantum generator architecture in the context of generative adversarial learning for Monte Carlo event generation, used to simulate particle physics processes at the Large Hadron Collider (LHC). We validate this methodology by implementing the quantum network on artificial data generated from known underlying distributions. The network is then applied to Monte Carlo-generated datasets of specific LHC scattering processes. The new quantum generator architecture leads to a generalization of the state-of-the-art implementations, achieving smaller Kullback-Leibler divergences even with shallow-depth networks. Moreover, the quantum generator successfully learns the underlying distribution functions even if trained with small training sample sets; this is particularly interesting for data augmentation applications. We deploy this novel methodology on two different quantum hardware architectures, trapped-ion and superconducting technologies, to test its hardware-independent viability.
titleStyle-based quantum generative adversarial networks for Monte Carlo events
Computer Science - Machine Learning,High Energy Physics - Phenomenology,Quantum Physics

abstractIn many scientific fields which rely on statistical inference, simulations are often used to map from theoretical models to experimental data, allowing scientists to test model predictions against experimental results. Experimental data is often reconstructed from indirect measurements causing the aggregate transformation from theoretical models to experimental data to be poorly-described analytically. Instead, numerical simulations are used at great computational cost. We introduce Optimal-Transport-based Unfolding and Simulation (OTUS), a fast simulator based on unsupervised machine-learning that is capable of predicting experimental data from theoretical models. Without the aid of current simulation information, OTUS trains a probabilistic autoencoder to transform directly between theoretical models and experimental data. Identifying the probabilistic autoencoder's latent space with the space of theoretical models causes the decoder network to become a fast, predictive simulator with the potential to replace current, computationally-costly simulators. Here, we provide proof-of-principle results on two particle physics examples, $Z$-boson and top-quark decays, but stress that OTUS can be widely applied to other fields.
journaltitleScientific Reports
shortjournalSci Rep
titleLearning to Simulate High Energy Particle Collisions from Unlabeled Data
High Energy Physics - Experiment,High Energy Physics - Phenomenology

abstractIn recent years fully-parametric fast simulation methods based on generative models have been proposed for a variety of high-energy physics detectors. By their nature, the quality of data-driven models degrades in the regions of the phase space where the data are sparse. Since machine-learning models are hard to analyse from the physical principles, the commonly used testing procedures are performed in a data-driven way and can't be reliably used in such regions. In our work we propose three methods to estimate the uncertainty of generative models inside and outside of the training phase space region, along with data-driven calibration techniques. A test of the proposed methods on the LHCb RICH fast simulation is also presented.
issn1742-6588, 1742-6596
journaltitleJournal of Physics: Conference Series
shortjournalJ. Phys.: Conf. Ser.
titleGenerative models uncertainty estimation
10.09767 [hep-ex, physics:hep-ph]
ine Learning,G.3,High Energy Physics - Experiment,High Energy Physics - Phenomenology,I.2.6,J.2

abstractEnsembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different \emph{forms} of uncertainty. However, ensembles come at a computational and memory cost which may be prohibitive for many applications. There has been significant work done on the distillation of an ensemble into a single model. Such approaches decrease computational cost and allow a single model to achieve an accuracy comparable to that of an ensemble. However, information about the \emph{diversity} of the ensemble, which can yield estimates of different forms of uncertainty, is lost. This work considers the novel task of \emph{Ensemble Distribution Distillation} (EnD$ˆ2$) — distilling the distribution of the predictions from an ensemble, rather than just the average prediction, into a single model. EnD$ˆ2$ enables a single model to retain both the improved classification performance of ensemble distillation as well as information about the diversity of the ensemble, which is useful for uncertainty estimation. A solution for EnD$ˆ2$ based on Prior Networks, a class of models which allow a single neural network to explicitly model a distribution over output distributions, is proposed in this work. The properties of EnD$ˆ2$ are investigated on both an artificial dataset, and on the CIFAR-10, CIFAR-100 and TinyImageNet datasets, where it is shown that EnD$ˆ2$ can approach the classification performance of an ensemble, and outperforms both standard DNNs and Ensemble Distillation on the tasks of misclassification and out-of-distribution input detection.
titleEnsemble Distribution Distillation
Computer Science - Machine Learning,Statistics - Machine Learning

abstractVariational inference relies on flexible approximate posterior distributions. Normalizing flows provide a general recipe to construct flexible variational posteriors. We introduce Sylvester normalizing flows, which can be seen as a generalization of planar flows. Sylvester normalizing flows remove the well-known single-unit bottleneck from planar flows, making a single transformation much more flexible. We compare the performance of Sylvester normalizing flows against planar flows and inverse autoregressive flows and demonstrate that they compare favorably on several datasets.
titleSylvester Normalizing Flows for Variational Inference
Computer Science - Artificial Intelligence,Computer Science - Machine Learning,Statistics - Machine Learning,Statistics - Methodology

abstractRecently, we introduced CaloFlow, a high-fidelity generative model for GEANT4 calorimeter shower emulation based on normalizing flows. Here, we present CaloFlow v2, an improvement on our original framework that speeds up shower generation by a further factor of 500 relative to the original. The improvement is based on a technique called Probability Density Distillation, originally developed for speech synthesis in the ML literature, and which we develop further by introducing a set of powerful new loss terms. We demonstrate that CaloFlow v2 preserves the same high fidelity of the original using qualitative (average images, histograms of high level features) and quantitative (classifier metric between GEANT4 and generated samples) measures. The result is a generative model for calorimeter showers that matches the state-of-the-art in speed (a factor of $10ˆ4$ faster than GEANT4) and greatly surpasses the previous state-of-the-art in fidelity.
shorttitleCaloFlow II
titleCaloFlow II: Even Faster and Still Accurate Generation of Calorimeter Showers with Normalizing Flows
Computer Science - Machine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors

abstractThere has been a lot of recent interest in designing neural network models to estimate a distribution from a set of examples. We introduce a simple modification for autoencoder neural networks that yields powerful generative models. Our method masks the autoencoder's parameters to respect autoregressive constraints: each input is reconstructed only from previous inputs in a given ordering. Constrained this way, the autoencoder outputs can be interpreted as a set of conditional probabilities, and their product, the full joint probability. We can also train a single network that can decompose the joint probability in multiple different orderings. Our simple framework can be applied to multiple architectures, including deep ones. Vectorized implementations, such as on GPUs, are simple and fast. Experiments demonstrate that this approach is competitive with state-of-the-art tractable distribution estimators. At test time, the method is significantly faster and scales better than other autoregressive estimators.
titleMADE: Masked Autoencoder for Distribution Estimation
Computer Science - Machine Learning,Computer Science - Neural and Evolutionary Computing,Statistics - Machine Learning

abstractA normalizing flow models a complex probability density as an invertible transformation of a simple base density. Flows based on either coupling or autoregressive transforms both offer exact density evaluation and sampling, but rely on the parameterization of an easily invertible elementwise transformation, whose choice determines the flexibility of these models. Building upon recent work, we propose a fully-differentiable module based on monotonic rational-quadratic splines, which enhances the flexibility of both coupling and autoregressive transforms while retaining analytic invertibility. We demonstrate that neural spline flows improve density estimation, variational inference, and generative modeling of images.
titleNeural Spline Flows
Computer Science - Machine Learning,Statistics - Machine Learning

abstractThe framework of normalizing flows provides a general strategy for flexible variational inference of posteriors over latent variables. We propose a new type of normalizing flow, inverse autoregressive flow (IAF), that, in contrast to earlier published flows, scales well to high-dimensional latent spaces. The proposed flow consists of a chain of invertible transformations, where each transformation is based on an autoregressive neural network. In experiments, we show that IAF significantly improves upon diagonal Gaussian approximate posteriors. In addition, we demonstrate that a novel type of variational autoencoder, coupled with IAF, is competitive with neural autoregressive models in terms of attained log-likelihood on natural images, while allowing significantly faster synthesis.
titleImproving Variational Inference with Inverse Autoregressive Flow
Computer Science - Machine Learning,Statistics - Machine Learning

abstractIn this paper, we explore the potential of generative machine learning models as an alternative to the computationally expensive Monte Carlo (MC) simulations commonly used by the Large Hadron Collider (LHC) experiments. Our objective is to develop a generative model capable of efficiently simulating detector responses for specific particle observables, focusing on the correlations between detector responses of different particles in the same event and accommodating asymmetric detector responses. We present a conditional normalizing flow model (CNF) based on a chain of Masked Autoregressive Flows, which effectively incorporates conditional variables and models high-dimensional density distributions. We assess the performance of the \cnf model using a simulated sample of Higgs boson decaying to diphoton events at the LHC. We create reconstruction-level observables using a smearing technique. We show that conditional normalizing flows can accurately model complex detector responses and their correlation. This method can potentially reduce the computational burden associated with generating large numbers of simulated events while ensuring that the generated events meet the requirements for data analyses.
titleGenerative Machine Learning for Detector Response Modeling with a Conditional Normalizing Flow
High Energy Physics - Experiment,Physics - Instrumentation and Detectors

abstractAutoregressive models are among the best performing neural density estimators. We describe an approach for increasing the flexibility of an autoregressive model, based on modelling the random numbers that the model uses internally when generating data. By constructing a stack of autoregressive models, each modelling the random numbers of the next model in the stack, we obtain a type of normalizing flow suitable for density estimation, which we call Masked Autoregressive Flow. This type of flow is closely related to Inverse Autoregressive Flow and is a generalization of Real NVP. Masked Autoregressive Flow achieves state-of-the-art performance in a range of general-purpose density estimation tasks.
titleMasked Autoregressive Flow for Density Estimation
Computer Science - Machine Learning,Statistics - Machine Learning

abstractWe introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.
titleGenerative Modeling by Estimating Gradients of the Data Distribution
Computer Science - Machine Learning,Statistics - Machine Learning

abstractEnergy-Based Models (EBMs), also known as non-normalized probabilistic models, specify probability density or mass functions up to an unknown normalizing constant. Unlike most other probabilistic models, EBMs do not place a restriction on the tractability of the normalizing constant, thus are more flexible to parameterize and can model a more expressive family of probability distributions. However, the unknown normalizing constant of EBMs makes training particularly difficult. Our goal is to provide a friendly introduction to modern approaches for EBM training. We start by explaining maximum likelihood training with Markov chain Monte Carlo (MCMC), and proceed to elaborate on MCMC-free approaches, including Score Matching (SM) and Noise Constrastive Estimation (NCE). We highlight theoretical connections among these three approaches, and end with a brief survey on alternative training methods, which are still under active research. Our tutorial is targeted at an audience with basic understanding of generative models who want to apply EBMs or start a research project in this direction.
titleHow to Train Your Energy-Based Models
88 [cs, stat]
ine Learning,Statistics - Machine Learning

abstractWe present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. Our implementation is available at https://github.com/hojonathanho/diffusion
titleDenoising Diffusion Probabilistic Models
39 [cs, stat]
ine Learning,Statistics - Machine Learning

abstractDiffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. Code is available at https://github.com/google-research/vdm .
titleVariational Diffusion Models
Computer Science - Machine Learning,Statistics - Machine Learning

abstractContrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.
titleHierarchical Text-Conditional Image Generation with CLIP Latents
04.06125 [cs]

abstractScore-based generative models are a new class of generative algorithms that have been shown to produce realistic images even in high dimensional spaces, currently surpassing other state-of-the-art models for different benchmark categories and applications. In this work we introduce CaloScore, a score-based generative model for collider physics applied to calorimeter shower generation. Three different diffusion models are investigated using the Fast Calorimeter Simulation Challenge 2022 dataset. CaloScore is the first application of a score-based generative model in collider physics and is able to produce high-fidelity calorimeter images for all datasets, providing an alternative paradigm for calorimeter shower simulation.
issn2470-0010, 2470-0029
journaltitlePhysical Review D
shortjournalPhys. Rev. D
titleScore-based Generative Models for Calorimeter Shower Simulation
06.11898 [hep-ex, physics:hep-ph, physics:physics]
ine Learning,High Energy Physics - Experiment,High Energy Physics - Phenomenology,Physics - Data Analysis,Statistics and Probability,Physics - Instrumentation and Detectors

abstractMany particle physics datasets like those generated at colliders are described by continuous coordinates (in contrast to grid points like in an image), respect a number of symmetries (like permutation invariance), and have a stochastic dimensionality. For this reason, standard deep generative models that produce images or at least a fixed set of features are limiting. We introduce a new neural network simulation based on a diffusion model that addresses these limitations named Fast Point Cloud Diffusion (FPCD). We show that our approach can reproduce the complex properties of hadronic jets from proton-proton collisions with competitive precision to other recently proposed models. Additionally, we use a procedure called progressive distillation to accelerate the generation time of our method, which is typically a significant challenge for diffusion models despite their state-of-the-art precision.
titleFast Point Cloud Generation with Diffusion Models in High Energy Physics
66 [hep-ex, physics:hep-ph]
Energy Physics - Experiment,High Energy Physics - Phenomenology