the data in a low dimension space. Exploring the distribution in this
bottleneck layer is useful to find structures in the data. Ma et al. (2019)
used a Convolutional Autoencoder (CAE) to explore a sample of radio active
galactic nuclei (AGNs). By feeding the CAE with images of radio AGNs with
hosts of different morphologies, they showed that the network naturally
clusters similar objects together in the bottleneck. In that particular work,
the low dimensional representation is also used for a downstream supervised
classification using the learned feature space as input for a supervised
network. This is also a common application of the dimensionality reduction
approach. When only a reduced subsample of labeled examples is available, the
reduced dimensionality space can be used to train a supervised network with
smaller training sets. A similar approach was followed by Cheng et al. (2020).
They used a CAE to represent images of galaxies in a low dimension space with
the goal of finding strong gravitational lenses. Sine the samples of labeled
lensed systems are typically small (see subsubsection 3.1.3 for more details),
unsupervised representation offers an alternative way to find lenses without
labels. The authors perform a clustering step in the latent space learned by
the neural network to automatically find groups of objects which similar
properties. They find that the CAE based method successfully isolates $\sim
60$ per cent of all lensing images in the training set. In Cheng et al.
(2021a) they extend the same approach to the unsupervised exploration of
galaxy morphology. Using a modified version of a Variational Autoencoder, they
obtain an unsupervised representation of nearby galaxies from the SDSS survey.
They then perform a hierarchical clustering in the latent space to conclude
that the neural network representations share some properties with the
classical Hubble sequence but provide a more meaningful representation,
especially for ambiguous intermediate morphological types. See also the work
by Spindler et al. (2021) for an application of VAEs to representation of
galaxy morphology. A similar approach is presented in the work by Zhou et al.
(2021). The authors apply a combination of CAE based representation with a
multi-clustering model to study the morphologies of high redshift galaxies
from the CANDELS survey. Portillo et al. (2020) also used the same type of
approach involving a Variational Autoencoder to represent spectra of nearby
galaxies. The authors projected SDSS spectra into a latent space of 6
dimensions and showed that the different types - i.e star-forming, quiescent -
are naturally separated without labeling (Figure 32). Interestingly, the non-
linear components of VAEs seem to enable a better separation than a simple PCA
decomposition if the latent space remains of dimension lower than $\sim 10$.
The conclusion is that dimensionality reduction with neural networks is a
sensitive way of exploring data of high dimension. Notice however that they
did not use convolutional layers.

Figure 32: Variational Autoencoder for dimensionality reduction and data
visualisation. The figure show how the different types of spectra (labeled
with different colors) populate different projections of the latent space.
Figure adapted from Portillo et al. (2020)

In a recent work, Teimoorinia et al. (2021) follow a similar approach, but
with additional layers of complexity, to explore Integral Field Unit (IFU)
data from the Manga survey. In their approach, called DESOM, the authors
propose to combine a convolutional Autoencoder and a SOM to represent spectra.
The spectra are fed to a CAE and projected into a latent space of lower
dimension. In the same training loop, the representations are used to train a
SOM that further clusters similar objects. That way, all spectra of a galaxy
can be passed into the machinery to obtain a _fingerprint_ for every object
which corresponds to the final projection into the SOM plane. The authors
propose going a step forward by passing again the obtained _fingerprint_ into
the DESOM to obtain a single location for a galaxy based on the 2D
distribution of all spectra beloging to the same galaxy.

Other works have also applied deep learning based dimensionality reduction to
assess data quality. For example, Mesarcik et al. (2020) used an Autoencoder
to explore radio data and identify possible technical failures in the
observations. This illustrates another interesting use of deep learning
dimensionality reduction techniques to quickly explore complex datasets.

Another deep learning approach for dimensionality reduction which is
increasing in popularity in the recent years, is what is generally known as
self-supervised learning through contrastive learning. As opposed to the
Autoencoder approach, where the projection depends on the architecture, in
contrastive learning, the computation of representations is more data
oriented. The general idea is to apply some perturbations to the input data so
that the networks learn to ignore those and cluster together data points
coming from the same parent input data. This is obtained by what is called a
contrastive loss term. We emphasize that it is not the goal of this review the
technical details of the different deep learning approaches, but to review how
they are being used in astronomy. We refer the reader to Chen et al. (2020)
and references therein for more details (see Appendix A for references on the
different deep learning methods mentioned in this review). Figure 33 shows a
very schematic representation of a contrastive learning setting. The output is
in essence similar to the one obtained with an Autoencoder - i.e. a
representation of data in a reduced latent space - but the underlying idea is
significantly different. One of the key advantages of a contrastive approach
is that the perturbations applied to the input data can be tuned for a science
case and turn the representations independent to a known undesired effect. In
astronomy, it can enable to mitigate the effects of instrumental or selection
biases for example. Contrastive learning has only started to be applied in
astrophysics relatively recently. The first work exploring self-supervised
learning is by Hayat et al. (2021). The authors used an existing network to
compute representations for multi-band SDSS images. Among the perturbations
applied to the images, they included standard rotations and cropping, but also
some adapted to astronomy, such as extinction. They showed that the
contrastive learning model successfully clusters galaxies with similar
morphological properties and therefore constitutes a promising way for data
exploration in astrophysics (Figure 34).

Figure 33: Illustration of a self supervised contrastive learning
architecture. Multiple random augmentations of the same image (positive pairs)
are fed to two different CNNs which map them into a latent representation
space. Also during training, pairs of completely different images (negative
pairs) are also fed to the two CNNs. The contrastive loss is optimized to
increase (decrease) the dot product of representations of positive (negative)
pairs. Contrastive learning is starting to be used for dimensionality
reduction and as a generalized feature extraction process for multiple
downstream tasks such as galaxy classification or photometric redshift
estimation. Figure 34: Self-supervised learning applied to multi band SDSS
images. The left panels shows a UMAP of the representations obtained with
contrastive learning. The panels on the right show the same UMAP color coded
with different galaxy properties. Similar images are clustered together.
Figure adapted from Hayat et al. (2021)

Sarmiento et al. (2021) also applied contrastive learning to visualize data of
nearby galaxies from the Manga survey. Instead of images, they used post
processed maps of stellar populations properties (metallicity, age) as well as
stellar kinematic maps. They also show that the self-supervised learning
setting is able to condense the information from this high dimensional dataset
into a subset of meaningful representations which contain information about
the physical properties of galaxies. Interestingly, this is a case in which
other simpler dimensionality reduction techniques such as PCA, or even
Autoencoders, fail, given the large amount of instrumental effects present in
the data. The authors show that more standard techniques tend to organize
galaxies based on properties of the instrument (i.e. fiber size) instead of
physical ones. Because the contrastive setting allows one to tune the
augmentations to a specific problem, it can be trained so that the
representations become independent of instrumental biases (Figure 35). The
inferred representations can be used for example to perform a clustering step
and identify different classes of objects. Sarmiento et al. (2021) show that
well-known types of galaxies appear naturally without any human supervision
from a data-driven perspective.

Figure 35: Representations of Manga maps using PCA and contrastive learning,
and projected into a UMAP. The two leftmost columns show the plane color coded
with non-physical parameter (e.g. number of fibers in the IFU). The rightmost
columns show the same maps color coded with physical properties. Self-
supervised representations cluster galaxies according to physical parameters
while PCA focuses mostly on the number of fibers. Figure adapted from
Sarmiento et al. (2021)

In addition to visualization, a common application of self-supervised
representations is to use them as input for other downstream tasks. For
example, the representations can be used for a subsequent supervised
classification. The fact that objects have already been clustered together,
helps to converge faster, which makes it specially appealing when a small
amount of labeled data is available (see section 3). Hayat et al. (2021)
showed indeed that, by using the latent space for morphological classification
of galaxies, they can reach a similar accuracy as with a fully supervised CNN
but using $>10$ times less labeled data. In the follow up work by Stein et al.
(2021a), they also explore how the self supervised representations can be used
to find strong gravitational lenses reaching similar conclusions. Similarly to
the work by Cheng et al. (2020), the representations are used with a small
sample of lenses to train a simple linear classifier and find new strong
lenses candidates.

###  5.2 Outlier detection

Anomaly or outlier detection is a fundamental aspect of discovery in physical
sciences. A fair amount of new results in astrophysics have been triggered by
serendipitous discoveries through the exploration of observational datasets.
With the arrival of new big-data surveys, finding potentially interesting
objects becomes increasingly difficult with purely human based approaches. In
the past year, unsupervised deep learning has been explored by several groups
as a way to assist astronomers in the search of potentially interesting
objects.

An anomaly or outlier is usually defined as a data point which properties
deviate from the average properties of objects in the sample, under some
metric. The visualization techniques described in the previous subsection,
which tend to cluster together data points with similar properties, can
therefore be useful as well to identify deviant objects.

From a probabilistic point of view, an outlier can be also defined as an
object which probability of observation under the probability density
distribution of a data set is smaller than a given $\epsilon$. In that
context, modern generative models (e.g. VAEs, GANs), can approximate the
probability density function $p(X)$ of a dataset $X$ with increasing accuracy.
Therefore, they can be employed to look for objects with a small probability
of observation (see Chalapathy & Chawla (2019) for a generic review of deep
learning techniques applied to anomaly detection).

####  5.2.1 Transient astronomy

The field of transient astronomy has been particularly active in this context.
As previously summarized, the field is about to experience a data revolution.
The forthcoming LSST survey will observe all the Southern Hemisphere sky every
$\sim 2-3$ nights, producing an unprecedented real time movie of the night
sky. The community expects to discover a significant amount of new types of
variable objects using this dataset (Li et al., 2022). Therefore there have
been over the past years a number of works exploring deep learning and machine
learning in general, to identify anomalous light curves in preparation of LSST
We emphasize again that deep learning is not the unique machine learning
approach to identify anomalies. Malanchev et al. (2021) performed a comparison
of several anomaly detection algorithms - isolation forests, on-class SVMs,
Gaussian Mixture Models and Local Outlier Factor - to identify outliers in the
Zwicky Transient Facility (ZTF). See also the work by Martínez-Galarza et al.
(2021) which used decision trees and manifold learning. Pruzhinskaya et al.
(2019) uses Isolation Forests as well on a set of features derived from the
light curves using interpolation with Gaussian Processes. In the following, we
will however focus on efforts relying on deep learning.

Villar et al. (2021b) used a Variational Recurrent Autoencoder (VRAE) network
described in Villar et al. (2020) to identify anomalous light curves from the
simulated PLAsTiCC dataset. The proposed methodology is based on three main
steps involving three different ML algorithms. First the light curves are
interpolated using Gaussian Processes (GPs). The resulting interpolations are
then fed into a Variational Autoencoder. The temporal aspect is encoded by
appending the time step to the elements representing the time series. The low
dimension representation of the time series is finally passed through an
Isolation Forest algorithm to assign an anomaly score. As opposed to well
defined supervised problems, evaluating and comparing anomaly detection
algorithms is always difficult since by definition the objective is not well
defined. In that particular work, the authors quote a $\sim 95\%$ purity in
identifying light curves others than the ones generated by well known types of
objects. However, by definition the exercise is incomplete, since the
sensitivity to unknown unknowns cannot be assessed. This work illustrates an
interesting way of combining multiple ML approaches though. In particular, the
introduction of a GP for preprocessing turns the model agnostic to the
sampling frequency of the time series which is a very interesting feature for
astronomical applications. A Variational Recurrent Autoencoder is also used by
Sánchez-Sáez et al. (2021) to identify changing-look Active Galactic Nuclei.
The approach is analogous but it is applied to real observations from the ZTF
survey. The VRAE is trained with light curves to obtain a representation in a
low dimension space and then Isolation Forest is used to associate anomaly
scores. Boone (2021) also employs a VAE to learn how to reconstruct light
curves and then uses the latent space to assign anomaly scores to potentially
deviant curves. The architecture used is slightly different than in the
previous two works. Namely, they add a layer introducing some physical
information about the light curve so that the setting can also be used for
classification. However, the overall idea is analogous in essence.

Muthukrishna et al. (2021) explored a different approach. They trained instead
an Autoregressive Generative Model to generate three known types of light
curves (SNIa, SNII, SNIb). They try then to use the trained models to
reconstruct other light curves and use as anomaly score the $\chi^{2}$
difference between the input light curve and the reconstructed one. The
underlying idea is that common light curves will be well reconstructed by the
Neural Network if they have properly learned $p(X)$ \- the probability density
function of the data distribution - while rare events will have larger
reconstruction errors. Interestingly, they find that Autoregressive models
used that way are not very efficient to identify outliers as compared for
example to a Bayesian reconstruction of light curves. The explanation put
forward is that neural networks are too efficient and are therefore able to
generate, with descent accuracy, even light curves which were not part of the
original dataset (Figure 36). This behavior of AutoRegressive models has also
been reported in the ML community (Ren et al., 2019). These models are indeed
able to easily reconstruct less structured data than the data used for
training, leading to small out-of-distribution probabilities. Some solutions
have been suggested, which will be discussed in the following subsection.

Figure 36: Anomaly scores for different types of light curves obtained with
deep AutoRegressive Model (bottom panel) and with a Bayesian Reconstruction
algorithm (top panel). Unknown light curves not used for training have larger
anomaly scores when using the Bayesian method than with the Neural Network.
Figure from Muthukrishna et al. (2021)

Although these works are very recent and the community is still at an
exploration phase, they confirm that outlier detection is in general a very
complex task. Deep learning offers interesting options, especially because of
the potential it has to accurately model the probability distribution of the
data. However, there is still no satisfactory solution available and all
require some level of human interaction to identify the most interesting
objects. This is eventually a structural issue, since the definition of
interesting depends on the scientific case. Following these conclusions, the
group of M. Lochner and collaborators have created dedicated tools to apply
different outlier detection methods and enable a human inspection of
potentially interesting candidates (see Lochner & Bassett, 2021). One
interesting feature this approach puts forward is the fact that another layer
of ML is added to adapt the definition of interesting objects to each user.

####  5.2.2 Imaging and spectroscopic outliers

Anomalies can also be found in static data, i.e. spectra or images of
galaxies. This is again specially relevant in the context of future large
imaging and spectroscopic surveys in which a manual inspection of all data
points is prohibitively time consuming. Therefore the community has also
explored several approaches to identify outliers in large imaging surveys. The
underlying idea is analogous to what has been described for time series, i.e.
identifying objects which present some deviations from the general properties
of the sample. Here again, there exist a large number of ML algorithms that
can be employed; many of them not based on deep learning. For example, the
recent work by Shamir (2021) uses a set of manually engineered features to
find outlier candidates in HST images. Dimensionality reduction techniques
such as Self Organizing Maps have also been explored as a way to identify
anomalous spectra Fustes et al. (2013). Baron & Poznanski (2017) used
unsupervised Random Forests to isolate the rarest spectra in the SDSS by using
individual fluxes as input. As done for the previous sections, we will focus
here on deep learning applications.

Storey-Fisher et al. (2021) trained a Generative Adversarial Network using
postage stamps of observed galaxies from the Hyper Suprime Cam (HSC) survey.
They selected galaxies above an apparent magnitude limit and trained a
Wassertsein Generative Adversarial Network (WGAN) to generate realistic images
of galaxies. The underlying idea is that the model will learn how to
accurately reproduce common galaxies but will fail when confronted to objects
which appear with a small frequency in the training set. Once trained, WGANs
do not provide an explicit latent space to sample. In order to associate
anomaly scores to all galaxies, the authors perform an iterative search to
identify the closest object that the WGAN can generate. They compute then an
anomaly score based on a combination of the quadratic difference between the
real and reconstructed image and an additional L2 difference of the features
of the last layer of the critic network of the WGAN. They show that the
framework is able to identify potentially interesting objects. However, a
significant fraction of them are only image artifacts. The authors propose to
add another dimensionality reduction layer with a CAE trained on the residual
images (difference between the WGAN reconstruction and the original image).
They show that, after this additional step, the different types of anomalies
cluster together and a visual inspection is proposed to identify the most
interesting anomalies (Figure 37). Interestingly, the work also compares the
anomalies obtained with a less complex approach based on a CAE to reduce the
dimension of the data. The WGAN is able to find more subtle anomalies because
of the improved quality of the reconstruction. However, Tanaka et al. (2021)
showed on the same dataset, that a Convolutional Autoencoder is also able to
identify interesting anomalies. They quantify the performance of the anomaly
detection algorithm on a set of known extreme emission lines galaxies and
quasars. They report that $\sim 60\%$ of the objects belonging to these under
represented classes are identified.

Figure 37: Example of anomalous objects identified with a combination of WGAN
and a CAE applied to the HSC survey. The top panel shows the latent space from
the image residuals of the WGAN reconstruction obtained with the CAE. The
images below show examples of different regions of the parameter space. Figure
from Storey-Fisher et al. (2021)

Margalef-Bentabol et al. (2020) use the same WGAN approach outlined in Storey-
Fisher et al. (2021) but in a slightly different context. In this work, the
anomaly detection setting is used to assess the realism of galaxies produced
by cosmological simulations. In that context the WGAN is trained with mock
images from simulations. The trained model is then confronted with real
observations from the HST. The authors compare then the anomaly scores from
both datasets and conclude that the neural network struggles to reconstruct
some of the observed galaxies, meaning that they do not exist among the
simulated galaxies. Zanisi et al. (2021) also explores whether anomaly
detection approaches with deep learning can be used to compare galaxy images
from cosmological simulations to observations. They use however a different
approach based on the Autoregressive Model pixelCNN. Similar to GANs, pixelCNN
is a generative model which can be use to learn the probability density
function of data and generate new samples. However it provides an explicit
expression of the likelihood function built in an Autoregressive fashion; i.e.
the values of a given pixel are determined based on the values of the previous
ones. They apply the method to the comparison of SDSS and TNG galaxies.
Similarly to what was reported by Muthukrishna et al. (2021) for time series,
they find that the model can easily learn to generate simple objects and
therefore very smooth galaxy profiles or even pure noisy images achieve high
likelihoods of observation under the regressive model. To correct for this
effect, the authors use instead the likelihood ratio presented in Ren et al.
(2019) which forces the metric to become sensitive to the fine grained
structure of galaxies. They can then show how the likelihood ratio metric is
able to measure the improvement in the realism of cosmological simulations
from the first Illustris model to the updated TNG one (Figure 38).

An additional way to identify outliers is through the representation space
learned by contrastive learning, as one would do with a latent space from an
Autoeconder.Stein et al. (2021b) explored this approach on images from the
DESI survey and demonstrated the efficiency of self-supervised representations
to identify outliers and perform similarity searches. See also the work by
Walmsley et al. (2021) which uses the tools developed by Lochner & Bassett
(2021) for similarity search and anomaly detection on the representation
spaces.

Figure 38: Distribution of likelihood ratios obtained with two pixelCNN
networks of observations (SDSS) and different models as labeled. The closer
the histograms from simulations are to the one from SDSS, the more realistic
the simulation is. The unsupervised model is able to capture the improvement
of simulations with time. Figure from Zanisi et al. (2021)

###  5.3 Discovery of Physical Laws

Arguably one of the final goals of science is to find universal physical laws
which can explain a broad set of observations. As said several times in the
previous sections, deep neural networks offer an excellent predictive power
but their interpretability is low as compared to model-driven approaches.
Symbolic regression is the general term employed for the ensemble of
techniques that aim at uncovering an analytical equation from data. They can
be seen as a generalization of polynomial regression to the space of all
possible mathematical formulas that best predict the output variable taking as
input the input variables. See Schmidt & Lipson (2009) for more details. One
way to enable discovery with deep learning is therefore to apply symbolic
regression techniques to the trained deep neural network model. This is
usually very challenging given that neural network models are usually
parametrized by a large number of parameters. There is one work in
astrophysics attempting this by Cranmer et al. (2020). The authors train a GNN
in a supervised manner to predict the properties of some dataset encouraging a
sparse representation by the neural network. They then apply symbolic
regression to the trained model (Figure 39). The authors show that they are
able to recover for example some known Newtonian laws by predicting the
movement of particles. More interestingly, they discover a new analytic
formula which can predict the concentration of dark matter from the mass
distribution of nearby cosmic structures. The formula is learned by applying
symbolic regression to a GNN which learned the properties of a Dark Matter
only simulation. In a follow up work, Lemos et al. (2022) apply a similar
approach to study orbital mechanics.

Figure 39: Cartoon illustrating the method to extract physical equation from a
deep learning model applied to different datasets. Figure from Cranmer et al.
(2020)

Summary of Deep learning for discovery
• Deep learning has been explored as a discovery tool. The main motivation is
that future big data surveys are too large and too complex for efficient
human-based exploration. Deep learning is therefore mainly used for
visualization and anomaly - outlier - detection. The transient astronomy
community has been a particularly active field on this front in preparation
for LSST. • As opposed to previous applications, these applications rely on
unsupervised deep learning. There is a variety of different approaches which
have been tested: Autoencoders, Generative Models - GANs, VAEs, Autoregressive
Flows. Self-supervised approaches using contrastive learning have also started
to be used for this task. • For visualization, the usual approach is to use
deep learning to obtain a low dimensional representation of the data which can
be explored more easily. • Anomaly detection implies learning a probabilistic
description of the data and identifying objects with low likelihoods, i.e.
which can hardly reproduced by the trained models. • A common result is that
deep learning techniques correctly identify some anomalies, however the
quantification of performance is challenging because by definition the problem
is ill-posed. Some works find that complex deep learning networks might not be
the most efficient way of detecting outliers because they are flexible enough
to properly reproduce _simpler_ data than the data used for training. • In
addition, filtering out interesting anomalies from artifacts remains an
unsolved issue. Current solutions consists in providing anomalous candidates
for further inspection. • The issue of discovering physical laws from deep
learning models has been just recently explored by applying symbolic
regression methods on the trained models. It is difficult to generalize at
this stage given the large amount of parameters of current deep learning
models and the limited interpretability.

##  6 Deep learning for cosmology

In addition to galaxy formation, a key goal of modern deep surveys is to
constrain cosmology. Deep learning is playing an increasingly large and
promising role in at least two fronts: accelerating simulations - which are
needed for efficient cosmological inference - and direct inference of
cosmological parameters. This section is focused on these applications. We
first review approaches aimed at producing simulations and then we move to the
inference of cosmological parameters. .

###  6.1 Accelerating simulations

Numerical simulations play an important role in our understanding of galaxy
surveys, from shedding light into the physics of galaxy formation and
evolution to the modeling of the large scale structure of the Universe and its
connection to galaxies. Fast and efficient simulations are needed to interpret
the data. However, a major bottleneck is computational type. Ideally one would
like to have high resolution and large volume simulations including both
N-body and hydrodynamics. However, this is usually prohibitively time
consuming and for that reason fast emulators are desirable. In the past years,
deep learning has appeared as a promising solution, In this first subsection,
we review how it has started to impact the field of cosmological simulations.

####  6.1.1 Learning N-body Simulations

We begin by reviewing different strategies to either partially or entirely
learn the physics of N-body simulations and to some extent complement a
physical model to provide a significant acceleration compared to a full
simulation. We will split these methods into two categories, depending on
whether they act on and model matter density fields, or Lagrangian particle
displacements.

##### Lagrangian displacement models

N-body simulations typically model the matter distribution using tracer
particles and evolving in time the position and velocity of these particles
under the effect of gravitational forces. In this Lagrangian approach, the
full output of a simulation can be seen as the displacement that each particle
has undergone from its initial position on a regular lattice, along with its
final velocity. The methods described here all aim at modeling this
displacement field and therefore are not acting on 3D density fields, but on
this displacement sampled at the initial particle positions on a regular grid.

Modeling residual displacements against fast simulation: In Dai et al. (2018),
the authors propose a physically motivated post-processing technique, dubbed
Potential Gradient Descent (PGD), able to recover the small-scales of fast
Particle-Mesh (PM) simulations, and mimic the output of high-resolution N-body
simulations, or even mimic the baryonic feedback from hydrodynamical
simulations. The advantage of these fast PM simulations (such as FastPM (Feng
et al., 2016) or COLA (Tassev et al., 2013)) is that they can be run
inexpensively on very large comoving volumes, but their lack of force
resolution and their coarse time stepping limit their resolution, typically
leading to a lack of power on small scales and inaccurate halo profiles. The
method proposed in Dai et al. (2018) is to learn an additional displacement of
the particles, moving them deeper into their local gravitational potential,
which has the effect of sharpening the halo profiles. To compute this
displacement, instead of using Convolutional Neural Networks, the authors use
a physically motivated parameterization in Fourier space, defined by an
overall amplitude and a band-pass filter applied to the gravitational
potential for a total of only 3 parameters, which respects the translational
and rotational invariance of the problem. Training of the parameters of this
model is done by either minimizing the Mean Square Error (MSE) on the power
spectrum or on the density field between a reference simulation and a fast
simulation ran from the same initial conditions. With this simple, yet
powerful, scheme the authors can emulate to within 5% accuracy the Illustris-3
simulation from only a 10 step FastPM simulation.

In one of the first works applying deep learning to N-body simulations, He et
al. (2018) proposed a model based on a 3D convolutional U-Net (see subsection
3.2) that learned to predict full non-linear particle displacements given as
an input analytic Zel’dovich Approximation (ZA) displacements (which
corresponds to a single step of a FastPM algorithm). The 3D CNN takes as
inputs a 3-channel 3D field providing this ZA displacement field sampled at
the initial particle positions, and outputs the final displacement vector,
still as a 3-channels 3D field. The model is then trained by Mean Squared
Error loss on the reference displacement field provided by a FastPM
simulation. An important realization from that work was that a 3D CNN was able
to accurately model this displacement field. Figure 40 illustrates the
approximation error of this method (right column) compared to other fast
approximations for the displacement field. It is found to be significantly
more accurate than analytic solutions.

Figure 40: Illustration of learned displacement field in an N-body simulation
from He et al. (2018). The first column is the reference simulation (FastPM),
the second column shows a simple linear displacement (ZA), the third column
shows a second order Lagrangian Perturbation Theory displacement (2LPT) and
the last column shows the output of the 3D U-Net (D3M). The top row shows
final particle positions, the bottom row shows the final displacement field.
The color scale shows the error in position or displacement vectors between
each approach and the reference FastPM simulation.

With a similar model, Giusarma et al. (2019) showed that it was possible to
learn the residual displacements between a $\Lambda$CDM N-body simulation and
a simulation with massive neutrinos. They used a modified version of D3M to
learn this residual displacement at $z=0$ between the two sets of simulations,
and found excellent results down to $k<0.7h$Mpc-1.

Upsampling the displacement field from a low resolution simulation:
Recognizing that upsampling the displacement field is equivalent to increasing
the number of particles in a simulation, Li et al. (2020a) proposed a Super-
Resolution technique based on a conditional GAN directly inspired from a
StyleGAN2 (Karras et al., 2019) architecture. The generator takes as an input
a low resolution displacement field, and outputs an upsampled high-resolution
displacement field. This approach achieves impressive results up to an
upsampling factor of $\times 8$ translating into a direct computational
speedup of a factor $\times 1000$ in a setting where the goal would be to
produce a 100$h^{-1}$Mpc simulation with $512^{3}$ particles. A visual
illustration of this model is shown on Figure 41 where the rightmost panel is
the output of the model.

Figure 41: Illustration of N-body simulation super-resolution from Li et al.
(2020a) showing from left to right, the Low Resolution (LR) input, High
Resolution (HR) target, and Super Resolution output of the model. The bottom
row is a zoom-in on the region marked A.

As a further extension of this approach Ni et al. (2021) trained a similar
Lagrangian conditional GAN to model not only displacements but also
velocities, yielding a full phase-space information that a real N-body
simulation would produce. They further tested this model to demonstrate
accurate matter power spectrum recovery up to within 5% up to k $\geq$ 10
$h^{-1}$ Mpc for an upsampling factor of $\times 8$ of a 100 $h^{-1}$ Mpc box,
and validated the recovery of halo and sub-halo abundance.

##### Density field models

The first method for simulation super-resolution, proposed by Ramanah et al.
(2020), relied on a 3D Convolutional WGAN with a generator taking as inputs
the density field from a low resolution simulation run and a high resolution
set of initial conditions, and tasked with outputting a high resolution final
density field. The 3D convolutional discriminator compared the high resolution
density fields from the full simulation to the generator output. With this
approach, the authors were able to upsample by a factor of 2 the resolution of
the final density field, while reproducing faithfully a number of field
properties including the power spectrum, the density contrast probability
density function, and the bispectrum. This upsampling ratio represented a
computational speedup of about $\times 11$ for a 1 $h^{-1}$Mpc and $512^{3}$
particles simulation. More recently, Schaurecker et al. (2021) proposed a
similar model acting directly at the level of the density field, but only
using the low-resolution final density field as an input (without needing the
high-resolution initial conditions of Ramanah et al. (2020)).

####  6.1.2 N-body emulation by Deep Generative Modeling

All of the models from the previous section had the particularity of trying to
model the residuals compared to a physical model instead of completely
supplanting the physical simulation. In this section, we now cover the works
that have taken the approach of trying to learn from scratch the entire
simulation using a Deep Generative Model.

A number of works started to apply to the emulation of cosmological fields the
new Deep Generative Models that were gaining in popularity at the time, and
especially GANs. In the first instance of such an application, Mustafa et al.
(2019) applied a conventional DCGAN to the modeling of weak-lensing
convergence maps and demonstrated that the model was able to accurately
reproduce a number of statistics of these fields, including their power
spectra and Minkowski functionals. Shortly after, Rodriguez et al. (2018)
presented a similar application of using a DCGAN to model slices of N-body
simulations with results demonstrating that these models were able to capture
most of the relevant statistics of the cosmic web.

These early works on generative modeling for cosmological fields were quickly
confronted to the difficulty of building high quality models for very large or
even 3D fields. Perraudin et al. (2019a) explored more in depth the
limitations of simple DCGANs for generating large 3D N-body volumes and
highlighted two key strategies enabling high quality results in this setting:
1. generating the field by patches, 2. using a multi-resolution approach based
on a Laplacian pyramid. To generate N-body meshes of size $256^{3}$, their
proposed model uses 3 independent GANs that are trained on 3 increasingly high
data resolution ($32^{3}$, $64^{3}$, $256^{3}$). The first model trained on
the coarsest resolution is a conventional DCGAN while the other models are
conditioned on lower resolution inputs. In addition, the GANs are made
conditional on the neighboring patches in the simulation, which allows at
inference time to generate a large volume patch-by-patch in a sequential
fashion, thus avoiding the need of storing the entire volume in GPU memory.
Figure 42 illustrates the proposed strategy, which is shown in the paper to be
able to recover both the power spectrum and peak counts to satisfying levels
whereas a non-multiscale approach fails significantly.

Figure 42: Sequential generation and upsampling strategy of the scalable GAN
for N-body modeling presented in Perraudin et al. (2019a) scaling up to
$256^{3}$. The left illustration shows how the sequential generation of a
large volume would proceed. The right plot illustrates the proposed
architecture where the generator is conditioned on both neighboring patches,
and on the lower resolution patch, which at sampling time would be generated
by a separate GAN trained on coarser volumes. Distribute under the Creative
Commons CC BY licence (http://creativecommons.org/licenses/by/4.0/).

Additional works have investigated other possible improvements to GANs for
N-body simulations. In particular Feder et al. (2020) proposed modeling the
latent space prior of a DCGAN with a heavy-tailed distribution instead of a
Gaussian. In that work, using a Student’s t-distribution is shown to improve
the model’s ability to capture the sampling variance of the peaks in the dark
matter distribution, and overall improves power spectrum recovery on all
scales.

These generative models of the matter distribution start to become useful when
they are made conditional on some external parameters, for instance
cosmological parameters or redshifts. The GAN model proposed in Feder et al.
(2020) was for instance made conditional on redshift by simple concatenation
of the conditional variable to the latent vector of the GAN, allowing the
authors to generate volumes at intermediate redshifts, which would be useful
to create lightcones. (Perraudin et al., 2020) proposed a conditional GANs to
produce 2D weak lensing mass-maps conditioned on $(\sigma_{8},\Omega_{m})$
through a remapping of the latent vector of the GAN by a function that
rescales the norm of that vector based on the conditional variable. More
recently, (Wing Hei Yiu et al., 2021) extended that work to the sphere, using
a DeepSphere (Perraudin et al., 2019b) graph convolutional architecture, to
emulate the KiDS-1000 survey footprint as a function of
$(\sigma_{8},\Omega_{m})$.

While conditional GANs could be useful as emulators, they however cannot be
directly used for cosmological inference due to the fact that GANs do not
possess tractable likelihoods. One significantly different approach to
generative modeling of the dark matter distribution proposed in Dai & Seljak
(2022) relies on a Normalizing Flow approach instead of a GAN to model
explicitly the conditional distribution $p(x|\theta)$ where $x$ is the dark
matter distribution, and $\theta$ are cosmological parameters. Once trained,
such a model can directly be used as the likelihood function of the high-
dimensional data in a Markov-Chain Monte-Carlo. In this paper, the authors
introduce a Translation and Rotation Equivariant Normalizing Flow (TRENF)
model. It builds an n-d normalizing flow based on learning filters and
performs convolutions in Fourier space which impose by construction
translation and rotation equivariance. The authors demonstrate that this
approach accurately captures the high-dimensional likelihood of dark matter
density fields and that it can be used not only for generating these fields,
but also for inferring cosmological parameters.

####  6.1.3 Finding Dark Matter Halos

In the pipeline needed to go from N-body dark matter simulations to observable
galaxy distributions, a typically essential step is the identification of dark
matter halos, which can then be populated with galaxies under a variety of
techniques (e.g. HOD or SHAM). In this section, we review the various
approaches which have been proposed to go from the dark matter density field
to dark matter halos.

One of the first approaches to learn this connection was proposed in Modi et
al. (2018), and assumed a shallow neural network mapping between the local 3D
dark matter density and "halo mask" and "halo mass" fields. The binary halo
mask field was essentially used to model whether a given voxel actually
contained a halo, while the halo mass field was predicting in each voxel a
likely total halo mass. Given this model, a halo field could be recovered by
multiplying these two outputs of the neural network. The actual neural network
was based on a simple MLP taking as inputs a 3x3x3 voxel region of the dark
matter density field itself, the field smoothed on a given scale, and the
difference between the field smoothed on two difference scales. The authors
find that the predicted halo-mass field exhibits over a 95% correlation with
the true field up to $k=0.7h$Mpc-1. Perhaps most interestingly, this model
provided effectively a differentiable mapping between dark matter and halos, a
differentiable halo finder, and the authors demonstrated that this mapping
could be used in a reconstruction scheme to infer initial conditions from a
halo field by gradient descent through the neural network and a differentiable
N-body simulation as illustrated on Figure 43.

Figure 43: Illustration of an application of differentiable neural mapping
between the dark matter density and dark matter halo fields from Modi et al.
(2018). The top row shows the initial conditions dark matter field, final dark
matter field (at z=0), and the dark matter halo field obtained by a FoF halo
finder. The bottom row shows the result of a reconstruction by gradient
descent of these initial conditions, using a neural network to provide the
mapping between the final density field and the halo field.

Charnock et al. (2019) proposed a different approach to the same problem,
building a probabilistic model based on a Mixture Density Network for the halo
mass distribution in each voxel conditioned on the underlying dark matter
density. The overall goal of the authors in that paper was to make the model
minimalistically parametric while respecting the important physical properties
of the problem, namely that the halo bias should be globally rotational and
translational invariant. To reach this goal, instead of relying on standard
CNNs, the authors proposed a model they refer to as the Neural Physical Engine
(NPE), which applies a set of convolutional kernels parametrized to exhibit by
design the desired symmetries. This leads to a reduced number of parameters
compared to a similar 3D CNN. They apply this NPE on the dark matter density
field, and use its output to condition a simple Gaussian MDN tasked with
representing the local halo mass distribution. They demonstrate that a minimal
model with only 17 parameters is able to accurately capture the halo mass
distribution and its dependence on local environment, and further present an
application where the model is used as part of a Bayesian reconstruction of
initial conditions in a simulation from a given halo distribution.

Going in a deeper direction, a number of papers have looked into identifying
dark matter halos, not from Eulerian space, but directly as Lagrangian patches
at the level of the input density field of the simulation using a deep CNN
model. Berger & Stein (2018) introduced the first deep CNN model to identify
dark matter halos with this approach, relying on V-net model illustrated in
Berger & Stein (2018). They used as a training set simulated initial density
fields of size 1283 with a 1 $Mpc$ voxel resolution, along with a binary
$128^{3}$ segmented map indicating the position of Lagrangian patches
identified as proto-halos by the Peak Patch semi-analytic code Stein et al.
(2019). The network is then trained with a binary classification loss to
predict the presence or not of a halo in a given voxel. To build an actual
halo catalog given a segmented volume outputted by the trained model, the
authors then use a hierarchical Lagrangian halo-finding procedure that
ultimately returns a list of halos. This entire procedure is found to lead to
a halo-mass function and power spectrum within 10% of the ground truth
simulation. As a variation of this approach, Bernardini et al. (2019) proposed
to replace a binary segmentation by a regression problem, where the target
value corresponds to a normalized distance to the center of the halo, leading
to similar performance but with a smaller model.

In related work, Lucie-Smith et al. (2020); Etezad-Razavi et al. (2021)
propose to use a 3D CNN to predict from a initial conditions centered on the
location of halos, the mass of the collapsed halos at $z=0$. They use these
models to investigate the relative importance of various properties of the
initial conditions for halo formation. Lucie-Smith et al. (2020) reports for
instance that removing anisotropies in the initial conditions does not
significantly affect the masses predicted by the model, hinting that initial
shears may not be a significant factor in the halo formation process. Etezad-
Razavi et al. (2021) reports that velocity information becomes more important
to accurately predict masses at lower values of $A_{s}$.

####  6.1.4 Painting baryons on N-body simulations

As a way to bridge the gap between full hydro-dynamical simulations and
cheaper Dark Matter Only (DMO) simulations, a number of works have
investigated the possibility of "painting" baryons on top of DMO simulations.

A first category of papers proposed to model this mapping between dark matter
density and baryonic fields in a probabilistic fashion to account for the
inherent uncertainty. In the first work to attempt such modeling, Tröster et
al. (2019) investigated the use of both conditional VAEs and conditional GANs
to learn from 2D dark matter density slices a probabilistic map to 2D thermal
Sunyaev-Zeldovich (tSZ) maps, which capture the electron pressure. They found
excellent agreement with ground truth simulations, at different redshifts,
indicating that this approach was very promising, but did report a tradeoff of
GANs leading to more accurate results but being harder to train and less
stable than VAEs.

More recently, Bernardini et al. (2021) explored the use of a conditional
WGANs to learn a similar mapping to predict gas and HI density on 2D maps. To
achieve the generation of high-resolution and large maps, the authors adopt a
multi-resolution strategy in which a U-Net generator outputs maps at 3
different resolutions that the critic compares to similarly downsampled
versions of the target fields. This strategy allows them to successfully train
the model on images of size $512^{2}$, but after training this purely
convolutional model can be applied on much larger fields. The authors report
accurate HI power spectra to within 10% accuracy up to the 10 kpc scales while
being able to map simulation boxes of 100 $h^{-1}$Mpc on the side.

Extending these conditional generative approach to 3D fields, Horowitz et al.
(2021) proposed a conditional VAE with a customized U-Net architecture.
Contrary to a standard conditional VAE, this model features skip-connections
between the conditional branch (the dark matter encoder) and the generative
branch (the hydrodynamics decoder) which create a U-Net structure. It remains
however probabilistic thanks to the variational bottleneck block which
contains stochastic latent variables capable of capturing the aleatoric nature
of the mapping. The resulting model inherits from the stability and robustness
of VAEs but also benefits from this U-Net structure to enable high resolution
mapping. Again, the model is kept strictly convolutional to make it
insensitive to the size of the input field, so that it can be applied on
larger volume than it is trained on. This model can also be made conditional
on redshift and the authors demonstrate the possibility of generating
lightcones with this approach.

A second class of papers propose similar but deterministic mappings, using 3D
U-Nets trained to regress particular baryonic fields. Thiele et al. (2020)
proposed a 3D U-Net to learn a similar mapping, although no longer
probabilistic, between the 3D dark matter field and electron density,
momentum, and pressure. This work reports two significant challenges in
learning this mapping in 3D, one being the sparsity of interesting voxels (as
in 3D most of the voxels are in empty regions), and the other being the high
dynamic range of the fields to model. They address these challenges by biasing
the loss function towards high density regions, and applying range-compression
schemes. Overall they report better agreement with the reference simulations
compared to semi-analytical models. In similar work, Wadekar et al. (2020)
trained a U-Net to output HI density maps and again reported better quality
results than a standard HOD approach, while Harrington et al. (2021) used a
U-Net to predict hydrodynamical fields (density, temperature, velocity)
subsequently used to model Ly$\alpha$ fluxes with a physical prescription.
Zhang et al. (2019) proposed a two-step approach to map the dark matter field
to a 3D galaxy distribution, where a deep 3D CNN would predict a mask of the
likely non-empty voxel, and a second CNN would regress the number of galaxies
in these regions. They find that their CNN model is able to predict a galaxy
distribution recovering the expected power spectrum to within the $10\%$ level
up to k= $10h$Mpc-1 scale.

Finally, a singular approach was proposed in Dai & Seljak (2020) which
extended the PGD method of Dai et al. (2018) \- mentioned in the previous
section - to parametrize a mapping between a dark matter only simulation and
various fields accessible in hydrodynamical simulations using a combination of
particle displacements and voxel-wise non linearities. This method, dubbed
Lagrangian Deep Learning, reused the same Fourier-based parametrisation to
displace particles from a dark matter only simulation (e.g. FastPM), thus
using a very small number of parameters (order 10) and providing translation
and rotation equivariance, before painting them on a 3D mesh and applying a
non-linear transform. The few parameters can be fitted by gradient descent on
a single pair of dark matter only and hydrodynamical simulations. This scheme
was successfully demonstrated to reproduce a range of maps from IllustrisTNG,
including stellar mass distribution and electron momentum and pressure.

###  6.2 Deep learning for cosmological inference

In this section we will review in particular how emulators and Likelihood-Free
Inference techniques are enabling the inference not only of cosmological
parameters but also of cosmological fields.

####  6.2.1 Field-level Cosmological Constraints

##### Weak Gravitational Lensing

It was realized early on that given access to simulations to act as a training
set, neural networks can be used to extract cosmological information from
high-dimensional data such as maps of weak gravitational lensing maps. The
first example, presented in Schmelzle et al. (2017), demonstrated that a CNN-
based classification model could discriminate between different discrete
cosmological models, especially along the $\sigma_{8}-\Omega_{m}$ degeneracy
that conventional 2pt correlation functions are unable to resolve. Similar
results in Peel et al. (2019); Merten et al. (2019) showed that a CNN
classifier was able to distinguish between $\Lambda CDM$, modified gravity and
massive neutrinos models from weak lensing maps, with better discriminating
power than more conventional higher-order statistics such as peak statistics.
These results sparked a lot of interest into the potential use of CNNs to
extract cosmological information from weak lensing surveys, which resulted in
a number of subsequent publications with the ultimate goal of yielding proper
Bayesian posteriors on cosmological parameters.

Going beyond a classification task between discrete models, a second class of
papers (Gupta et al., 2018; Fluri et al., 2018; Ribli et al., 2019b) built CNN
regression models where the network is tasked with directly predicting
$(\sigma_{8},\Omega_{m})$, either using a Maximum Absolute Error (MAE) loss
(Gupta et al., 2018; Ribli et al., 2019b), or Gaussian-parameterized negative
log likelihood loss (Fluri et al., 2018). It is important to note, as reported
in all these papers, that the output of the network trained for regression
will not be an unbiased estimator for the cosmological parameters, but should
be interpreted as a low-dimensional summary statistic, which can then be used
for inference in a second step, independent from the network training. To
retrieve cosmological parameters, these papers assume a Gaussian likelihood on
the output of the network and characterize the mean and covariance of that
likelihood on a set of simulation, similarly to what is conventionally done
for peak count statistics or other Higher-Order Statistics without an analytic
likelihood. With this approach, all these papers reported the ability to
extract more information than a 2pt function analysis, even on realistically
noisy data, with Ribli et al. (2019b) reporting a factor of $\sim 2$ smaller
contours on $(\sigma_{8},\Omega_{m})$ in a Euclid or LSST setting.

Building on these promising results, the next phase of papers deployed these
approaches to actual survey data. Fluri et al. (2019) followed a similar
strategy to Fluri et al. (2018) and trained a ResNet model, on a suite of
tomographic lensing simulations mimicking the KiDS-450 survey and spanning a
range of $(\sigma_{8},\Omega_{m},A_{IA})$ values, where $A_{IA}$ is the
amplitude of the intrinsic galaxy alignment signal. This study found
constraints broadly consistent with the fiducial 2pt function analysis of
KiDS-450 (Hildebrandt et al., 2017) but when compared with an internal power-
spectrum analysis yielded a 30% tighter posterior. In the most recent
extension of that work, Fluri et al. (2022) performed a $w$CDM analysis of the
KiDS-1000 weak lensing maps including a large number of refinements. In
particular, they used a spherical CNN architecture, DeepSphere (Perraudin et
al., 2019b), in order to process spherical fields and extended their
simulation suites to include a baryonic prescription, and left the dark energy
equation of state parameter $w_{0}$ free to vary along with 5 other
cosmological parameters. They find again broad agreement with KiDS-1000
$w$CDM, and internally consistent results with a power spectrum analysis, but
with only a meagre $11\%$ improvement on S8 constraints. The most likely
reason for the limited constraining power of the CNN analysis comes from the
low-resolution of maps used in the analysis (HEALPix nside=1024), but
available non-Gaussian information content when baryonic systematics are taken
into account is also an open question discussed below.

Jeffrey et al. (2021) performed an analysis of the DES Science Verification
(SV) data using a slightly different approach to previous works. Instead of
training a CNN for regression, the authors introduced an information loss
which explicitly trains the network to compress the input lensing maps into a
low dimensional (asymptotically) sufficient statistic, that can further be
used for inference with a Likelihood-Free Inference approach. More
specifically, they used a Variational Mutual Information lower bound to train
the model, which relies on using a Normalizing Flow (NF) to approximate the
posterior distribution on cosmological parameter from the low dimensional
output of the convolutional compressor network, and training both models
jointly as to minimize the negative log likelihood of the NF. Once trained
under this information loss, the model can be applied to data, and a robust
estimate of the posterior was achieved by Neural Likelihood Estimation using
the pyDELFI package (Alsing et al., 2019). Compared to previous papers, this
approach has asymptotic optimality guarantees, and does not rely on any
Gaussian assumptions for the likelihood of the summary statistic. In this
paper, the authors found consistent but tighter constraints with this approach
compared to a power spectrum analysis, but the constraints remained very large
due to the small size of the SV dataset.

One question remains unclear, however, regarding the amount of additional
information deep learning can extract over the power spectrum or simpler
higher-order statistics, when systematics like baryonic effects are taken into
account. Lu et al. (2022) investigated this question using a simple baryonic
correction model (BCM Aricò et al., 2020) for dark matter only simulations and
trained a deep CNN to infer both cosmology and baryonic parameters from
simulated lensing maps under a realistic HSC-like setting. The authors find
that using a CNN instead of a power spectrum ($100<\ell<12000$) improves the
constraining power on $(\Omega_{m},\sigma_{8})$ (in terms of 1-sigma area) by
a factor of a few if the astrophysical parameters are kept fixed. However, the
improvement degrades significantly when marginalizing over astrophysical
parameters. Indicating that there is some amount of non-Gaussian information
left even after marginalizing on baryons, but how sensitive are the resulting
constraints to the specific baryonic model assumptions is uncertain.

With the success of these methods, several papers have attempted to introspect
the CNN trained on weak lensing maps, to try to identify or recognize what
features of the data are being used to extract the cosmological information.
Ribli et al. (2019a) proposed using at the first convolutional layer of the
model a large 7x7 kernel, and recognized after training that this first layer
was learning a kernel close to a Laplace operator and concluded that this
operator allowed the network to be sensitive to the steepness of the peaks in
convergence maps. Building on that insight, the authors handcrafted a new
summary statistic based on histograms of Sobel-filtered lensing-maps, which
are sensitive to peak steepness. This simple statistic was found to outperform
a deep CNN on noiseless data, while deteriorating in the presence of noise,
but still outperforming conventional peak counts.

Using a different methodology, Matilla et al. (2020) performed a similar study
using saliency methods to identify the features of the lensing maps relevant
to the cosmological inference task. They found that in all cases, the most
relevant pixels in the input maps were the ones with extreme values. In
noiseless maps, regions with negative convergence accounted for the majority
of the attribution, while on realistically noisy maps, the high value
convergence regions (positive peaks) account for the majority of the
attribution.

##### Large Scale Structure

Although not as actively researched, cosmological information can also be
extracted from the galaxy distribution with deep learning. This was first
illustrated by Ravanbakhsh et al. (2017) which used a 3D CNN to regress
cosmological parameters $(\sigma_{8},\Omega_{m})$ from the 3D dark matter
density in a suite of N-body simulations. Although not directly applicable to
actual surveys, this work demonstrated that convolutional approaches where
able to retrieve cosmological information from the 3D large-scale structure.
Addressing the same problem, but with the computational aspects of training 3D
CNNs on large-scale High Performance Computing (HPC) systems in mind,
Mathuriya et al. (2018) presented a similar result on cubes of size 1283, and
showcasing distributed training on 2048 and up to 8192 CPU nodes on the NERSC
Cori machine.

Going further in that direction, Ntampaka et al. (2019) presented a 3D CNN
model acting this time on the 3D distribution of galaxies, with spectroscopic
surveys in mind. This work relied on a suite of 40 dark matter simulations,
populated with galaxies with a range of various HOD models (15 different
models) as a way to marginalize over uncertainties on the galaxy-halo
connection. Galaxies in a given comoving volume were painted on 3D slabs of
size 550 x 550 x 220 $h^{-1}$Mpc to estimate a galaxy density field. A 3D CNN
was then tasked with outputting $(\sigma_{8},\Omega_{m})$ and the model was
trained by Maximum Absolute Error. The authors also proposed a variants on
that model, with an MLP branch taking directly as an input the power spectrum
of the volume, and combined or not with the CNN branch to provide the
cosmological parameter estimates. The main takeaways of that paper were that
the CNN was able to extract more information than the power spectrum alone,
and that the model trained in this fashion generalized well to unseen HOD
models.

##### Robustness to Baryonic Effects

Because deep learning-based cosmological inference schemes remain opaque, one
important question is how to robustify such an analysis to modeling
uncertainties and systematics. Just like in modern 2 point function analyses,
one of the most prominent questions is how to account for uncertainties in
Baryonic physics.

Answering this question is one of the motivations for the Cosmology and
Astrophysics with Machine-learning Simulations (CAMELS) suite (Villaescusa-
Navarro et al., 2021b), a set of about 4,000 simulations of
$25(h^{-1}$Mpc$)^{3}$ volumes which break down into $2,000$ dark matter only
simulations, $1,000$ hydrodynamical simulations following the IllustrisTNG
model, and $1,000$ hydrodynamical simulations using the SIMBA model (Davé et
al., 2019). Each of these 2 different hydrodynamical sets of simulations
varies not only cosmological parameters $(\Omega_{m},\sigma_{8})$ but also
astrophysical parameters, namely $(A_{SN1},A_{SN2})$ which regulate supernova
feedback and $(A_{AGN1},A_{AGN2})$ which parameterize AGN feedback. This suite
of simulations therefore not only allows the study of the dependency between
cosmological parameters and a given set of astrophysical systematics, but also
can be used to check the robustness to an assumption of a particular baryonic
feedback model (IllustrisTNG or SIMBA). On this dataset, the authors
demonstrated that not only does the total matter distribution contain
significant cosmological information accessible by deep neural networks, but
also baryonic fields such as the $H_{I}$ density (Villaescusa-Navarro et al.,
2021a), and that even individual galaxies bare some imprints of cosmological
parameters (Villaescusa-Navarro et al., 2022). The question however is: can
this information be retrieved under the uncertainties of the baryonic model?

In Villaescusa-Navarro et al. (2020), the authors illustrated on an
analytically tractable toy model that a Neural Network can be trained to
optimally marginalize over baryonic effects as long as the training data left
the associated parameters free to vary according to a given prior. While
promising, this result didn’t necessarily imply that this implicit
marginalization would be robust to a change in baryonic model. To investigate
precisely this question, Villaescusa-Navarro et al. (2020) trained a CNN on 2D
projected density fields from the CAMELS dataset to regress
$(\Omega_{m},\sigma_{8})$. They showed that models trained on IllustrisTNG
lead to almost unbiased results on SIMBA and vice-versa, implying that in the
process of learning a summary statistic that marginalizes over baryonic
effects, the neural networks are discarding the part of the signal affected by
baryons, and are therefore no longer extremely sensitive to the details of the
modeling of those effects. This result remains of course limited, but is very
encouraging for the analysis of data.

####  6.2.2 Dark Matter substructures from Strong Gravitational Lensing

In previous sections we have described how deep learning has significantly
impacted the detection and characterization of strong gravitational lenses.
Strong lensing systems can be used as well to constrain the substructure of
dark matter on extended arcs which contains a wealth of information about the
properties and distribution of dark matter on small scales and, consequently,
about the underlying nature of the dark matter particle. The information can
therefore be used to distinguish between various dark matter models - warm or
cold dark matter for example.

However, probing this effect is challenging since the likelihood function for
realistic simulations of population-level parameters is intractable. Alexander
et al. (2020) followed a simple approach which consists in converting the
inference problem into a classification of CNNs in classification mode to
distinguish various types of dark matter models. They show they can reach AUC
scores above $90\%$, for images with no substructure, spherical sub-halos, and
vortices on idealized simulations. Varma et al. (2020) also used a CNN for
multi-class classification in seven different categories corresponding to
different lower mass cut-offs of the subhalo mass function. They report being
able to correctly identify the lower mass cut-off within an order of magnitude
to better than $\sim 90\%$ accuracy.

Other works have attempted to go a step beyond by estimating the parameters
describing the dark matter substructure in a regression mode using simulation
based inference with deep learning. The first work exploring this is by
Brehmer et al. (2019). The authors characterize substructure with a set of
parameters and show, in a proof-of-concept application to simulated data, that
neural networks can be trained to accurately estimate likelihood ratios
associated to the dark matter substructure parameters (Figure 44). They
conclude that $\sim 100$ strong lenses might be enough for characterizing the
abundance of substructure down to $\sim 10\%$. Coogan et al. (2020) also used
a likelihood-free approach to infer posterior distribution of the mass and
positions of subhalos in simulated systems. Vernardos et al. (2020) applied a
similar approach combining a simulator based on Gaussian Random Fields for the
potential but combined with images of real galaxies for the lensed source and
show they can also constrain the substructure parameters.

Figure 44: Illustration of lensed systems and the corresponding likelihood
ratio maps estimated with simulation based inference and deep learning. The
black crosses show the true values. Figure from Brehmer et al. (2019)

####  6.2.3 Reconstructing cosmological fields

Applications in cosmology also go beyond cosmological parameter estimation,
and an increasingly large number of works explore applications of deep
learning for inferring latent cosmological fields from observations.

##### Weak Lensing Mass-Mapping

One active research question in weak gravitational lensing is the
reconstruction of the matter distribution that gives rise to the measured
lensing effect, a task known as mass-mapping. This problem is made
particularly difficult by the noisy nature of the observations (intrinsic
galaxy ellipticities being much larger than the weak gravitational shear) and
the need to invert a linear operator mapping shear to projected mass (also
known as convergence) which becomes ill-posed in the presence of survey masks.
This is therefore an instance of an ill-posed inverse problem, which does not
have any unique solution, in the sense that different mass-maps can lead to a
shear signal equally compatible with the data. For these problems, the hope of
Deep Learning approaches is that they can learn, implicitly or explicitly, a
prior on the signal to recover from training data, and use that prior to solve
the inverse problem in an optimal fashion.

The first class of methods, explored in Shirasaki et al. (2019); Shirasaki et
al. (2021), used a conditional adversarial network adapted from the pix2pix
model (Isola et al., 2016) for image-to-image translation. In this approach, a
first network with a U-Net structure is tasked with taking a noisy convergence
obtained by a rough direct inversion of the shear field as an input, and
outputting an estimate of the true convergence map. To train this denoiser, a
second network is introduced to act as a discriminator, taking as an input
both noisy and denoised convergence maps, coming either from the denoiser or
from the training set, and outputting a probability between 0 and 1 of the
denoised image being real. The model is then trained with a combination of a
standard adversarial loss and an l1 loss between the recovered denoised mass-
map and the truth from simulations. It is to be noted here that this
adversarial model is not a generative model, the denoiser does not take random
variables as an input and is therefore deterministic. Instead the adversarial
loss can be understood as a learned similarity metric to compare recovered to
true map. Training such a model typically requires a set of ray-traced lensing
simulations, that are corrupted to include the same noise properties and masks
as present in the data. Shirasaki et al. (2021) generated mock HSC
observations including photometric redshift uncertainties, shape measurement
uncertainties, realistic galaxy ellipticity noise and distribution on the sky,
and actual HSC survey masks. On simulations the authors find that about 60% of
the peaks identified on the denoised maps have significant clusters
counterparts, against about 85% of positive matches on true maps, highlighting
that the recovered map still correlate well with real structures, and the
authors further show that the 1-point statistics of the recovered map shows
stronger cosmological dependence than the noisy maps, hinting at interesting
applications in constraining cosmological parameters. While this approach
provides empirically good results, one drawback of this pix2pix training is
that the recovered map does not have a clear Bayesian interpretation. As we
will see below, subsequently developed techniques abandon this effective
adversarial training but gain a proper Bayesian interpretation of the output
of the models.

In Jeffrey et al. (2020), the authors introduce a method, called DeepMass,
using a similar Unet architecture but trained under a simple Mean Squared
Error loss between true convergence map and output from the Unet. As
highlighted by the authors, a regression model trained under an MSE implicitly
learns to predict the mean of the posterior distribution of the target given
the input. In the present case, the authors generate a suite of ray-tracing
lensing simulations matching the DES Science Verification setting, the
noiseless convergence maps from these simulations provide an implicit prior,
while the simulated shear observations (including realistic noise and masks)
provide an implicit likelihood. By training the model to reconstruct the true
convergence given simulated shear data under an MSE loss, the model will
therefore learn to output the mean posterior convergence map, under the
implicit prior and implicit likelihood that are provided by the training set.
In this DES SV setting, the authors demonstrate that this approach leads to an
11% improvement in MSE evaluated on simulations compared to a standard Wiener
filter approach.

While Jeffrey et al. (2020) had the benefit of providing a Bayesian
understanding of the network output, it did not provide any sort of
uncertainty on the recovered map, which makes the interpretation and
scientific exploitation of these results difficult. To overcome these
limitations, Remy et al. (2022) introduced an approach allowing to sample from
the full posterior distribution of the mass-mapping problem. They proposed to
use a similar U-Net architecture, not to directly estimate the convergence
map, but to learn from simulations a generative prior on convergence maps
using a Denoising Score Matching technique (Song & Ermon, 2019). With this
approach, it can be shown that a neural network trained as a Gaussian denoiser
under a simple MSE loss will actually learn the score of the data
distribution, i.e. the gradient of the log likelihood of the data. Once
trained on simulations, this U-Net gives explicit access to the prior. The
authors show that it is possible to combine this learned prior with an
explicit data likelihood in an Hamiltonian Monte-Carlo sampling procedure to
sample from the full posterior distribution of the problem. Figure 45
illustrates on the bottom row posterior samples achieved with this method on a
simulation of the HST/ACS COSMOS field, compared to the ground truth (top
left). Most interestingly, it is shown that the mean of the posterior samples
indeed converge to the same solution as the DeepMass estimate.

Figure 45: Illustration of weak lensing mass-map reconstructions in a
simulated COSMOS survey setting with the posterior sampling method of Remy et
al. (2022) and the DeepMass direct posterior mean estimation method of Jeffrey
et al. (2020). As can be seen, individual posterior samples (bottom row) are
visually similar to a real convergence map (e.g. ground truth at the top left)
but exhibit variability on structures not strongly constrained by data (e.g.
outside of the survey region marked by the white contours). The top row
illustrates that the DeepMass estimate indeed recovers the Bayesian posterior
mean.

##### Initial Conditions Reconstructions

One particularly interesting problem for the analysis of galaxy surveys is the
reconstruction of the initial density field from the observed Large Scale
Structure. This can for instance be used to refine Baryonic Acoustic
Oscillations (BAO) measurements (e.g. Schmittfull et al., 2017), or part of a
Bayesian forward modeling inference scheme (Seljak et al., 2017).

In a first example of applying deep learning to this problem, Mao et al.
(2020) proposed a 3D CNN trained on N-body simulations to recover the initial
density field at $z=10$ given the final density field at $z=0$ under a density
weighted mean squared error loss. They find that their convolution model is
capable of beating a standard linear reconstruction on scales smaller than
$k\leq 0.2h$Mpc-1, but under performs on larger scales. Interestingly they
find that their learned inversion can extrapolate to some extent to other
cosmological parameters; a model trained on WMAP7 cosmology is capable of
reconstructing initial conditions on WMAP5 simulations that lead to a slightly
biased BAO signal, but still significantly different from the WMAP7 signal of
the training data.

Figure 46: CosmicRIM initial conditions reconstruction technique of Modi et
al. (2021a) based on a 3D LSTM recurrent neural network and includes
explicitly at each iteration the gradients of the data likelihood. The plot
shows a given ground truth initial condition field (top left) and associated
final field (bottom left), along with the reconstructed initial conditions
(top right) and reconstructed final field (bottom right).

Going beyond a direct inversion method, Modi et al. (2021a) proposed an
iterative reconstruction scheme based on Recurrent Inference Machines (RIM,
Putzky & Welling, 2017). This approach can be thought of as a learned
iterative reconstruction algorithm. At each iteration a recurrent neural
network proposes an update of the current reconstruction based on the
knowledge of previous iterations and on the gradient of an explicit data
likelihood term. In the absence of this neural network, the algorithm would
result in a standard gradient descent scheme leading to a Maximum Likelihood
Estimation of the initial conditions. By training the Neural Network to
minimize at each iteration the Mean Squared Error between the current solution
and the true initial conditions, the network will learn both an implicit
prior, and a fast inference scheme to minimize the number of updates needed.
The result will therefore be a fast convergence towards the mean posterior
solution. Most interestingly, in order to compute this explicit likelihood, a
differentiable forward model is needed i.e. in this case an N-body simulation.
The authors make use of the FlowPM TensorFlow-based fast N-body code (Modi et
al., 2021b) for this likelihood which becomes just a layer within a neural
network (Figure 46). The authors propose to initialize the reconstruction at
the standard linear reconstruction, and show that in only 10 iterations this
method yields a better solution than an iterative reconstruction based on 400
iterations of an LBFGS minimizer.

Summary of Deep learning for Cosmology
1. Emulation • The main motivation for using deep learning in simulations is to bypass some of the expensive computational steps needed to generate large volume and high-resolution simulations needed to model modern surveys. Applications have targeted in particular: emulating N-body simulations, enhancing the resolution of existing simulations (super-resolution), and learning mappings between the 3D dark matter distribution and dark matter halos or a range of hydrodynamical fields. • Despite many impressive results, these methods have not yet been used for scientific applications. The main difficulty is that deep models can only be used within their training regimes (defined by specific training resolution, specific sets of cosmological parameters, or specific hydrodynamical run), and thus it is unclear whether they will bring concrete computational gains when the cost of the training sets are taken into account. • An alternative class of models, which so far as attracted limited attention, aims instead for a minimal set of parameters, building upon known symmetries and physical insight, greatly reducing the amount of training simulations needed and even opening the possibility of inferring these parameters from the data itself. 2. Cosmological Inference • Deep learning is opening a new way of comparing observations to theory: 1. it allows for the automatic extraction of cosmological information from high-dimensional data without requiring analytic summary statistics; 2. Neural Density Estimation makes it possible to perform Bayesian inference by leveraging numerical simulations. • Although in theory a deep learning approach is statistically sound, it assumes that the simulators provide an accurate physical model of the observations. Any unaccounted for systematics may result in biases, which due to the black-box nature of deep neural networks are difficult to test/detect. • A number of papers are starting to apply these methodology on data. In weak lensing, the gains compared to a more standard power spectrum analysis have remained limited on current generation surveys when systematics are included and marginalized over in the analysis. • A few applications have proposed to perform high-dimensional inference of cosmological fields (e.g. dark matter maps, reconstructing initial conditions). These works however do not yet attempt joint inference of fields and cosmological parameters.

##  7 Final thoughts: assessing the present and future of deep learning for
galaxy surveys

This final section is devoted to extract some indicators about the impact that
deep learning has had in the analysis of galaxy surveys. We also attempt to
highlight some of the key challenges these methods are facing, which in some
cases might prevent or delay the general deployment of deep learning for
scientific analysis. Some of these challenges have already been highlighted in
the previous sections, but this section tries to extract the most commonly
encountered

###  7.1 On the penetration of deep learning techniques in astronomy

We start by questioning what are the deep learning techniques most commonly
used in astronomy and how efficiently the rapid progresses made in the ML
community reach our community. In the previous sections, we have described
different applications of deep learning making use of a variety of techniques.
We summarize in Table 1, the broad type of neural network architectures used
in the four categories of scientific applications we defined in this review.
We have divided the neural network models in seven big groups. CNNs
encapsulates all variants of convolutional neural networks, from Vanilla to
more complex Residual Networks. The second group contains, generally speaking,
image to image networks such as Encoder-Decoders, Autoencoders but also
segmentation specific networks such as Mask R-CNNs or YOLO. The third family
of models are Generative Models which include Variational Autoencoders,
Generative Adversarial Networks and also Autoregressive models. We then
include Bayesian Neural Networks which allow for uncertainty quantification
(Mixture Density Networks and Flow models are also included in this category),
Recursive Neural Networks and Transformers mostly suited for sequences. The
last group is made of Graph Neural Networks. The table first shows that
applications in astronomy cover a wide range of deep learning techniques.
Although standard CNNs are the most commonly used - probably because it is the
most established approach and because imaging is the most common type of data
- other more recent models are regularly applied to astronomical data. On the
one side, this reflects the fact that astronomical data is rather diverse -
including images, but also spectra, time sequences, simulations and
observations. On the other hand, it suggests that the penetration of new ML
techniques is efficient. State-of-the art methods are rapidly applied to
astronomy. This is likely a consequence of the fact that, even advanced ML
methods are becoming increasingly easy to use for non-experts. It is almost
straightforward to test a new technique on an astrophysical problem with
current high level implementations. The downside is that, generally speaking,
the methods are often applied _blindly_ off the shelf, with little domain
specific adaptation. Consequently, a feature that is still lacking in a fair
amount of the applications of deep learning to astronomy is the inclusion of
previous physical knowledge into the data driven models. This can be done by
adapting the loss functions or by modifying the neural network architectures
to incorporate known symmetries (see work by Villar et al., 2021a; Bowles et
al., 2021). It obviously requires deeper knowledge of the machine learning
aspects which is something that will likely take more time.

Another interesting feature that emerges from Table 1 is that training on
simulations is the most common approach in astronomy. Almost all supervised
approaches rely at some stage on simulated data. It reflects that the samples
of labeled data remain small and/or that the measurements in observations are
noisy. Relying on simulations to train the models adds however an important
element of uncertainty to all applications. Machine learning approaches are
indeed very sensitive to domain shift issues. According to Table 1, almost all
recent applications are affected by those at some extent. We will discuss this
further in subsection 7.3.

Application Model |  |  |  CNNs |  Enc. |  Gene. |  BNN |  RNN |  Trans. |  GNN
---|---|---|---|---|---|---|---|---|---
1\. Computer Vision | Classification | Morphology |  |  |  |  |  |  |
|  | Strong Lenses | * | * |  |  |  |  |
|  | Transients |  |  |  |  | *) | *) |
| Segmentation |  |  | * | * |  |  |  |
2\. Galaxy Properties |  | Photoz |  |  |  |  |  |  |
|  | Structure | *) |  |  |  |  |  |
|  | Stellar Populations | * |  |  |  |  |  |
|  | Lensing | * |  |  | * |  |  |
|  | Physical Processes | * |  |  |  |  |  |
|  | Dark Matter | * |  |  | * |  |  | *
3\. Discovery |  | Visualization |  |  |  |  |  |  |
|  | Outliers |  |  |  |  |  |  |
|  | Laws |  |  |  |  |  |  | *
4\. Cosmology |  | Emulation | * | * | * | * |  |  |
|  | Cosmological inference | * |  | * | * |  |  |
Table 1: Overview of the different deep learning techniques used in the fields
of galaxy formation and cosmology, divided by type of application (see text
for details). CNNs: Standard classification and regression Convolutional
Neural Networks including modern architectures such as ResNets. Enc: Encoder-
Decoder networks and variants. Gene: Generative Models. BNNs: Bayesian Neural
Networks; we also include Mixture Density Networks. RNNs: Recursive Neural
Networks. Trans: Transformers. GNNs; Graph Neural Networks. A blue (red)
background indicates supervised (unsupervised) learning. The star symbol
highlights applications which require simulations to train the neural
networks. The bracket after the star symbol indicates that the use of
simulations is not always mandatory.

###  7.2 Measuring the impact of deep learning

We now move to measuring the impact of works using deep learning in the
astronomical literature. We have seen in the introduction that the number of
papers making use of neural networks has increased exponentially over the past
half decade. In this subsection, we try to measure the impact of these works
with some standard metrics. Figure 47 shows the evolution of the number of
papers, number of citations and average citations per paper in the period
2015-2021. The publications are divided in the four different categories
defined in this review, i.e. computer vision, galaxy properties, discovery and
cosmology. We have only included in the figure the works explored for this
work. As a consequence, it is very likely that the figure is not complete and
that the numbers presented are closer to a lower limit. However, it should
provide a good overview of the general trends and represents a more controlled
experiment than a purely automatic search. We also emphasize that the division
in categories is a choice by the authors of this review. Therefore, there is
some obvious overlap between the different types of applications.

Nevertheless, the figure reveals some interesting behaviors. We first confirm
the global increasing trend of the number of papers using deep learning for
galaxy surveys. Since 2015 there is a clear exponential increase. In 2021,
there are at least 70 papers using deep learning in the context of galaxy
surveys, while there were less than 5 in 2015. This is factor of $\sim 15$
increase and implies more than a paper per week on average. If we look at the
division per type of application, we see that computer vision type of
applications (i.e. classification, segmentation) concentrate the largest
fraction of publications. All the other remaining classes share similar
fractions. However, there is clearly a decreasing trend of the relative
importance of classification and segmentation applications. While the fraction
of these papers was around $70-80\%$ in 2015-2017, it is only of $\sim 20\%$
for papers published in 2021. The trends seem to suggest a diversification of
the applications of deep learning to astronomy moving from computer vision
tasks - mainly classification and segmentation - to a large variety of
different applications. The number of yearly works for data processing seems
to flatten indeed after 2019, while other applications like data exploration
(_discovery_) rapidly rise.

A similar behavior is observed in terms of citations. In 2016, roughly $80\%$
of the citations are for papers using deep learning for computer vision tasks.
The fraction is steadily decreasing, but still remains close to $\sim 50\%$ in
2021 even though the number of papers is only 20%. It probably reflects a
delay between the publication time and the time papers start to be cited.

(a) (b)

(c) (d)

Figure 47: Impact of works using deep learning for galaxy surveys. Each symbol
shows a different class of application as labeled (see text for details). The
top left and right panels show the number of papers and number of citations as
a function of time respectively. The bottom left and right panels show the
fraction of of papers and citations in each class of application.

We attempt to remove this effect in Figure 48. We plot the number of papers
per year as a function of the number of citations per paper and per year,
averaged over the time elapsed since the first citation in a given group. That
way, for papers focusing on computer vision, the time window is set to 6
years, while for the others, we consider 4 years. We also show in the figure,
for reference, the location of publications flagged with the keywords _galaxy
evolution_ , according to the NASA ADS search engine. Using this
normalization, we observe several interesting trends. Works using deep
learning for galaxy surveys, represent roughly $>5$% of all works focusing on
galaxies. This is not a large fraction but it is still remarkable given the
relatively recent emergence of deep learning. They receive on average $\sim
1.5$ times less citations per publication. It suggests that the impact of deep
learning papers remains moderate as compared to the average. We might
speculate about possible reasons. One possibility is that most of the works
making use of deep learning are thought in preparation of future big data
surveys which have not arrived yet (e.g. Euclid, LSST). The works are
therefore more a demonstration of feasibility. It could be argued that
existing surveys such as DES or SDSS for example are already good targets for
data driven science. This is certainly true, and as we have seen in this
review, there are many works targeting these surveys. However, the sizes and
dimensionality of current surveys still allow one to use relatively optimized
codes and leave some room for some manual checks. This is also supported by
the trends observed when the deep learning papers are divided by topic. The
highest citation rate is measured for works focusing on simulation (i.e.
emulation of cosmological simulations), which by definition do not require new
observational data. Publications focused on unsupervised discovery, which
strongly rely on new data being available, present the lowest impact, although
they show a strong increase in numbers (Figure 47). It could also be that the
technology is still too young, and it is just a matter of time that the impact
increases. The majority of the works are still at the proof-of-concept stage
and have not reached the deployment stage. This could also mean eventually
that there are some challenges / limitations which have not been deeply
explored yet and that prevent these new methods to be fully adopted by the
community. We explore these challenges more carefully in the following
section.

Figure 48: Number of citations normalized by the number of papers and years
(from the first publication in that category) as a function of the number of
papers per year. Each symbol shows a different category as defined in this
work (see text for details). The "all galaxy evolution" group includes an
automatic search of all publications in the field of galaxy formation (with or
without deep learning).

###  7.3 Challenges

We list in Table 2 what we think are some of the major challenges that deep
learning works face and which need to be addressed in the coming years by the
community based on the works reported in this review. Some of these challenges
are not specific to the astronomical community and can benefit from solutions
arising from the field of Machine Learning. However, in some cases, the
requirements are more strict in astronomy. The table also provides some
possible solutions along with some list of - non exhaustive - references which
have explored these solutions.

####  7.3.1 Small (and biased) labeled datasets

A major challenge in applications of deep learning for astronomy is the lack
of large enough labeled data sets to train supervised deep learning models.
This issue has been highlighted multiple times during the description of
works. We have seen in section 3 that deep learning was first applied to
classification of galaxy morphologies. This was mainly triggered by the fact
that large samples of labeled images existed from previous citizen science
efforts. A similar behavior is observed in the radio domain. However, the vast
majority of the data are unlabeled preventing the deployment of supervised
deep learning methods. Although this issue is common to many disciplines, it
is particularly critical in astronomy since the properties of the images
change depending on the instrument used. Therefore, a sample of labeled images
from one observational dataset cannot directly be exported to another survey,
hampering the use of deep learning. We notice however, that large dataset are
not always needed. Some works (e.g. Ackermann et al., 2018; Walmsley et al.,
2019) have demonstrated that reliable results can be obtained with small
training samples. This is probably a consequence of the relatively limited
complexity of astronomical images compared to natural images.

Nevertheless, reducing the amount of labels is usually desirable for most
supervised applications. There exist several types of solutions listed in
Table 2. An obvious one is to manually label more data. It is however very
time consuming and cannot be done for every new survey. Active learning
approaches allow to label only the more informative examples for the deep
learning model, hence reducing the needed time. Active learning has been
particularly explored in the framework of galaxy morphology (e.g. Walmsley et
al., 2020). Another fairly straightforward solution is Transfer Learning. This
is usually done by training a model in a similar dataset to the target dataset
for which labeled data exist. The neural network weights are then refined with
a small sample of labeled examples from the target sample. Transfer learning
has also been explored for galaxy morphology (e.g. Domínguez Sánchez et al.,
2019). Other works use simulations to overcome the lack of labels. As seen in
the previous section, using simulations is a fairly common approach in
astronomy. This is probably because the complexity of astronomical objects -
especially images - allows one to obtain quite realistic simulations with
fairly simple approaches. This is the case, for instance, for the
classification of strong lenses, which exclusively rely on simulated training
sets. Although this is a very efficient approach, it also implies some
potential issues related to the change of domain. This is a general challenge
which we have grouped into Challenge 4 in Table 2. Finally, the ML community
has recently started to explore the so-called self-supervised approaches to
reduce the amount of labeled examples. The underlying idea is to first compute
some meaningful representations of the data in an unsupervised way and then
use the obtained representations to train a supervised network. Because the
self-supervised step filters out non informative features, the amount of
needed examples for training is reduced. Hayat et al. (2021) and Sarmiento et
al. (2021) have recently explored self-supervised learning for galaxy
morphology, photometric redshifts and galaxy kinematics respectively.
Contrastive self-supervised learning can also be potentially employed to
reduce the gap between simulations and observations as well as Autoencoder
representations.

In addition to the size of the labeled datasets, a major challenge is how
representative are those. A common problem in astronomy is that the sample for
which labels are available does not always overlap with the inference
datasets. This typically could happen because it is easier to obtain labels
for a sub population of objects. The extrapolation of the trained neural
network results to a dataset which was not exactly used for training is
usually a problem. This issue has become very obvious for photometric redshift
estimation (see section 4) in which ML approaches tend to fail for examples
poorly represented in the training. The only solution adopted by the community
has been to obtain more training data.

####  7.3.2 Uncertainty

Uncertainty quantification is a major challenge for applications of deep
learning to physical sciences. This is a major difference with respect to
standard computer vision applications on natural imaging, which usually do not
require well calibrated uncertainties. Therefore, standard deep learning
methods directly exported to astronomy do not quantify uncertainties and this
is generally not acceptable for scientific applications. Some tasks such as
classification can sometimes be accepted without precise uncertainty and rely
on statistical measurements (i.e. ROC, Precision-Recall curves). However, if
deep learning is intended to be used for accurate measurements of galaxy
properties or to constrain models, they need to incorporate error
measurements. We see a changing trend in the community. The early efforts did
not include uncertainty estimation; however more and more works are
introducing at least some quantification of errors. The community has explored
several solutions, but the problem is far from being solved. A promising
approach are Bayesian Neural Networks (BNNs) which aim at measuring posterior
distributions from a Bayesian perspective. Several approaches have been
proposed over the past years and the implementation has also become more
straightforward, which favors the use by non-experts. For example BNNs are the
common approach for the modeling of strong lenses for example (e.g. Perreault
Levasseur et al., 2017). However, one need to keep in mind that BNNs compute
an approximation of the true posterior distributions which sometimes might not
be accurate enough (see review by Charnock et al. (2020)). Density Estimator
networks such as Regressive Flows or Mixture Density Networks are another
approach to sample from complex posterior distributions. Although several
works in astronomy pay careful attention to the quantification of
uncertainties, they represent still a minority of the deep learning literature
in astronomy. We believe this is a major challenge for the future deployment
of deep learning in the analysis of deep surveys.

####  7.3.3 Interpretability

Related to the problem of uncertainty quantification is interpretability. By
moving to a data driven approach to data analysis, we unavoidably loose some
control on what type of information is extracted and used by the neural
network models. This effect is sometimes referred as the black box effect.
Deep learning models are in general opaque black boxes which perform complex
non linear mappings, difficult to unveil. Although this might be a problem for
most applications, it is particularly worrying for scientific ones, and
therefore constitutes a major challenge for the acceptance of deep learning by
the astronomical community. For example, not properly understanding the
information used can generate some biases, as demonstrated by Dhar & Shamir
(2022) which show that deep learning based classifications are sensitive to
the location of the galaxy in the sky. The field of interpretability of deep
neural networks is even less developed than the one of uncertainty estimation
and in general the techniques employed provide a limited amount of
information. A common approach is to identify the regions of the input data
that provide most of the information for the network decisions. These methods
can be generally useful to identify biases - e.g. the neural network model
focuses on background noise - but are still far from providing any physical
interpretation of what is being measured. Some works have looked at ways of
enhancing the explainability (Huertas-Company et al., 2018; Bhambra et al.,
2022) but the amount of extracted information typically consists on the
identification of pixels in the input images which contribute most to the
decision. Although this is certainly valuable information to identify biases
in particular, it does not provide a true explainability in terms of physical
meaning. Interpreting the results is easier when the inputs are parameters
instead of raw pixel values. In such cases, there exists the possibility of
performing symbolic regressions to try to dig into the relations learned by
the neural networks (e.g. Cranmer et al., 2020; Villaescusa-Navarro et al.,
2022). An interesting research line to ease interpretability is the inclusion
of prior physical constraints in the neural network model. Architectures that
preserve known symmetries of the physical problem are for example an
interesting way to keep a control on what the networks are extracting (e.g.
Scaife & Porter, 2021; Villar et al., 2021a; Bowles et al., 2021).

####  7.3.4 Domain shift

Another key challenge faced by deep learning applications to astronomy is
related to the change of domains between the training and the inference steps.
As highlighted in Table 1, a majority of the applications of deep learning to
astronomy rely on simulations for training the models. This is typically
justified because the availability of labeled samples is limited (see
subsubsection 7.3.1), because we aim at accessing information that is only
available on simulations, e.g. galaxy mergers, dark matter or information
about the cosmological model or because the likelihood is intractable but we
have an idea on how to simulate the data (see review by Cranmer et al., 2019).
For example, the work by Bottrell et al. (2019) examines very carefully the
impact of using more or less realistic simulations for training. The community
has explored several solution to mitigate the impact of training on
simulations and apply to data. A simple approach is to use transfer learning.
This is only possible when there exist some measurements in observations which
can be used to fine tune the weights from the neural network model trained on
simulations (e.g. Tuccillo et al., 2018). This is not always possible though,
especially when we try to infer the parameters of a model and has also the
problem of propagating the biases of any existing method previously applied to
observations. Domain adaptation techniques are another alternative approach
which attempt the make the features learned by the model agnostic to the
differences between domains. As opposed to transfer learning, this is done
during training so that no domain specific features are learned. Ćiprijanović
et al. (2021a) have recently quantify the gain of such techniques for the
identification of galaxy mergers. It remains however an open issue for the
future.

####  7.3.5 Benchmarking and Deployment

A final challenge which has not been discussed much in the literature so far
is related to how the different approaches can be robustly compared. As we
have thoroughly described in this review, the past years have witnessed an
emergence of a large number of deep learning methods applied to a diversity of
scientific topics. In many cases, the results are shown for a specific
dataset, with a specific configuration, which makes it hard to compare with
existing approaches. The ML community has been using since many years, what is
called standardized datasets. These are common datasets which are publicly
shared and on which any new approach is usually tested. This benchmarking
approach has been an important channel for progress in the community. The
astronomical one is not used to this type of approach and therefore, with some
noticeable exceptions (e.g. PLAsTiCC, Galaxy Zoo for classification), we lack
of a coherent way of comparing methods. We argue that this is an important
aspect on which to work as a community to boost progress. Having standardized
datasets on which test models can not only help comparing methods but also
identify pitfalls and biases and therefore contribute to make the neural
network models more robust. This is an important step towards a full
deployment of these approaches into scientific pipelines.

Challenge 1 Small (and biased) labeled datasets |
---|---
Solution 1.A Transfer Learning |  Domínguez Sánchez et al. (2019) Samudre et al. (2022) Lukic et al. (2019)
Solution 1.B Simulated dataset |  Jacobs et al. (2017) Vega-Ferrero et al. (2021)
Solution 1.C Self-supervised learning |  Hayat et al. (2021)
Solution 1.D Active Learning and similar |  Walmsley et al. (2020)
Challenge 2 Uncertainty |
Solution 2.A Bayesian approximations |  Walmsley et al. (2020) Perreault Levasseur et al. (2017)
Solution 2.B Density Estimators |  Kodi Ramanah et al. (2020)
Challenge 3 Interpretability
Solution 3.A Saliency maps and similar |  Huertas-Company et al. (2018); Bowles et al. (2021); Bhambra et al. (2022)
Solution 3.B Symbolic regression |  Cranmer et al. (2020)
Solution 3.C Physics informed |  Scaife & Porter (2021); Villar et al. (2021a); Charnock et al. (2019)
Challenge 4 Domain shift
Solution 4.A Transfer Learning |  Tuccillo et al. (2018); Domínguez Sánchez et al. (2019); Ghosh et al. (2020)
Solution 4.A Domain Adaptation |  Ćiprijanović et al. (2021b)
Challenge 5 Benchmarking
Solution 5.A Standardized datasets |  PLAsTiCC, SKA data challenge, Galaxy Zoo
Table 2: Major challenges that deep learning works applied to astronomy might
suffer and that will need to be addressed in the coming years. We also provide
elements of solutions already being explored along with the corresponding
references.

##  8 Summary

This work reviews the use of modern deep learning techniques for the analysis
of deep galaxy surveys. Although machine learning has been used in astronomy
for several decades, the recent deep learning revolution as induced an
unprecedented number of new works exploring the use of these novel techniques
in astronomy. The purpose of this review is to assess how deep learning has
been used in astronomy and what are the key achievements and challenges. We do
not describe however the technical aspects of deep learning techniques.

We have divided deep learning applications in four broad categories defined by
the type of application: 1- computer vision, 2 - galaxy properties, 3 -
discovery and 4 - cosmology. The first sections of these review (section 3 to
section 6) describe the most relevant works in each category. A summary of the
main points for each type of application is included at the end of the
corresponding section. The first category (section 3) includes general
computer vision applications such as classifications and object detection. The
second (section 4) is related to measure galaxy properties, deep learning acts
as a fast emulator and as universal aproximator. The third category (section
5) illustrates all efforts related to visualization and identification on new
types of objects. The fourth group (section 6) contains publications which use
deep learning for cosmology. Namely we include two main applications: more
efficient simulation and cosmological inference.

The last section (section 7) focuses on extracting some lessons about the use
of deep learning techniques in astronomy, on the impact they have had so far
and on what are - in our humble opinion - the key challenges that will need to
be addressed in the near future. We list below key take away messages from
this analysis:

  * •

The first work using deep learning in astronomy is from 2015. Since then, the
number of works using deep learning for galaxy surveys has increased
exponentially. There is factor of $\sim 15$ increase between 2015 and 2021.

  * •

The most common deep learning method used are sequential Convolutional Neural
Networks with different degrees of complexities. However, there is a good
variety of techniques which have been tested for astronomy including recent
developments such as Transformers or self-supervised approaches. This reflects
a _democratization_ of these techniques which are becoming increasingly easy
to use. However, the methods are often applied with a limited amount of
physically driven modifications. The combination of previous physical
knowledge with data driven models is still an open issue, even if it is a
rapidly changing field.

  * •

The majority of the works ($>50\%$) focus on what we call computer vision
applications - which essentially include classification and segmentation. This
is also the field in which deep learning has brought the most important
breakthroughs. These are applications which are more prone to a direct import
from the ML community. However, we measure a diversification of the
applications which span a variety of topics such as the acceleration of
cosmological simulations, the inference of galaxy properties or constraints on
cosmology cosmology.

  * •

The works using deep learning represent $\sim 5\%$ of all works on galaxy
formation which is remarkable. The receive however $\sim 3$ citations per
paper and per year on average. This is roughly $\sim 1.5$ times less citations
than publications on galaxies for example, although computer vision
applications also perform better in this front. It suggests a moderate impact
of deep learning so far which might be explained because most of the works are
still at the exploratory stage.

  * •

We have identified a set of 5 major challenges which frequently appear in deep
learning applications and that we believe need to be addresses in the nearby
future. 1- Small labeled datasets; 2 - Uncertainty estimation; 3-
Interpretability; 4 - Domain shift and 5 - Benchmarking.

##  Appendix A Acronyms

We summarize in this appendix the acronyms used for designating types of deep
learning methods, galaxy surveys as well as simulated datasets. For every
method we also indicate a reference where more details can be obtained.

Machine Learning:

  * •

ANN: Artificial Neural Network

  * •

ARF: Auto Regressive Flow - Papamakarios et al. (2017)

  * •

BNN: Bayesian Neural Network - Charnock et al. (2020); Goan & Fookes (2020)

  * •

CAE: Convolutional Autoencoder

  * •

CNN: Convolutional Neural Network

  * •

DT: Decision Tree

  * •

(W)GAN: (Wasserstein) Generative Adversarial Network - Goodfellow et al.
(2014); Arjovsky et al. (2017)

  * •

GNN: Graph Neural Network

  * •

Mask R-CNN: Mask Region Convolutional Neural Network - He et al. (2017)

  * •

MLP: Multi-Layer Perceptron

  * •

MDN: Mixture Density Network - Bishop (1994)

  * •

RF: Random Forest

  * •

RNN: Recursive Neural Network

  * •

SOM: Self-Organizing Map

  * •

SVM: Support Vector Machines

  * •

VAE: Variational Autoencoder - Pu et al. (2016)

  * •

YOLO: You Only Look Once - Redmon et al. (2015)

Deep galaxy surveys where deep neural networks have been applied:

  * •

CANDELS: Cosmic Assembly Near-Infrared Deep Extragalactic Legacy Survey;
Koekemoer et al. (2011)

  * •

DECALS: The Dark Energy Camera Legacy Survey; Dey et al. (2019)

  * •

DES: The Dark Energy Survey; Dark Energy Survey Collaboration et al. (2016)

  * •

Euclid:  Laureijs et al. (2011)

  * •

HSC:  Hyper Suprime Cam Aihara et al. (2018)

  * •

MANGA:  Mapping Nearby Galaxies at APO; Bundy et al. (2015)

  * •

Pan-STARRS: Panoramic Survey Telescope and Rapid Response System; Chambers et
al. (2016)

  * •

PAU:  Physics of the Accelerating Universe;

  * •

SDSS: Sloan Digital Sky Survey

  * •

LSST:  Legacy Survey of Space and Time; Ivezić et al. (2019)

  * •

S-PLUS: Southern Photometric Local Universe Survey; Mendes de Oliveira et al.
(2019)

  * •

GAMA:  Galaxy and Mass Assembly; Driver et al. (2011)

  * •

ZTF:  Zwicky Transient Facility; Bellm (2014)

Simulated datasets used to train deep neural networks:

  * •

CAMELS: Villaescusa-Navarro et al. (2021b)

  * •

PLAsTiCC: Photometric LSST Astronomical Time-Series Classification Challenge

  * •

IllustrisTNG: Pillepich et al. (2018)

  * •

EAGLE: Schaye et al. (2015)

  * •

SIMBA:Davé et al. (2019)

  * •

VELA:Ceverino et al. (2015)

  * •

Horizon-AGN: Dubois et al. (2014)

## References

  * Ackermann et al. (2018) Ackermann S., Schawinski K., Zhang C., Weigel A. K., Turp M. D., 2018, MNRAS, 479, 415
  * Aihara et al. (2018) Aihara H., et al., 2018, Publications of the Astronomical Society of Japan, 70, S4
  * Alexander et al. (2020) Alexander S., Gleyzer S., McDonough E., Toomey M. W., Usai E., 2020, The Astrophysical Journal, 893, 15
  * Alhassan et al. (2018) Alhassan W., Taylor A. R., Vaccari M., 2018, Monthly Notices of the Royal Astronomical Society, 480, 2085
  * Allam & McEwen (2021) Allam Jr. T., McEwen J. D., 2021, Technical report, Paying Attention to Astronomical Transients: Photometric Classification with the Time-Series Transformer, https://ui.adsabs.harvard.edu/abs/2021arXiv210506178A. https://ui.adsabs.harvard.edu/abs/2021arXiv210506178A
  * Alsing et al. (2019) Alsing J., Charnock T., Feeney S., Wandelt B., 2019, Monthly Notices of the Royal Astronomical Society, 488, 4440
  * Andreon et al. (2000) Andreon S., Gargiulo G., Longo G., Tagliaferri R., Capuano N., 2000, Monthly Notices of the Royal Astronomical Society, 319, 700
  * Aniyan & Thorat (2017) Aniyan A. K., Thorat K., 2017, The Astrophysical Journal Supplement Series, 230, 20
  * Aragon-Calvo & Carvajal (2020) Aragon-Calvo M. A., Carvajal J. C., 2020, Monthly Notices of the Royal Astronomical Society, 498, 3713
  * Arcelin et al. (2021) Arcelin B., Doux C., Aubourg E., Roucelle C., Collaboration L. D. E. S., 2021, MNRAS, 500, 531
  * Arjovsky et al. (2017) Arjovsky M., Chintala S., Bottou L., 2017, Technical report, Wasserstein GAN, https://ui.adsabs.harvard.edu/abs/2017arXiv170107875A. https://ui.adsabs.harvard.edu/abs/2017arXiv170107875A
  * Armitage et al. (2019) Armitage T. J., Kay S. T., Barnes D. J., 2019, Monthly Notices of the Royal Astronomical Society, 484, 1526
  * Ball & Brunner (2010) Ball N. M., Brunner R. J., 2010, International Journal of Modern Physics D, 19, 1049
  * Ball et al. (2004) Ball N. M., Loveday J., Fukugita M., Nakamura O., Okamura S., Brinkmann J., Brunner R. J., 2004, Monthly Notices of the Royal Astronomical Society, 348, 1038
  * Ball et al. (2006) Ball N. M., Brunner R. J., Myers A. D., Tcheng D., 2006, The Astrophysical Journal, 650, 497
  * Banerji et al. (2008) Banerji M., Abdalla F. B., Lahav O., Lin H., 2008, Monthly Notices of the Royal Astronomical Society, 386, 1219
  * Banfield et al. (2015) Banfield J. K., et al., 2015, Monthly Notices of the Royal Astronomical Society, 453, 2326
  * Baron (2019) Baron D., 2019, Technical report, Machine Learning in Astronomy: a practical overview, https://ui.adsabs.harvard.edu/abs/2019arXiv190407248B. https://ui.adsabs.harvard.edu/abs/2019arXiv190407248B
  * Baron & Ménard (2021) Baron D., Ménard B., 2021, The Astrophysical Journal, 916, 91
  * Baron & Poznanski (2017) Baron D., Poznanski D., 2017, Monthly Notices of the Royal Astronomical Society, 465, 4530
  * Bazell & Peng (1998) Bazell D., Peng Y., 1998, The Astrophysical Journal Supplement Series, 116, 47
  * Becker et al. (2021) Becker B., Vaccari M., Prescott M., Grobler T., 2021, Monthly Notices of the Royal Astronomical Society, 503, 1828
  * Bellm (2014) Bellm E., 2014, The Zwicky Transient Facility. eprint: arXiv:1410.8185, https://ui.adsabs.harvard.edu/abs/2014htu..conf...27B
  * Benítez (2000) Benítez N., 2000, The Astrophysical Journal, 536, 571
  * Berger & Stein (2018) Berger P., Stein G., 2018, Monthly Notices of the Royal Astronomical Society, Volume 482, Issue 3, p.2861-2871, 2019
  * Bernardini et al. (2019) Bernardini M., Mayer L., Reed D., Feldmann R., 2019, ] 10.1093/mnras/staa1911
  * Bernardini et al. (2021) Bernardini M., Feldmann R., Anglés-Alcázar D., Boylan-Kolchin M., Bullock J., Mayer L., Stadel J., 2021, ] 10.1093/mnras/stab3088
  * Bertin & Arnouts (1996) Bertin E., Arnouts S., 1996, A&AS, 117, 393
  * Bhambra et al. (2022) Bhambra P., Joachimi B., Lahav O., 2022, MNRAS, 511, 5032
  * Bickley et al. (2021) Bickley R. W., et al., 2021, Monthly Notices of the Royal Astronomical Society, 504, 372
  * Bishop (1994) Bishop C. M., 1994, Mixture density networks
  * Bluck et al. (2022) Bluck A. F. L., Maiolino R., Brownson S., Conselice C. J., Ellison S. L., Piotrowska J. M., Thorp M. D., 2022, Technical report, The quenching of galaxies, bulges, and disks since cosmic noon: A machine learning approach for identifying causality in astronomical data, https://ui.adsabs.harvard.edu/abs/2022arXiv220107814B. https://ui.adsabs.harvard.edu/abs/2022arXiv220107814B
  * Bolzonella et al. (2000) Bolzonella M., Miralles J. M., Pelló R., 2000, A&A, 363, 476
  * Bom et al. (2019) Bom C., Poh J., Nord B., Blanco-Valentin M., Dias L., 2019, Technical report, Deep Learning in Wide-field Surveys: Fast Analysis of Strong Lenses in Ground-based Cosmic Experiments, %****␣dl_review.bbl␣Line␣200␣****https://ui.adsabs.harvard.edu/abs/2019arXiv191106341B. https://ui.adsabs.harvard.edu/abs/2019arXiv191106341B
  * Bom et al. (2021) Bom C. R., et al., 2021, Monthly Notices of the Royal Astronomical Society, 507, 1937
  * Bonnett (2015) Bonnett C., 2015, Monthly Notices of the Royal Astronomical Society, 449, 1043
  * Boone (2021) Boone K., 2021, The Astronomical Journal, 162, 275
  * Bottrell et al. (2019) Bottrell C., et al., 2019, Monthly Notices of the Royal Astronomical Society, 490, 5390
  * Bottrell et al. (2021) Bottrell C., Hani M. H., Teimoorinia H., Patton D. R., Ellison S. L., 2021, Monthly Notices of the Royal Astronomical Society
  * Boucaud et al. (2020) Boucaud A., et al., 2020, Monthly Notices of the Royal Astronomical Society, 491, 2481
  * Bowles et al. (2021) Bowles M., Bromley M., Allen M., Scaife A., 2021, arXiv e-prints, p. arXiv:2111.04742
  * Brehmer et al. (2019) Brehmer J., Mishra-Sharma S., Hermans J., Louppe G., Cranmer K., 2019, The Astrophysical Journal, 886, 49
  * Bretonnière et al. (2021) Bretonnière H., Boucaud A., Huertas-Company M., 2021, arXiv e-prints, p. arXiv:2111.15455
  * Bretonnière et al. (2022) Bretonnière H., et al., 2022, A&A, 657, A90
  * Buck & Wolf (2021) Buck T., Wolf S., 2021, Technical report, Predicting resolved galaxy properties from photometric images using convolutional neural networks, https://ui.adsabs.harvard.edu/abs/2021arXiv211101154B. https://ui.adsabs.harvard.edu/abs/2021arXiv211101154B
  * Bundy et al. (2015) Bundy K., et al., 2015, The Astrophysical Journal, 798, 7
  * Burhanudin et al. (2021) Burhanudin U. F., et al., 2021, Monthly Notices of the Royal Astronomical Society, 505, 4345
  * Burke et al. (2019) Burke C. J., Aleo P. D., Chen Y.-C., Liu X., Peterson J. R., Sembroski G. H., Lin J. Y.-Y., 2019, MNRAS, 490, 3952
  * Cabayol-Garcia et al. (2020) Cabayol-Garcia L., et al., 2020, Monthly Notices of the Royal Astronomical Society, 491, 5392
  * Cabayol et al. (2021) Cabayol L., et al., 2021, Monthly Notices of the Royal Astronomical Society, 506, 4048
  * Cai et al. (2020) Cai M. X., Bédorf J., Saletore V. A., Codreanu V., Podareanu D., Chaibi A., Qian P. X., 2020, Technical report, DeepGalaxy: Deducing the Properties of Galaxy Mergers from Images Using Deep Neural Networks, https://ui.adsabs.harvard.edu/abs/2020arXiv201011630C. https://ui.adsabs.harvard.edu/abs/2020arXiv201011630C
  * Calderon & Berlind (2019) Calderon V. F., Berlind A. A., 2019, Monthly Notices of the Royal Astronomical Society, 490, 2367
  * Campagne (2020) Campagne J.-E., 2020, Technical report, Adversarial training applied to Convolutional Neural Network for photometric redshift predictions, https://ui.adsabs.harvard.edu/abs/2020arXiv200210154C. https://ui.adsabs.harvard.edu/abs/2020arXiv200210154C
  * Carrasco-Davis et al. (2019) Carrasco-Davis R., et al., 2019, Publications of the Astronomical Society of the Pacific, 131, 108006
  * Cavanagh et al. (2021) Cavanagh M. K., Bekki K., Groves B. A., 2021, Monthly Notices of the Royal Astronomical Society, 506, 659
  * Ceverino et al. (2015) Ceverino D., Dekel A., Tweed D., Primack J., 2015, Monthly Notices of the Royal Astronomical Society, 447, 3291
  * Chalapathy & Chawla (2019) Chalapathy R., Chawla S., 2019, Technical report, Deep Learning for Anomaly Detection: A Survey, https://ui.adsabs.harvard.edu/abs/2019arXiv190103407C. https://ui.adsabs.harvard.edu/abs/2019arXiv190103407C
  * Chambers et al. (2016) Chambers K. C., et al., 2016, Technical report, The Pan-STARRS1 Surveys, https://ui.adsabs.harvard.edu/abs/2016arXiv161205560C. https://ui.adsabs.harvard.edu/abs/2016arXiv161205560C
  * Charnock & Moss (2017) Charnock T., Moss A., 2017, The Astrophysical Journal, 837, L28
  * Charnock et al. (2019) Charnock T., Lavaux G., Wandelt B. D., Boruah S. S., Jasche J., Hudson M. J., 2019, ] 10.1093/mnras/staa682
  * Charnock et al. (2020) Charnock T., Perreault-Levasseur L., Lanusse F., 2020, Technical report, Bayesian Neural Networks, https://ui.adsabs.harvard.edu/abs/2020arXiv200601490C. https://ui.adsabs.harvard.edu/abs/2020arXiv200601490C
  * Chen et al. (2020) Chen T., Kornblith S., Norouzi M., Hinton G., 2020, Technical report, A Simple Framework for Contrastive Learning of Visual Representations, https://ui.adsabs.harvard.edu/abs/2020arXiv200205709C. https://ui.adsabs.harvard.edu/abs/2020arXiv200205709C
  * Cheng et al. (2021b) Cheng T.-Y., et al., 2021b, Monthly Notices of the Royal Astronomical Society, 507, 4425
  * Chianese et al. (2020) Chianese M., Coogan A., Hofma P., Otten S., Weniger C., 2020, Monthly Notices of the Royal Astronomical Society, 496, 381
  * Cohen et al. (2003) Cohen S. H., Windhorst R. A., Odewahn S. C., Chiarenza C. A., Driver S. P., 2003, The Astronomical Journal, 125, 1762
  * Collister & Lahav (2004) Collister A. A., Lahav O., 2004, Publications of the Astronomical Society of the Pacific, 116, 345
  * Conselice (2003) Conselice C. J., 2003, The Astrophysical Journal Supplement Series, 147, 1
  * Coogan et al. (2020) Coogan A., Karchev K., Weniger C., 2020, Technical report, Targeted Likelihood-Free Inference of Dark Matter Substructure in Strongly-Lensed Galaxies, https://ui.adsabs.harvard.edu/abs/2020arXiv201007032C. https://ui.adsabs.harvard.edu/abs/2020arXiv201007032C
  * Cranmer et al. (2019) Cranmer K., Brehmer J., Louppe G., 2019, Technical report, The frontier of simulation-based inference, https://ui.adsabs.harvard.edu/abs/2019arXiv191101429C. https://ui.adsabs.harvard.edu/abs/2019arXiv191101429C
  * Cranmer et al. (2020) Cranmer M., Sanchez-Gonzalez A., Battaglia P., Xu R., Cranmer K., Spergel D., Ho S., 2020, Technical report, Discovering Symbolic Models from Deep Learning with Inductive Biases, https://ui.adsabs.harvard.edu/abs/2020arXiv200611287C. https://ui.adsabs.harvard.edu/abs/2020arXiv200611287C
  * D’Abrusco et al. (2007) D’Abrusco R., Staiano A., Longo G., Brescia M., Paolillo M., De Filippis E., Tagliaferri R., 2007, The Astrophysical Journal, 663, 752
  * D’Isanto & Polsterer (2018) D’Isanto A., Polsterer K. L., 2018, A&A, 609, A111
  * Dai & Seljak (2020) Dai B., Seljak U., 2020, ] 10.1073/pnas.2020324118
  * Dai & Seljak (2022) Dai B., Seljak U., 2022, Technical report, Translation and Rotation Equivariant Normalizing Flow (TRENF) for Optimal Cosmological Analysis, https://ui.adsabs.harvard.edu/abs/2022arXiv220205282D. https://ui.adsabs.harvard.edu/abs/2022arXiv220205282D
  * Dai et al. (2018) Dai B., Feng Y., Seljak U., 2018, Journal of Cosmology and Astroparticle Physics, 2018, 009
  * Dark Energy Survey Collaboration et al. (2016) Dark Energy Survey Collaboration et al., 2016, Monthly Notices of the Royal Astronomical Society, 460, 1270
  * Davé et al. (2019) Davé R., Anglés-Alcázar D., Narayanan D., Li Q., Rafieferantsoa M. H., Appleby S., 2019, MNRAS, 486, 2827
  * Davidzon et al. (2019) Davidzon I., et al., 2019, Monthly Notices of the Royal Astronomical Society, 489, 4817
  * Davies et al. (2019) Davies A., Serjeant S., Bromley J. M., 2019, MNRAS, 487, 5263
  * Dey et al. (2019) Dey A., et al., 2019, The Astronomical Journal, 157, 168
  * Dey et al. (2021) Dey B., Andrews B. H., Newman J. A., Mao Y.-Y., Rau M. M., Zhou R., 2021, Technical report, Photometric Redshifts from SDSS Images with an Interpretable Deep Capsule Network, https://ui.adsabs.harvard.edu/abs/2021arXiv211203939D. https://ui.adsabs.harvard.edu/abs/2021arXiv211203939D
  * Dhar & Shamir (2022) Dhar S., Shamir L., 2022, Technical report, Systematic biases when using deep neural networks for annotating large catalogs of astronomical images, https://ui.adsabs.harvard.edu/abs/2022arXiv220103131D. https://ui.adsabs.harvard.edu/abs/2022arXiv220103131D
  * Diaz et al. (2019) Diaz J. D., Bekki K., Forbes D. A., Couch W. J., Drinkwater M. J., Deeley S., 2019, Monthly Notices of the Royal Astronomical Society, 486, 4845
  * Dieleman et al. (2015) Dieleman S., Willett K. W., Dambre J., 2015, MNRAS, 450, 1441