PASA 2024

# The Dawes Review 10: The impact of deep learning for the analysis of galaxy
surveys

M. Huertas-Company1,2,3 and F. Lanusse2
1Instituto de Astrofísica de Canarias, c/ Vía Láctea sn, 38025, La Laguna,
Spain  2Universidad de La Laguna. Avda. Astrofísico Fco. Sanchez, La Laguna,
Tenerife,Spain  3LERMA, Observatoire de Paris, CNRS, PSL, Université Paris-
Cité, France  4AIM, CEA, CNRS, Université Paris-Saclay, Université Paris-Cité,
Sorbonne Paris Cité, F-91191 Gif-sur-Yvette, France

###### Abstract

The amount and complexity of data delivered by modern galaxy surveys has been
steadily increasing over the past years. New facilities will soon provide
imaging and spectra of hundreds of millions of galaxies. Extracting coherent
scientific information from these large and multi-modal data sets remains an
open issue for the community and data driven approaches such as deep learning
have rapidly emerged as a potentially powerful solution to some long lasting
challenges. This enthusiasm is reflected in an unprecedented exponential
growth of publications using neural networks, which have gone from a handful
of works in 2015 to an average of one paper per week in 2021 in the area of
galaxy surveys. Half a decade after the first published work in astronomy
mentioning deep learning, and shortly before new big-data sets such as Euclid
and LSST start becoming available, we believe it is timely to review what has
been the real impact of this new technology in the field and its potential to
solve key challenges raised by the size and complexity of the new datasets.
The purpose of this review is thus two-fold. We first aim at summarizing, in a
common document, the main applications of deep learning for galaxy surveys
that have emerged so far. We then extract the major achievements and lessons
learned and highlight key open questions and limitations, which in our
opinion, will require particular attention in the coming years. Overall,
state-of-the art deep learning methods are rapidly adopted by the astronomical
community, reflecting a democratization of these methods. This review shows
that the majority of works using deep learning up to date are oriented to
computer vision tasks (e.g. classification, segmentation). This is also the
domain of application where deep learning has brought the most important
breakthroughs so far. However, we also report that the applications are
becoming more diverse and deep learning is used for estimating galaxy
properties, identifying outliers or constraining the cosmological model. Most
of these works remain at the exploratory level though which could partially
explain the limited impact in terms of citations. Some common challenges will
most likely need to be addressed before moving to the next phase of massive
deployment of deep learning in the processing of future surveys; e.g.
uncertainty quantification, interpretability, data labeling and domain shift
issues from training with simulations, which constitutes a common practice in
astronomy.

###### doi:

10.1017/pas.2024.xxx

###### keywords:

keyword1 – keyword2 – keyword3 – keyword4 – keyword5

##  1 Introduction

Most fields in astronomy are rapidly changing. Unprecedentedly large
observational data exists or will soon become available. Modern spectro-
photometric surveys such as the Legacy Survey of Space and Time (LSST; Ivezić
et al., 2019) or Euclid (Laureijs et al., 2011) will provide high quality
spectra and images for hundreds of millions of galaxies. Integral field
spectroscopic surveys at low and high redshift are reaching statistically
relevant sizes (e.g. MaNGA - Bundy et al., 2015) enabling to resolve the
internal structure of galaxies beyond integrated properties. In addition, new
facilities like the James Webb Space Telescope (JWST) are opening the window
to a completely new redshift and stellar mass regime both in imaging and
spectroscopy and we will be able to witness the emergence of the first
galaxies in the universe. X-ray and radio facilities (e.g. SKA, Athena) will
probe cold and hot gas in galaxies with improved resolution. On the theory
side, computing power has evolved to the extent that we can now generate
realistic simulations of galaxies in a cosmological context spanning most of
the Universe’s history (e.g. TNG - Pillepich et al., 2018) which properly
reproduce a large number of observable properties. In this context of growing
complexity and rapid increase of data volumes, it has become a new challenge
for the community to combine and accurately extract scientifically relevant
information from these datasets.

Although Machine Learning applications to astronomy exist since at least
thirty years ago (see section 2), the past years have witnessed an
unprecedented increase of deep learning methods translated on an exponential
increase of publications (Figure 1). This _revival_ is fueled by significant
breakthroughs in the field of Machine Learning since the popularization of
Convolutional Neural Networks (CNNs) a decade ago (Krizhevsky et al., 2012).
The first published work mentioning deep learning in astronomy is from 2015 in
which CNNs were applied for the classification of galaxy morphology. Since
then, the number of works using deep learning in astrophysics has been growing
exponentially, being the fastest growth of other topics in the field (Figure
1). The generalization of deep learning represents to some extent a change of
paradigm in the way we approach data analysis. By using gradient-based
optimization techniques to extract meaningful features directly from the data,
we move from an approach based on specific algorithms and features to a fully
data driven one. It has potentially profound implications for astronomy and
science in general.

Figure 1: Relative change of the number of papers on arXiv:astro-ph with
different keywords in the abstract as a function of time. The number of works
mentioning neural networks in the abstract has experienced an unprecedented
growth in the last $\sim 6$ years, significantly steeper than other topic in
astrophysics. Source: ArXivSorter

After half a decade of this new wave of applications of deep learning to
astrophysics, we thus think it is timely to look back at the impact that this
new technology has had in our field. Given the large amount of existing
publications, we will only review works focusing on the analysis of galaxy
surveys and we will restrict to recent works after the so called deep learning
boom. We believe however that most of the lessons learned can be extrapolated
to other areas of astrophysics.

It is not the goal of this review to provide technical details about how deep
learning techniques work, but to describe its applications to cosmology and
galaxy formation in a unique reference document. Over the past years, deep
learning has been used for a large variety of tasks such as classification,
object detection, but also to derive physical properties of galaxies such as
photometric redshifts, to identify anomalous objects or to accelerate
simulations and constrain cosmology among many others. The review is thus
structured following major areas of application. We provide a quick
description and some keywords about the technical solutions adopted for each
science case but we emphasize it is not the goal of this work to focus on
technical aspects. The reader is encouraged to read the publications referring
to each method - which are provided in a best effort basis - to obtain a
complete and formal description of the deep learning methods. For completeness
and easy access, we provide in the Appendix A a list of different methods
mentioned in the review with the corresponding references. A list of current
and future galaxy surveys which are referred in this work is also provided in
Appendix A.

Following this idea, we have divided the applications of deep learning into
four major categories:

  * •

Deep learning for general computer vision tasks. These are applications that
we consider closest to standard computer vision applications for natural
images for which deep learning has been shown to generally outperform other
traditional approaches. It typically includes classification and segmentation
tasks.

  * •

Deep learning to derive physical properties of galaxies (both posteriors and
point estimates). These are applications in which deep learning is used to
estimate galaxy properties such as photometric redshifts or stellar
populations properties. Neural Networks are typically used to replace existing
algorithms with a faster and more efficient solution, hence more suited for
large data volumes. In addition, we also review applications in which deep
learning is employed to derive properties of galaxies which are not directly
accessible with known observables, i.e. to find new relations between
observable quantities and physical properties of galaxies from simulations.

  * •

Deep learning for assisted discovery. Neural networks are used here for data
exploration and visualization of complex datasets in lower dimension. We
include also in this category, efforts to automatically identify potentially
interesting new objects, i.e. anomalies or outliers.

  * •

Deep learning for cosmology. Cosmological simulations including baryonic
physics are computationally expensive. Deep learning can be used as a fast
emulator of the galaxy-halo connection by populating dark matter halos. In
addition, a second major application is cosmological inference. Cosmological
models are traditionally constrained using summary statistics (e.g. 2 point
statistics). Deep learning has been used to bypass these summary statistics
and constrain models using all available data.

This is of course a subjective division of the applications of deep learning
to cosmology and galaxy formation. There necessarily exist overlaps between
the different categories. The review is organized such that, for each family
of applications, we review the state of the art and key publications,
highlight where the limitations are and what could be the most promising
research lines for the future (section 3 to section 6). In the final sections
(section 7), we assess the impact of deep learning for galaxy surveys and
extract some global lessons learned. We have tried to provide a fair and
complete description of the different works. However, as previously stated,
the field has exploded in the last years and it has become more and more
difficult to keep track of all new publications. This is partly one of the
motivations for this review. It is also implies that we might easily miss some
relevant works. We apologize in advance.

##  2 A very brief historical overview - or what we are not covering in this
review

Before we start discussing the most relevant results and applications, we
would like to clarify that this review focuses on recent applications of deep
learning, essentially after the first applications of CNNs to astronomy. As
described in the introduction section, we consider as _deep learning_ all
recent developments around neural networks which have arisen in the last
decade approximately, since the first practical application of convolutional
neural networks for image classification. Deep learning generally designates
gradient-based optimization techniques of modular architectures of varying
complexity; it is therefore a sub field of the more general machine learning
discipline. There is a long history of machine learning applications in
astronomy which started since well before the more recent deep learning boom.
Different types of machine learning algorithms including early Artificial
Neural Networks (ANNs), Decision Trees (DTs), Random Forests (RFs) or kernel
algorithms such as Support Vector Machines (SVMs) have been applied to
different areas of astrophysics since the second half of the past century. For
example ANNs, decision trees and Self Organizing Maps (SOMs) have been
extensively applied to the classification of stars and galaxies (e.g. Odewahn
et al., 1992; Weir et al., 1995; Miller & Coe, 1996; Bazell & Peng, 1998;
Andreon et al., 2000; Qin et al., 2003; Ball et al., 2006); an ANN being the
primary way of identifying point sources in the commonly used SExtractor
software Bertin & Arnouts (1996) for segmentation of astronomical images. The
problem of galaxy morphology classification has also been subject to a
significant amount of machine learning related works led in particular by the
group of O. Lahav and collaborators using ANNs and DTs (e.g. Storrie-Lombardi
et al., 1992; Lahav et al., 1995, 1996; Odewahn et al., 1996; Naim et al.,
1997; Madgwick, 2003; Cohen et al., 2003). Ball et al. (2004) is likely the
first work to use ANNs to classify galaxies in the SDSS. In the first decade
of the present century SVMs became more popular and were also used to provide
catalogs of galaxy morphology (e.g. Huertas-Company et al., 2008, 2011).
Decision Trees have also been applied to other classification tasks such as
AGN/galaxy separation (e.g. White et al., 2000; Gao et al., 2008). Beyond
classification, machine learning, and especially ANNs have been extensively
applied to the problem of estimating photometric redshifts (e.g. D’Abrusco et
al., 2007; Li et al., 2007; Banerji et al., 2008). This review will not
describe these works though. We refer the reader to Ball & Brunner (2010) and
Baron (2019) for a complete and extensive review of pre-deep learning machine
learning techniques applied to astronomy. This obviously does not mean that
other machine learning approaches are less interesting for astrophysics. There
have been recently very relevant applications of RFs for example for anomaly
detection (see the works by Baron & Poznanski, 2017) and to assess the main
causes of star formation quenching in galaxies (e.g. Bluck et al., 2022) among
many others. However, we have made the choice not to include a detailed
description of these works in this review. We will focus essentially on how
deep learning has changed the landscape in the past half decade.

What is different with deep learning and why a dedicated review? In many
aspects, deep learning represents a change in the way we approach data
analysis. Because, we now have access to large datasets and the computing
resources are powerful enough - especially thanks to Graphic Processing Units
- we can move from an algorithmic-centered approach relying on manually
engineered features to a fully data-driven unsupervised feature learning
approach. This implies that instead of developing advanced domain specific
algorithms for each task, we rely on a generic optimization algorithm to
extract the most meaningful features in an end-to-end training loop. This is a
new approach to data in astrophysics and in science in general. This change of
paradigm has enabled in fact tremendous progress in the computer vision
community, especially for image classification, but also for many other tasks
such as translation, speech recognition or image segmentation over the past
ten years. The purpose of this review is therefore to assess what has been the
impact so far of this new approach for data processing in the fields of galaxy
formation and cosmology.

##  3 Deep learning for computer vision tasks in astronomy

We begin by reviewing applications close to standard computer vision problems,
for which deep learning approaches have been demonstrated to be very
efficient. We focus on classification and source detection.

###  3.1 Classification

Source classification is a basic first order processing step in most deep
surveys, for which deep learning has had a noticeable impact in the past
years. The rapid penetration of deep learning can be naturally explained
because it is arguably the most straightforward out-of-the box application.
Deep learning started in fact to attract the attention of the computer vision
community, when convolutional neural networks first won the ImageNet contest
of image classification (Krizhevsky et al., 2012).

One of the first tasks scientists do when confronted with a complex problem is
to identify objects that look morphologically similar. In extra galactic
astronomy, object classification can be of different flavours. For imaging,
the most common applications which we review in the remaining of this section,
are galaxy morphology classification, star-galaxy separation, and strong
lenses detection. We understand this is not an exhaustive list of all image
classification applications, however the techniques and approaches used are
representative. From the spectroscopic point of view, there have been some
works attempting to classify galaxy spectra. However, this remains less common
than images. Finally, the classification of transients is something which has
been extensively explored over the last years, especially in view of the LSST
survey from the Rubin Observatory.

####  3.1.1 Optical/NIR galaxy morphology

Galaxy morphological classification is a paradigmatic example of a science
case where deep learning has rapidly become the state-of-the art. This task
was first done by E. Hubble who classified galaxies in the well known Hubble
sequence. The classification scheme, which is now more than a 100 years old,
establishes that galaxies in the Universe today come essentially in two
flavors. On one side, there are disk galaxies, like our own Milky Way; on the
other side elliptical like galaxies. We now know that, besides morphology, the
broad classes present different physical properties. Understanding the origin
of the morphological diversity remains an open issue in the field of galaxy
formation.

Therefore, galaxy morphological classification is still performed in almost
all extra galactic imaging surveys. The traditional way to estimate galaxy
morphology has been through visual inspection. However, this approach became
prohibitively time consuming in the last decade with the advent of large extra
galactic imaging surveys such as the Sloan Digital Sky Survey (SDSS).
Approaches to overcome this limitation came in two fronts essentially. Citizen
science approaches, of which the Galaxy Zoo project is the more popular
example (Lintott et al., 2008), were developed to classify large samples of
galaxies. Automation through Machine Learning has been always on the table as
early as 1990’s (e.g. Spiekermann, 1992) and continued in the 2000’s (e.g.
Huertas-Company et al., 2008). Huertas-Company et al. (2011) was the first to
provide the community with a Machine Learning based classification of SDSS
galaxies. However the accuracy reached by these early approaches based on
manually engineered features remained moderate - especially when dealing with
detailed morphological features such as bars or spiral arms - hampering their
penetration in the community. The main limitation is that the features
typically used by these methods present only weak correlations with detailed
morphological structures and are also very dependent on noise and spatial
resolution.

In this context, it is not a surprise that the first works using Convolutional
Neural Networks in astronomy focus on galaxy morphology (Dieleman et al.,
2015; Huertas-Company et al., 2015). The first one used labeled images from
the Galaxy Zoo2 sample and trained a supervised CNN to estimate the
morphological properties of SDSS galaxies going from global morphology to more
detailed properties such as the number of spiral arms. The work by Dieleman et
al. (2015) was the winner of a public challenge on the Kaggle platform
111https://www.kaggle.com/. It achieved unprecedented classification accuracy
of $>90\%$ in most of the tasks (Figure 2).

Figure 2:  Example of level of agreement (red circles) and model confidence
(blue squares) versus classification accuracy. Each panel shows a different
question in the Galaxy Zoo classification tree (smoothness, edge-on, bar). The
authors quote an unprecedented accuracy of $>90\%$. This is the first work
that uses CNNs in astrophysics. The figure is adapted from Dieleman et al.
(2015) Figure 3: Schematic view of a simple _Vanilla_ type Convolutional
Neural Network, the most common approach for binary, multi class
classification and regression in extragalactic imaging. The input, which is
typically an image is fed to a series of convolutional layers. The resulting
embedding is used an input of a Multi Layer Perceptron which outputs a float
or array of floats. If the problem is a classification, the standard loss
function is the crossentropy ($H$), while if it is a regression the quadratic
loss ($L_{2}$) is usually employed.

This is a major improvement compared to previous approaches and marks the
beginning of the penetration of deep learning techniques in astrophysics.

Soon after, Huertas-Company et al. (2015) applied a similar architecture to
high redshift galaxies observed with the Hubble Space Telescope (HST),
demonstrating again a similar improvement as compared to other approaches,
including feature based Machine Learning. Using CNNs to classify galaxy images
represents in some sense a change of paradigm in the way we approach the
classification problem, analogous to what happened with natural images.
Instead of manually trying to identify specific features that correlate with
the classes to distinguish, the features are learnt simultaneously with the
classification process. This of course entails loosing some interpretability,
since the features learned by the network are no longer directly associated
with physical properties. Interpretability is a major issue in deep learning
applications to natural sciences that we will address in subsection 7.3.

##### CNNs as state-of-the art approach

In the past years, the number of works using deep learning to classify
galaxies based on their morphology has exploded, and CNNs have been used to
classify the morphologies of galaxies in a variety of optical/Near Infrared
surveys (we address the radio domain in subsubsection 3.1.2). It demonstrates
that deep learning has fast become the state-of-the-art approach to estimate
galaxy morphology in big datasets. Figure 3 illustrates the most common
approach used for morphological classification. Images are fed into a series
of convolutional layers which extract some summary statistics. The extracted
statistics are then fed into a Multi Layer Perceptron which maps them into a
class. The employed loss function is usually a cross-entropy loss. We notice
that in the original approach by Dieleman et al. (2015), a series of siamese
networks were introduced to add rotational invariance. This approach has not
been used in other works. Domínguez Sánchez et al. (2018) revisited the proof-
of-concept work by Dieleman et al. (2015) by carefully cleaning the training
sets and released the first deep learning catalog of galaxy morphologies in
SDSS. Ghosh et al. (2020) explored the classification of distant galaxies from
the CANDELS survey based on their bulge-to-total ratios. Goddard & Shamir
(2020) applied a similar strategy for Pan-STARRS. Vega-Ferrero et al. (2021)
and Cheng et al. (2021b) used CNNs to classify galaxies in the Dark Energy
Survey. Bom et al. (2021) followed a similar strategy for the S-PLUS survey
and Walmsley et al. (2022) classified galaxies in the DECALS survey. In
addition to observations, CNNs have also been extended to classify images from
cosmological simulations in order to assess the realism of galaxy morphologies
(Huertas-Company et al., 2019; Varma et al., 2022). In such applications, the
CNN is trained on labeled observations and then applied to mock images. In
addition to global morphology, Tadaki et al. (2020) used also a similar
supervised CNN setting to classify spiral galaxies based on their resolved
properties (i.e. type of spiral arms).

####  3.1.2 Radio galaxy morphology

In addition to the Optical and Near Infrared domains, the radio astronomy
community has also been very active in developing and testing deep learning
techniques for the classification of radio galaxies. These efforts are
motivated by the forthcoming arrival of new radio facilities such as
SKA222https://www.skatelescope.org/the-ska-project/ that will change the
landscape by detecting hundreds of thousands of new radio galaxies. Similarly
to what happened in the optical, the availability of large datasets with
labels has enabled the community to extensively test deep learning for
classification. The Radio Galaxy Zoo project (Banfield et al., 2015) used
indeed citizen science to determine the host galaxy of the radio emission and
the radio morphology of $\sim 170,000$ galaxies. This therefore constitutes an
excellent database for applying neural networks. It highlights the importance
of the preparatory work done by the community for accelerating the adoption of
deep learning techniques.

The first work exploring CNNs for classification of radio galaxies is Aniyan &
Thorat (2017). They use a simple sequential CNN (Figure 3) and conclude that
an accuracy up to $\sim 95\%$ can be achieved in classifying galaxies in three
main classes - Fanaroff-Riley I (FRI), Fanaroff-Riley II (FRII) and bent-
tailed galaxies. A number of works have followed. Alhassan et al. (2018) also
reported similar accuracy when using CNNs to classify compact and extended
radio sources observed in the FIRST radio survey (see also Maslej-Krešňáková
et al., 2021 for similar conclusions). Lukic et al. (2018) explores different
network configurations and concludes that a three layer network is typically
enough to reach more than $90\%$ accuracy. Wu et al. (2019) explored Faster
Region-based Convolutional Neutral Networks to detect and classify radio
sources from the Radio Galaxy Zoo project.

####  3.1.3 Strong Lenses

Deep Learning based supervised classification has been widely extended to
other extragalactic classification tasks. An example of application which has
significantly benefited from the advent of deep learning is the detection of
strong gravitational lenses (e.g. Jacobs et al., 2017; Petrillo et al., 2017;
Lanusse et al., 2018; Davies et al., 2019; Schaefer et al., 2018; Metcalf et
al., 2019; Petrillo et al., 2019; Jacobs et al., 2019; Li et al., 2020b; Huang
et al., 2020). Gravitational lenses produce characteristic distortions of the
light of background sources, caused by the presence of a foreground massive
galaxy or cluster in the same line of sight. The analysis of strong lenses
provides information about the total matter distribution of the foreground
system. Strong lenses provide therefore a unique probe of the dark matter
distribution in galaxies. The first step consists in identifying the lenses on
large samples of galaxy images.

Similarly to what has been described for galaxy morphology, the usual method
to identify lenses is through Convolutional Neural Networks as done for galaxy
morphology (Figure 3). However, there are some specific issues related to
strong lensing detection since the problem is severely unbalanced. The number
densities of strong lenses are indeed several orders of magnitude smaller than
the ones of regular galaxies. This poses two main problems. First, it is
impossible to build a large enough training sample of observed lenses. Second,
in order to be scientifically useful, the classifier needs to reach extremely
high purity values because even a small contamination from negative examples
provokes that the sample of lenses is dominated by false positives. The
community has proposed two sorts of solutions to these problems which appear
in most of the works. To cope with the lack of training examples, the CNNs are
usually trained on simulations. The physics of strong lenses is sufficiently
well known so that lenses can be simulated with some degree of realism (see
Figure 4). This practice is rather common in astrophysical applications as we
will describe in the forthcoming sections and especially in section section 7.
It does not come free of biases though. The work by Lanusse et al. (2018)
highlights the importance of using realistically complex simulations for
training in order to limit the potential biases.

Figure 4: Example of two different simulated samples of strong lenses used for
training a CNN. These simulations were used to detect strong lenses in the
CFHTLS survey. Figure adapted from Jacobs et al. (2017)

The problem with false positives is in general more difficult to solve. The
usual solution consists in visually inspecting the strong lenses candidates to
remove the false positives. In that context, deep learning helps to reduce by
several orders of magnitude the amount of required visual inspections but does
not completely get rid of them.

Despite these issues, deep learning has rapidly become the state-of-the art
technique to find strong lenses in large surveys. It has been successfully
applied to multiple imaging surveys following very similar strategies as just
described. The first work using CNNs for identifying lenses focused on the
CFHTLS survey Jacobs et al. (2017). Petrillo et al. (2017) and Petrillo et al.
(2019) applied a similar strategy for KIDS galaxies and Jacobs et al. (2019)
extended the approach to DES. Huang et al. (2020) focused on the DECALS
datasets. Some other works focus more on simulations only in view of preparing
future surveys. Lanusse et al. (2018) demonstrated the performance of CNNs to
detect lenses in LSST images and Davies et al. (2019) focused on Euclid.

Overall, the conclusions are shared among all different works. Deep learning
approaches are shown to improve more traditional techniques and therefore will
likely be used on future surveys. In support for this, the work by Metcalf et
al., 2019 shows the results of a strong lensing detection challenge in the
framework of the Euclid survey where the five best algorithms were based on
CNNs.

####  3.1.4 Open issues

In summary, deep learning techniques have rapidly replaced traditional
approaches for classification of astronomical images to the point that it will
most likely be the adopted approach to classify galaxies in forthcoming
surveys across the electromagnetic spectrum. The main advantages are speed and
accuracy. Convolutional Neural Networks in particular have demonstrated to be
more accurate for these classification tasks than other feature based
automated methods as seen in other disciplines. Classification is also one of
the less risky applications in the sense that it is - in most cases - very
close to the application of deep learning in the computer vision community.
Some open issues remain still.

##### Beyond vanilla CNNs

Even though standard vanilla CNNs provide in general accurate results, several
works have explored more complex configurations commonly used in computer
vision such as ResNets (e.g. Zhu et al., 2019; Kalvankar et al.,
2020).Overall, the results are promising but do not significantly improve over
approaches based on simpler CNNs. Some works have systematically compared
different neural network architectures on the same dataset (e.g. Fielding et
al., 2021; Cavanagh et al., 2021). The general conclusion is that, even if
there are some differences in accuracy and in efficiency, all architectures
fall in the same ballpark. A possible explanation for this is that
astronomical images present in general less diversity than natural ones and
hence relatively simple CNNs suffice to extract the relevant information. The
works by Katebi et al. (2019) and Lukic et al. (2019) are among the few works
in astronomy exploring the use of Capsule Networks (Sabour et al., 2017).
Capsule Networks are proposed as an alternative to CNNs that incorporates
spatial information about the features present in the images. Very briefly, it
uses a sort of inverse rendering to encode the presence of a given object in
an image. It therefore encodes information, not only about the presence or not
of a given object - which is what a CNN would do - but also information about
where the object is and how it is oriented. We do not provide a detailed
representation of the architectures given that this type of approach has only
been marginally used. The conclusion of the work by Lukic et al. (2019) is
that, in the case of radio galaxy classification, Capsule Networks perform
less well than more standard CNNs, reaching only an accuracy of $\sim 75\%$
(Figure 5). One possible explanation for that is that Capsule Networks were
initially thought to identify scenes which do not look realistic. For example
a CNN would learn to recognize faces based on features like eyes or noses.
However, they do not take into account the position of these features in the
image. Capsule Network do, but this is not a common issue for astronomical
imaging. This might be one of the reasons why Capsule Networks have not been
very used in astronomy so far. Becker et al. (2021) did the exercise of
systematically testing the performance of CNNs for radio galaxy classification
using multiple performance metrics such as inference time, model complexity,
computational complexity, and mean per class accuracy. They report three main
types of architectures that perform best but they are all variations of
sequential CNNs. In a recent work, Tang et al. (2021) explores the use of
multi-branch CNNs to simultaneously learn from multiple survey inputs (NVSS
and FIRST). Interestingly they confirm that including multi-domain information
allows to reduce the number of miss classifications by $\sim 40\%$.

Figure 5: Comparison of Capsule Networks and CNNs applied to classify
morphologies of radio galaxies. The ROC curves show the different
performances. Capsule Networks do not offer a significant gain in this
context. Figure adapted from Lukic et al. (2019)

##### Labeled data

A common bottleneck for deep learning based classifcation is the availability
of labeled samples to train the supervised algorithms. In astronomy this is
particularly delicate since the data properties (e.g. noise, resolution)
change from one dataset to another so in theory the labeling process should be
repeated for every new dataset. A number of works have addressed this issue
with different approaches. The work by Walmsley et al. (2020) explores Active
Learning as a way to reduce the amount of required examples for training.
Active learning allows one to select the most informative examples for the
model which are the ones showed to human classifiers. The work by Walmsley et
al. (2020) is also the first to explore Bayesian deep learning as a way to
both estimate uncertainties an also identify the most informative examples
(Figure 6). Domínguez Sánchez et al. (2019) and Ghosh et al. (2020) explored
transfer learning, which consists on refining the weights of a neural network
trained on a similar labeled dataset to reduce the need of large training
samples (Figure 7). They showed that the amount of labeled examples can be
reduced by a factor of a few with this approach. The issue of the size of the
training set that is needed is also investigated by Samudre et al. (2022). The
authors explore whether reliable morphological classifications can be obtained
with a small sample of $2000$ labeled images. They namely test transfer
learning but also few-shot learning techniques based on twin networks. The
conclusion is that even with small datasets, reliable classifications can be
obtained using CNNs, with an adapted training strategy. The recent work by
Walmsley et al. (2021) explores another version of transfer learning. They
show that the features learned by the CNNs for a given task can be recycled to
estimate other morphological properties. Vega-Ferrero et al. (2021) used
instead a simulated training set built from an observational sample from SDSS
to classify more distant galaxies from the Dark Energy Survey.

Figure 6: Example of posterior distributions of morphologies estimated from
the votes of different classifiers. The leftmost column shows an image of a
galaxy. The middle column shows the posterior predicted by a single network
(black), while the right column shows the posterior averaged over 30
Montecarlo dropout approximated networks. The red vertical lines indicate the
ground truth value, which generally shows a good overlap with the posterior
distribution. Figure adapted from Walmsley et al. (2020)

Figure 7: Example of Transfer Learning adapted to galaxy morphological
classification exploring how to use a CNN trained on SDSS to the DES survey.
Each panel shows a different morphological feature (smooth/disk, bars from top
to bottom). The different lines show the ROC curves using different training
models and test sets. The work concludes that when using a small subset of DES
galaxies to refine the weights of a CNN trained on SDSS, the accuracy of the
classification becomes optimal (red solid lines compared to blue dashed
lines). Figure adapted from Domínguez Sánchez et al. (2019)

Despite some remaining issues, some of which are common to most of deep
learning application - see section 7 for a more detailed discussion - it is
probably safe to argue that the community has accepted that deep learning will
be employed to classify galaxies from future surveys such as LSST and Euclid.

####  3.1.5 Transient Astronomy

The field of transient astronomy is about to change dramatically with the
arrival of synoptic sky surveys such as the LSST survey by the Rubin
Observatory, which will observe large areas of the sky with an unprecedented
frequency to find variable and transient astronomical sources. The number of
detections per night is expected to easily exceed several thousands. The
community has seen in machine learning techniques and particularly in deep
learning a promising way of classifying the detected objects and filtering the
most potentially (unknown) interesting candidates (see the report by Ishida,
2019). We will address the discovery of new types of transients in section 5.
We focus here one the supervised classification of variable sources.

One key science topic for cosmology is the detection and characterization of
SuperNovae light curves. There are different types of Supernovae and not all
are useful for the same purposes. For example, SNIa are used for cosmology.
Rapidly identifying the type of object saves - among other things - telescope
time. Although this is ideally done with spectroscopy, it is unfeasible to
perform a spectroscopic follow up of all the sources that will be detected.
Therefore the community started as early as 2010 to prepare for this data
deluge with the creation of simulated datasets such as the Supernovae
Photometric Classification Challenge (SPCC) or the Photometric LSST
Astronomical Time-Series Classification Challenge
(PLAsTiCC)333https://plasticc.org/. Because of these early efforts, there
exists a consequent literature using _pre deep learning_ machine learning
methods (e.g. SVMs, RFs and ANNs) to address the problem of SN light curve
classification (e.g. Lochner et al., 2016; Villar et al., 2019; Hosseinzadeh
et al., 2020; Vargas dos Santos et al., 2020).

The first work to use deep learning for SN light curve classification is by
Charnock & Moss (2017). They use to that purpose Recurrent Neural Networks
(RNNs), which are a type of Neural Network architecture designed to handle
sequences of variable length (see Figure 8 for a simple illustration). They
are called recurrent because they keep a memory of the previous information in
the sequence and use it to make the predictions. They are typically used for
language modelling. The authors report an average accuracy above $90\%$ for
the classification of light curves in three types - supernovae types I, II and
III - on the simulated sample from the Supernovae Photometric Classification
Challenge (Figure 9). Despite the small training set of a hundred data points,
RNNs achieved state-of-the-art results compared with a combination of template
fits and boosted decision trees (Lochner et al., 2016). In addition of not
requiring feature engineering, one advantage of RNNs is the ability to
classify incomplete light curves. Moss (2018) also explores RNNs on the same
simulated dataset. They propose some improvements such as a stronger data
augmentation process to mitigate the effects of small samples and reach
$>95\%$ accuracy. A similar conclusion is reached by Möller & de Boissière
(2020) who also explored RNNs. They report a similar accuracy also on
realistic simulations and confirm accuracies above $85\%$ for incomplete light
curves. See also Burhanudin et al. (2021) for similar conclusions. This latter
work proposes handling imbalance with a focal loss function.

Figure 8: Schematic view of a Recursive Neural Network (RNN) which has been
used for classification of light curves in several works. The photometric
sequence is fed to a recursive block and trained with a crossentropy loss. The
RNN blocks keep a memory ($h_{t}$) of the previous time step which make them
suitable for handling time sequences. Figure 9: RNNs used for SN photometric
light curve classification. The figure shows an example of light curve in five
photometric bands (top panel) and the probability of classifications as a
function of the time step (bottom panel). After roughly 50 days, the supernova
type is classified with high confidence. Figure adapted from Charnock & Moss
(2017)

The processing of data in the form of sequences has experienced significant
breakthroughs in the machine learning community over the past years. In
particular, the so-called attention methods which identify the region of the
sequences that contain the most relevant information have been demonstrated to
be very powerful for sequence to sequence tasks such as translation and for
classification of time series (Vaswani et al., 2017). This type of attention
based architectures are commonly known as Transformers (see Figure 10). The
application of Transformers to astronomy is still rather limited. However some
works have already explored their performance for SN light curve
classification and other types of transients. Allam & McEwen (2021) use a
variation of the original Transformer architecture to classify photometric
light curves from the PLAsTiCC simulated dataset. The authors demonstrate they
achieve state-of-the art accuracies. They claim the Transformer is able to
deal with very unbalanced classes without need of augmentation, achieving the
lowest logarithmic loss to date (Figure 11). However, as the authors
emphasize, the comparison with other methods is not straightforward given that
they are evaluated under different conditions. This highlights a general
problem for the comparison of different works performing classification.
Astronomy lacks in general of standardized datasets on which algorithms can be
consistently tested (see section 7 for a general discussion). Pimentel et al.
(2022) is a second work exploring the use of attention mechanisms. They apply
their method to real data after a first training on simulations and a fine
tuning step. They also conclude that the attention network outperforms
classical approaches based on RFs and also RNNs especially for late-
classification and early-classification of light curves.

Figure 10: Example of typical transformer architecture for the classification
of light curves. A Gaussian Process is first used to homogenize the time
sampling and the resulting sequence is fed into a CNN for feature extraction.
The attention modules are used to extract the final features which are used to
classify. Figure 11: Transformer applied to SN photometric light curve
classification. The figure shows a confusion matrix on the PLAsTiCC simulated
dataset. Most of the classes are identified with accuracies larger than $90\%$
as seen in the diagonal of the matrix. Figure adapted from Allam & McEwen
(2021)

In addition to purely photometric data, the classification of variable sources
can also be performed on images. This is traditionally done by subtracting
images from different epochs. Several works have explored the use of deep
learning to work directly in the image sequences. Carrasco-Davis et al. (2019)
used for example Recursive Convolutional Neural Networks (RCNNs) to identify
real transients from artifacts. Gomez2020 extend the use of RCNNs to a multi-
class classification of transients. They also use RCNNs to extract information
from both temporal and spatial correlations.

In summary, deep learning approaches have naturally incorporated to other ML
approaches to classify photometric light curves. In particular RNNs and more
recently Transformers provide competitive results. However, for this
particular task, deep learning does not seem to have dramatically improved
previous techniques as it is the case for image classification. The work by
Hložek et al. (2020) summarizes the results of the PLAsTiCC challenge
organized in the Kaggle Platform. It can clearly be seen that both _classical_
and deep learning approaches provide competitive results.

####  3.1.6 Other classifications

Similar supervised approaches for other classification tasks have been tested
over the past years, reaching also similar conclusions and facing similar
challenges. Kim & Brunner (2017) used convolutional neural networks for
separating stars from galaxies. Star-galaxy separation is a classical task in
the analysis of deep surveys. As for other classification tasks, the advantage
of CNNs is that they use the pixel level information and do not rely on
summary statistics. The general conclusion however is that CNNs offer a
marginal gain over more ML feature based approaches for this classification
problem. Ono et al. (2021) used CNNs to distinguish between real Ly$\alpha$
emitters (LAEs) and contaminants using imaging data in six narrow band
filters. Ackermann et al. (2018) made the first tests to classify galaxy
mergers trained on Galaxy Zoo and reported significant improvements over
state-of-the-art approaches (the case of galaxy mergers is extensively
discussed in subsection 4.2). Walmsley et al. (2019) and Tanoglidis et al.
(2021b) explored the use of CNNs for the classification of low surface
brightness (LSB) structures in deep imaging surveys. The systematic
exploration of the low surface brightness universe will be enabled by future
surveys such as LSST or Euclid. Automatically identifying and classifying LSB
structures is therefore a new challenge. The authors test a CNN model on the
Dark Energy Survey data and report a $\sim 95\%$ accuracy in separating
artifacts from real LSB structures. Only a few works have explored this
science case, probably due to the lack of proper labeled datasets.

###  3.2 Segmentation, deblending and pixel-level classification

Object detection and deblending is another problem on which deep learning
techniques have been extensively tested in these past years. The detection of
sources to build catalogs with some measured properties is a first standard
step in the processing of imaging from deep surveys. In image processing this
typically fits into the field of image segmentation, which is precisely the
task of identifying the positions and boundaries of different objects in an
image. The type of segmentation can be semantic, if the objects belong to
different classes, or instance if we aim at detecting objects of the same
type.

Figure 12: Schematic representation of a Unet type of architecture. This is
typically used for addressing image to image problems (e.g. segmentation,
galaxy properties in 2D). A first CNN (encoder) encodes information into a
latent space ($z$), which is then fed to another CNN (decoder) which produces
an image. One particularity of the Unet is the presence of skipped connections
between the encoding and decoding parts which have been demonstrated to
provide higher accuracies.

Since the popularization of CCDs, object detection in astrophysics is
generally done through the software called SExtractor, which implements an
advanced multi thresholding technique to detect and separate objects. Although
it has been extensively used over the past years, the limitations become more
obvious with the advent of deeper surveys in which the confusion between
sources becomes very common. It is estimated that $\sim 80\%$ of galaxies will
be affected by some sort of overlapping or _blending_. Given that blending can
severely affect the scientific conclusions, it is important to have reliable
methods for detection and deblending (see Melchior et al., 2021 for a review
on deblending).

Over the past years, there has been significant progress in the computer
vision community on image segmentation by applying deep learning networks.
Therefore, similarly to what happens for classification, deep learning
segmentation techniques developed for general purpose computer vision
applications are available. An out-of-the box implementation is thus expected
to provide reasonable results. Astrophysical data has however some key
properties which are not found in other types of images. The dynamic range is
very large, typically spanning several orders of magnitude from the centers of
the objects to the outskirts. Objects do not have clear edges. This is a
fundamental difference with respect to natural imaging applications, which
makes the segmentation task with neural networks more challenging.

A first very popular approach for object detection is the use of encoder-
decoder networks. Unets (Ronneberger et al., 2015), which incorporate skipped
connections between the encoder and decoder branches, and have emerged as one
of the state-of-the art segmentation networks (see Figure 12). Originally
designed for medical imaging, they have been commonly applied to astrophysics
for detection over the past years. Boucaud et al. (2020) first applied a Unet
to detect objects in image stamps and measure the photometry of overlapping
sources. In this proof-of-concept work, it is shown that the measured
photometry of pairs of galaxies is improved with respect to the standard
SExtractor based approach (Figure 13).

Figure 13: Comparison between the photometry measured with the Unet and with
SExtractor. All panels show a comparison between input and recovered
photometry. The top row shows the central galaxy in the stamp while the bottom
row is the companion one. The two leftmost panels show two different deep
learning architectures. The rightmost panel shows the results of the baseline
SExtractor. Both dispersion and bias are improved with deep learning. Figure
adapted from Boucaud et al. (2020)

However, this was tested in an idealized setting where the stamps only
contained two images with one galaxy at the center. Paillassa et al. (2020)
explored a similar type of architecture, in a more realistic setting, to
identify and classify artifacts in CCD images. In that case, classification
and segmentation are performed simultaneously at the pixel level. The proposed
approach successfully identifies multiple types of artifacts in an image
(Figure 14).

Figure 14: Unet used to segment different types of artifacts on CCD images.
The leftmost figures show a field containing different artifacts. The middle
panel shows the ground truth and the rightmost panels the predictions obtained
by the Unet. Figure adapted from Paillassa et al. (2020)

A similar approach is explored by Hausen & Robertson (2020) combining this
time object detection and morphological classification of galaxies at the
pixel level. Using a sliding window, the Unet successfully classifies every
pixel of the CANDELS survey in five morphological classes (Figure 15).
Huertas-Company et al. (2020) used a similar type of architecture to study the
resolved properties of galaxies by detecting giant star forming clumps within
high-redshift galaxies in the CANDELS survey. Burke et al. (2019), Farias et
al. (2020) Tanoglidis et al. (2021a) explored an alternative approach based on
region based CNN architectures such as Mask R-CNNs to perform similar tasks,
i.e. detection, deblending and classification.

Overall, these applications all show very promising results and clearly
improve on more traditional methods both in terms of speed but also in
accuracy, especially when combined with pixel-level classifications. In most
cases, the application of out-of-the box architectures is enough to provide
accurate results, once the input data are properly rescaled to limit the
effect of the dynamic range. However, until now, and with the exception of the
work by Hausen & Robertson (2020), the majority of the works have focused more
on testing and demonstrating the performance of these deep learning based
approaches. The application to real data to produce scientifically exploitable
data products remains very moderate. These approaches suffer from similar
limitations as the classification tasks, i.e. training sets and uncertainty
quantification (see section 7). Finding suitable training sets is more
challenging in this case as one need to both label and identify the boundaries
of the objects. In astronomy, the definition of object boundaries strongly
depends on the noise levels. The adopted solutions so far are either training
on simulations or relying on previous detections. None of them seems fully
satisfactory. Simulations are usually too simplistic and the generalization to
data can induce unexpected behaviors. Using outputs from other software
packages tends to propagate the existing biases. Possible solutions include
the generation of more realistic training sets with generative models (e.g.
Feder et al., 2020; Lanusse et al., 2021; Bretonnière et al., 2022). We will
discuss this in more detail in subsection 3.4. Recent works have also started
to explore the implementation of uncertainties in the produced segmentation
maps (Bretonnière et al., 2021) which seems a promising way to limit the
impact of possible catastrophic failures when changing domain from simulations
to observations. However, all these works remain at the proof-of-concept
stage.

Figure 15:  Example of pixel-level classification of a region in the CANDELS
survey using a Unet. The leftmost image shows the original field. The six
smaller images show different channels with different morphological types and
the rightmost image is the combination of all channels with a pixel level
morphological classification. Figure adapted from Hausen & Robertson (2020)

In addition to these segmentation based approaches, other groups have
attempted to go a step forward by reconstructing the surface brightness
profiles of overlapping galaxies. This implies moving from a classification to
a regression problem, since the output of the networks is the flux at a given
pixel. This task typically requires the use of generative models to learn the
diversity of galaxy shapes in a data-driven way and then being able to
generate likely solutions. Reiman & Göhre (2019) first explored the use of
Generative Adversarial Networks (GANs) for this purpose (see Figure 16). In
that case, the network input is a stamp of two overlapping galaxies and the
output are two images containing each of the two galaxies separately. An
adversarial loss is employed to ensure that the two produced galaxies are
realistic and indistinguishable from observed galaxies (see Figure 17).
Following a similar goal, Arcelin et al. (2021) used Variational Autoencoders
(VAEs) to reconstruct the light distribution of blended galaxies applied to
simulations of the LSST survey. Recently, Hausen & Robertson (2022) attempted
an intermediate solution in between full reconstruction and detection. They
proposed a novel approach based on Partial-Attribution Instance Segmentation
to estimate the fraction of fluxes in each of the galaxies from the blended
system. This is interesting as it provides a solution specifically designed
for the astrophysical problem, which remains rare in deep learning
applications.

Works attempting this task are still at the exploration level as well, even
though the results seem very promising (e.g. Figure 17). Reiman & Göhre (2019)
used very simple simulations by just adding two SDSS images; Arcelin et al.
(2021) employed analytical simulations. As all other deblending efforts, this
approach suffers from finding a suitable training set which is close enough to
observations without being too simplistic. Arcelin et al. (2021) showed that
transfer learning from a network trained on simulations is a promising
approach. More important, the statistical versus individual accuracy problem
becomes more dramatic when generating images instead of masks. Generative
Models produce realistic images in a statistical sense, i.e. arising from the
same probability density function than observations. However, on an individual
basis, artifacts might appear on the generated images. These are in general
very difficult to track down and can therefore induce significant biases. This
individual versus statistical accuracy is an inherent problem to machine
learning which needs to be taken into account when using ML predictions for
scientific analysis.

Figure 16: Schematic representation of a standard Generative Adversarial
Network (GAN). It has been used as a generative model for deblending,
identifying outliers as well as for accelerating cosmological simulations
among others. A first CNN (generator) maps a random variable into an image,
which is then fed to a second CNN (discriminator) together with real images to
distinguish between both datasets. The generator and discriminator networks
are trained alternatively.  Figure 17: Example of galaxy deblending using
GANs. The central images show three overlapping galaxies. On each side of the
big images, the original (left) and reconstructed (right) images of the pair
of galaxies are shown. Figure adapted from Reiman & Göhre (2019).

In order to limit the black box effect, an interesting approach is proposed by
Lanusse et al. (2019). The authors use an hybrid model that combines a
physically motivated model with analytic expressions for known terms, with a
data-driven prior for galaxy morphology learned with a generative model. In
this approach, the output of the generative model is only used as a prior of
the inverse problem, and therefore the impact of unexpected artifacts is
reduced. The combination of physically motivated models with data-driven ones
appears as an appealing solution which will likely become important in the
future.

###  3.3 Improving data quality

Deep learning has also been explored to improve the quality of data, i.e.
denoising and deconvolution. Astronomical images are usually noisy and blurred
by the effect of the Point Spread Function (PSF), which for ground based data
includes both the telescope impulse response and the effect of the atmosphere.
The processes of denoising and deconvolution aim therefore at recovering the
information before degradation. This is typically a challenging inverse
problem which needs significant regularization. Data-driven approaches have
emerged as alternative solutions to more classical deconvolution techniques.
Schawinski et al. (2017) first explored the use of Generative Adversarial
Networks (Figure 16) to deconvolve images from SDSS. They show that GANs can
recover features even after degradation. This remains however a simple
experiment since galaxies were previously degraded. Gan et al. (2021) built up
on a similar idea using GANs to translate between ground and space based
observations. Jia et al. (2021) proposes an improved solution based on two
different GANs which reduces the need of large amount of paired images with
and without degradation. Lauritsen et al. (2021) extended the idea to the sub
millimeter regime by using Variational Autoencoders instead of GANs. Encoder-
Decoder networks can also be used for denoising, and some works have explored
this for astronomy. Vojtekova et al. (2021) showed that Unets can effectively
increase the exposure time by a factor of two.

Generally speaking all the attempts show impressive results in solving the
long standing problem of deconvolution. They remain however at the proof-of-
concept stage and have not been applied for scientific analysis. Similarly to
what happens with deblending, the main limitation is robustness. Generative
Models such as GANs produce very realistic images but can also introduce
artifacts which are statistically meaningful but not necessarily on an image
per image basis. These artifacts can introduce uncontrolled biases in the
scientific analysis.

###  3.4 Emulating astronomical images

In a number of applications, the ability of simulate survey data is very
valuable, for instance to test and calibrate measurement pipelines. One of the
difficulties faced in simulations of upcoming surveys such as LSST or Euclid
is the relatively small set of deep and high-resolution imaging data
(essentially limited to HST surveys such as COSMOS) that can be used to
provide complex galaxy light profiles as inputs. This is one of the
motivations behind the development of Deep Generative Models for galaxy
morphology, which can be trained on the existing data and then generate
significantly more examples realistic light profiles.

In one of the first works following that direction, Ravanbakhsh et al. (2017)
demonstrated that training relatively simple GANs (Figure 16) and a
Variational Autoencoders (see Figure 18) on HST COSMOS postage stamps
successfully captured most of the relevant population-level parameters, such
as size, magnitude, and ellipticity. In subsequent work, Lanusse et al. (2021)
extended that model to explicitly account for the PSF, and proposed a hybrid
Normalizing Flow (see Figure 19) and VAE architecture which allowed to achieve
diverse and high quality samples, while also making it possible to condition
the light-profile generation on galaxy properties. Normalizing Flows are a
type of generative models which, as opposed to GANs or VAEs, allow the exact
evaluation of the likelihood $p(y)$ and therefore, their weights can be
directly learned by maximizing the log likelihood of the training dataset. The
idea is to construct a bijective mapping $f$ such that $y=f(z)$ where z is a
variable with a simple base density $p(z)$, typically a Normal distribution.
As $f$ is invertible one can evaluate the density $y$ using the change of
variable theorem, i.e. simply inverting $f$ and keeping track of the Jacobian
of the transformation (see Figure 19). Bretonnière et al. (2022) used that
generative model to create simulations of the Euclid VIS instrument on a 0.4
deg2 field with complex galaxy morphologies and used those simulations to
assess the magnitude limit at which the Euclid surveys (both deep and wide)
will be able to resolve the internal morphological structure of galaxies.

Figure 18: Illustration of a Variational Autoencoder (VAE). This generative
model has been extensively used in astronomy for multiple emulation tasks. A
first CNN maps the input into a distribution, which is then sampled to provide
the input to a second CNN which reconstructs the input image. A regularization
term is added to ensure that the latent distribution behaves close to a prior
distribution.  Figure 19: Illustration of a Normalizing flow. This density
estimator model is used to transform a simple distribution (typically a Normal
distribution) into a more complex density using an invertible function $f$
parametrized by a neural network which is optimized by maximizing the log
likelihood. The transformation can also be conditioned to an additional set of
parameters ($s$ in the figure).

One of the limitations of standard GANs and VAEs for the simulations of such
galaxies is that it quickly becomes difficult to generate high quality samples
on large stamps. To address these technical difficulties, Fussell & Moews
(2019) developed for instance a StackGAN approach in which a first GAN is
trained to generate a low resolution image (e.g. 64x64), which is then up-
sampled by a second GAN (e.g. to 128x128). Smith & Geach (2019) proposed to
use a GAN variant, known as a Spatial-GAN (SGAN) to generate no longer only
postage stamps, but entire fields of arbitrary size through a purely
convolutional architecture. The authors demonstrated the ability to sample a
87040x87040 pixels image emulating the HST eXtreme Deep Field (XDF). More
recently, Smith et al. (2022) explored applying a Denoising Score Matching
approach (Song & Ermon, 2019) and were able to generate large high resolution
postage stamps of size 256x256 of remarkable quality.

Summary of deep learning for computer vision
• Deep learning has rapidly emerged as a solution for the classification of
objects in large surveys. Galaxy morphology, strong lenses, and also light
curves constitute the main applications. Deep learning based catalogs,
especially for galaxy morphology, exist and are being used for scientific
analysis. • The most common approach for classification are supervised
Convolutional Neural Networks with different degrees of complexity. • Overall,
CNNs achieve higher accuracy than previous approaches and are also faster. •
The lack of labeled training sets is a common bottleneck. Solutions involving
Transfer Learning and/or the use of simulated training sets have been
proposed. This implies some additional challenges which we discuss in section
7 • False positives in the case of very unbalanced samples (e.g. for strong
lensing) is also a commonly encountered problem. No satisfactory solution has
been found to date apart from visual inspection. • Standardized datasets to
consistently compare the performances of different classification approaches
are not common in astronomy which limits the possibility of comparing
different approaches (see section 7). • Deep learning has also been explored
for object detection and segmentation in images. • The most popular approach
for segmentation are convolutional encoder-decoder networks such as Unets,
although more complex architectures have been also employed. • Overall, the
results are promising and tend to outperform pre-deep learning approaches for
detection. • Most of the works remain still at the proof-of-concept stage.
Until now, there are no deep learning based catalogs in major deep imaging
surveys. The robustness of such approaches is still a concern, especially for
deblending. Physics informed models could be a solution to explore in the
future.

##  4 Deep learning for inferring physical properties of galaxies

###  4.1 Neural Networks as fast emulators

We now move to address the efforts done in the past years to estimate the
physical properties of galaxies using deep learning. Compared to the previous
section where we described computer vision tasks, these applications are
typically more domain specific since they target physical quantities of
galaxies from a regression point of view. The general approach followed by the
community has been to test neural networks to emulate - replace - more
specific tools, developed over the years, which are usually slow or not fast
enough to deal with future big data surveys.

####  4.1.1 Photometric Redshifts

All modern cosmological surveys require a more or less precise estimation of
redshifts. When spectroscopy is not available, which is the case for most deep
surveys, photometric redshift estimation is the standard way to proceed to
measure distances of large numbers of galaxies using a combination of broad
and narrow band photometry. Photometric redshift estimation is therefore a
non-linear mapping between a set of photometric points and a real number
measuring the redshift. The standard way to approach the problem is through
the fitting of Spectral Energy Distributions generated from Stellar Population
Models (e.g. Benítez, 2000; Bolzonella et al., 2000). However, since it is -
in theory - a well defined problem, it is among the most popular applications
of deep learning supervised regression in astrophysics. The first attempts of
estimating photometric redshifts with neural networks start well before the
deep learning boom, in the early 2000’s (Collister & Lahav, 2004; Vanzella et
al., 2004). These methods already relied on the idea of learning the mapping
between photometry and redshift from data through a Multi-Layer Perceptron
trained under a Mean-Squared Error (MSE) loss. The only difference with a
modern architecture would be in the depth of the model and the choice of
activation function. Perhaps the most successful of these early neural methods
for photometric redshifts, ANNz (Collister & Lahav, 2004) has continued to
evolve over time, with ANNz2 (Sadeh et al., 2016) including some
quantification of epistemic uncertainties through an ensemble of randomized
estimators techniques reminiscent of modern deep ensembles.

Two significant evolutions of these methods appeared in recent years with the
generalization of deep learning: 1. probabilistic modeling of the redshift
distribution to estimate posterior probabilities; 2. pixel level models based
on CNNs, thus going beyond photometric information and able to use morphology
as well.

##### Probabilistic Modeling of Photometric Redshifts

Going beyond a regression task, (Bonnett, 2015) proposed to use a neural
network (still an Multi-Layer Perceptron - MLP), which for a given photometry
would output a distribution in the form of a discretized probability density
function (PDF). The model would then be trained with a standard cross-entropy
loss to predict the redshift bin in which a given galaxy should fall, which in
fact mathematically corresponds to estimating the posterior distribution of
redshifts given photometry, under a prior given by the selection of the
training set. This approach was subsequently reused to train other, more
complex, neural networks for photometric redshifts (Pasquet-Itam & Pasquet,
2018; Pasquet et al., 2019), but can potentially suffer from the
discretization needed to represent the PDF. Indeed, the network has no built-
in notion that classes with adjacent indices actually correspond to adjacent
bins.

Another approach to model distributions at the output of a neural network is
to use a Mixture Density Network (MDN Bishop, 1994). MDNs use an MLP to
predict the parameters of a mixture of probability densities, and thus provide
an analytic and continuous PDF model for a given input (see Figure 20). This
approach was for instance proposed in D’Isanto & Polsterer (2018), where the
neural network outputs are the mean, variance, and weights for a mixture of
$n$ one dimensional Gaussians. Overall, the general consensus, is that, when
only using photometry as input, neural networks do not provide specially more
accurate results than other template based approaches (see e.g. Schmidt et
al., 2020). One important challenge is that training sets are generally biased
as we will discuss in subsection 7.3.

Figure 20: Representation of a Mixture Density Network (MDN). It is a
generalization of a standard CNN/ANN in which the output has been replaced by
a probability distribution parametrized with some parameters ($\mu$,$\sigma$
in the figure). MDNs provide an estimation of the random (aleatoric)
uncertainty. The loss function is the generalized log likelihood.

##### Convolutional Photometric Redshift Estimators

The second major evolution of neural network-based photometric redshifts has
of course been the introduction of CNNs to build a pixel level model capable
in principle of using the entire information content of a multi-band galaxy
image to refine a redshift estimate. In the first instance of this approach,
Hoyle (2016) used the state-of-the-art model at the time, AlexNet (Krizhevsky
et al., 2012), to build a 5-layers deep CNN model taking as inputs a
combination of $r$,$g$,$i$,$z$ images of a given galaxy and trained to
classify that galaxy into a discrete redshift bin. Interestingly, in this
initial study performed over a set of $60,000$ SDSS images, no significant
improvements in redshift prediction accuracy were reported when compared to a
more traditional photometric feature-based AdaBoost machine learning model. It
would take a couple more years for a broader development of CNN based methods,
starting with D’Isanto & Polsterer (2018) which proposed to combine a simple
3-layers deep convolutional architecture with a mixture density output, but
again reported only a mild improvement in terms of accuracy compared to a
feature-based approach on an SDSS sample.

The benefits of a convolutional approach started to become clear with Pasquet
et al. (2019), which used a much deeper convolutional network, comprised of
one input convolution layer and 5 inception blocks, trained under a redshift
bin classification loss. These inception blocks (Szegedy et al., 2014)
essentially replace one convolutional layer by multiple parallel convolutional
layers with different kernel sizes, the output of which are concatenated back
into a single tensor at the end of the block. This study, based again on
galaxies from the SDSS Main Galaxy Sample using $ugriz$ images, illustrated in
particular how the CNN is able to automatically make use of pixel-level data
to extract information beyond colors, improving redshift estimation. In
particular, the bottom row of Figure 21 shows the comparison between the
photometric redshift bias of a standard color-based k-NN photometric redshift
estimate and the proposed CNN approach as a function of galaxy ellipticity
(proxy for galaxy inclination) for star forming galaxies. As can be seen, the
color-based approach exhibits a strong inclination-dependent bias caused by
the various amounts of dust attenuation as a function of the viewing angle.
The CNN shows however comparatively very little bias, indicating that the
model is able to automatically estimate and account for inclination in its
prediction by directly drawing that information from the postage stamps. This
example illustrates the main advantage of a deep learning approach, it
alleviates the need for handcrafted features, leaving it to the model to
identify from the data itself the relevant information.

Figure 21: Comparison of photometric redshift performance between a deep CNN
and a color-based k-NN (B16) method reported in (Pasquet et al., 2019). The
top row shows the predicted redshifts vs spectroscopic redshifts for the CNN
(left) and the k-NN method (right). The distribution is noticeably tighter for
the CNN with smaller overall bias, scatter, and outlier rate. The bottom row
show the photometric redshift bias $\Delta z$ for the two methods, as a
function of extinction (left panel) and disk inclination (right panel) of
galaxies classified as star-forming or starbust. Having access to the whole
image information, the CNN is able to automatically account for the reddening
induced by looking at galaxies with high inclination, whereas the k-NN method
only using color information is blind to this effect and exhibit a strong bias
with inclination.

Pasquet et al. (2019) highlight however an important consideration when using
a CNN approach. Whereas flux measurements can be standardized to account for
varying noise and PSF, a CNN based only on the raw postage stamps, without
additional information, is blind to these observing condition factors. The
authors report for instance a noticeable bias as a function of seeing for the
CNN, and mention the fact that this information could be provided to the model
in future work to let the model compensate for these factors.

In a number of subsequent works (Menou, 2019; Henghes et al., 2021; Zhou et
al., 2021), it was proposed to extend this pure convolutional approach to
photometric redshift estimation to hybrid models, combining an MLP branch
tasked with processed photometric features (e.g. colors) and a CNN branch
having access to the full image of the galaxy. It was found in these multiple
studies that directly providing the model with highly informative features
through the MLP branch improved the overall accuracy. Although all the
relevant information is in principle already included in the images
themselves, this approach reduces the amount of automatic feature extraction
that the convolutional branch needs to perform. In all cases, the best results
are achieved when both photometric features and images are provided jointly to
the model.

##### Improving Scaling with Size of Training Set

One particular aspect that may limit the applicability of these deep learning
methods is the need for large training samples, and so in this case the need
for large (expansive) spectroscopic samples to properly train these supervised
methods. As a possible mitigation technique for this issue, Hayat et al.
(2021) demonstrated that a self-supervised encoding provided by contrastive
learning retained significant redshift information. We direct the interested
reader to section 5 for more details on contrastive learning. Here, the
authors proposed a two-step approach, first training in complete self-
supervision (without needing any spectroscopic redshifts) a 1d encoding of
galaxies postage stamps. And in a second step, training in a supervised way a
shallow MLP to predict redshifts on a small training set of available
spectroscopic redshifts. They find two surprising results: 1. This fine-tuned
self-supervised approach always outperforms a conventional supervised
training, 2. The accuracy of self-supervised estimated redshifts scales very
well into the low-data regime. They find for instance that their self-
supervised approach using 20% of labeled data achieves similar accuracy to a
fully supervised training using labels for the entire dataset.

With a different strategy for limiting the amount of data needed, Dey et al.
(2021) proposed to replace a conventional convolutional architecture by deep
capsule networks. Compared to CNNs, capsule networks are robust to rotations
and changes in viewpoints and thus provide a more natural representation for
randomly oriented objects like galaxies (see also section 3). The hope is
therefore to not require as much training data if the model already provides
the built-in invariances of the problem. In their propsosed architecture, the
capsule network generates a low dimensional encoding of the input image, which
is then used to perform two tasks jointly: reconstructed in the input image
with a CNN, and estimating the redshift of the galaxy with an MLP. In
addition, their model also classifies, at the level of the capsule outputs,
the morphology type of the galaxy (elliptical or spiral). The authors find
that this approach leads to a better scaling with size of the training set
than (Pasquet et al., 2019), especially at very small training set sizes, but
the benefits are not as significant as the ones offered by the self-supervised
approach of Hayat et al. (2021).

Another complementary approach to reduce the dependence on large spectroscopic
datasets is to use transfer learning, to build a model on simulated data, and
fine-tuning it on survey data. This approach was for instance explored in
(Eriksen et al., 2020) using a MDN trained on a combination of FSPS
simulations and data from the PAU Survey. Using a pretraining on simulations
was found to reduce the photo-z scatter by as much as 50% for faint galaxies.

Finally, the question of robustness and stability of these deep neural
networks was investigated in Campagne (2020) which highlighted that inception
models like the one proposed in Pasquet et al. (2019) can be highly sensitive
to adversarial attacks. Although these attacks are unlikely to happen on
astronomical data (see however Ćiprijanović et al., 2021a), this result
underlines again the fact that these black-box methods are not as
interpretable as more conventional approaches like template-fitting
(interpretability is discussed in subsection 7.3).

####  4.1.2 Galaxy Structural Parameters

In addition to classification, galaxy morphology can be also quantified with
some parameters that define an analytical description of the surface
brightness distribution of galaxies. The so called Sersic models are defined
by three quantities: the effective radius ($r_{e}$), the Sersic index ($n$)
and the axis ratio ($b/a$). By combining these parameters with a normalization
factor to account for the different galaxy luminosities, one can describe most
of the surface brightness distributions of galaxies. The standard way to
measure these parameters is by fitting PSF convolved analytic models to the 2D
surface brightness distributions of galaxies. The task can be formulated as a
mapping between pixel values and real numbers, which characterize the galaxy
shape. It is therefore well suited for a supervised regression problem,
provided that a reliable training set is available. Given that the input data
are galaxy images, Convolutional Neural Networks are the most common approach.
Tuccillo et al. (2018) first used a CNN to estimate galaxy structural
parameters. In this first work, the authors used a simple training set made of
analytic profiles with noise added and demonstrated that CNNs achieve
comparable or better performance than standard methods, with the key advantage
of being a factor of $\sim 50$ faster. As previously said, computational
efficiency is one of the main motivations behind these works aiming at
emulating existing software. Tuccillo et al. (2018) also attempted to apply
the trained CNNs to observed galaxies with HST. However, because the training
set was too simple, the results did not appear to be satisfactory. In
particular, the authors did not include foreground and background galaxies in
the training set which constitutes an important difference with observations.
The authors proposed a transfer learning step using measurements performed
with standard approaches. Although the results quickly improve, the final
results necessarily propagate the systematics of existing software, which
cannot be improved by construction. In that respect, the main gain is speed.
Ghosh et al. (2020) built on this and showed that with a transfer learning
step, CNNs can provide reliable structural parameters for both low and high
redshift galaxies. The authors estimate structural parameters for $\sim
120.000$ galaxies in the SDSS and CANDELS surveys.

More recently, Li et al. (2021) attempted a similar approach applied to
ground-based observations. The training is still done on simulations but with
realistic backgrounds. Additionally, the PSF is included as an input to the
CNN, so that the networks can learn the effects of varying PSFs across the
field of view. The authors show, that by including these improvements, the
CNNs generalise well to observations without need of transfer learning and
achieve comparable results to standard approaches, with the advantage of being
$\sim 3$ times faster (see Figure 22). Other works have attempted to decompose
the galaxies into bulges and disks. This is an equivalent problem but the
number of parameters is increased by a factor of two. Grover et al. (2021)
showed that CNNs can estimate the bulge-to-disk ratio - i.e. luminosity ratio
between the bulge and the disk components - of $20.000$ galaxies in less than
a minute. The main motivation is, once more, a gain in computational time.
Tohill et al. (2021) explored the use of CNNs to estimate other morphological
parameters of galaxies which quantify the distribution of light (i.e.
Concentration-Asymmetry-Smoothness - CAS - system; Conselice, 2003). The
conclusion is very similar; neural networks accurately reproduce measurements
compared with standard algorithms, but faster. Interestingly, they also show
that using CNNs makes the measurements more robust in the low signal-to-noise
regime, which is one of the main issues of the CAS system.

Overall, these approaches look very promising to deal with large amounts of
photometric data such as the datasets that will be delivered by Euclid and the
Rubin Observatory for example. The limitations are similar to other problems.
The networks need to be trained on simulations by definition. The
extrapolation to real observations is always complicated as one needs to make
sure that the training set covers all the observed parameter space. As this is
a common challenge, we discuss it in section 7. This is particularly
challenging for space based observations in which the enhanced spatial
resolution increases the differences between the simulated datasets used for
training and the observations. A possible solution is the inclusion of some
sort of uncertainty estimation which could help identifying failures. This has
been recently done by Ghosh et al. (2022) who showed that a combination of
MDNs and MonteCarlo Dropout can provide well calibrated uncertainties of
galaxy structural parameters. Aragon-Calvo & Carvajal (2020) explored a self-
supervised approach to avoid using a fully supervised training based on
simulations. However, the approach has not been applied so far to large
samples of galaxies. In addition, since the inference time is very short, the
bottleneck of this type of approaches is in the training. In current
approaches, a specific training set needs to be built for every different
application which is not an optimal solution.

Figure 22: Convolutional Neural Network to measure galaxy structure. The left
column shows the difference between true and estimated radius. The right
column is the same for Sersic index. Top row shows the results as a function
of magnitude while the bottom row is as a function of radius. Figure adapted
from Li et al. (2021).

####  4.1.3 Stellar populations, Star formation histories

A similar type of application of deep learning is to derive the properties of
the stellar populations of large samples of galaxies. This is also a
regression type of application, in which a mapping between the galaxy
photometry and properties like the stellar mass, the metallicity or stellar
age, is sought. As for the previous applications, it exists a standard
approach based on the fitting of the Spectral Energy Distributions (SEDs).
However, it is typically slow and not adapted to the large volumes of data
that will be delivered by future surveys. Deep learning is thus used as an
accelerator. Most of the published works so far follow a similar approach. A
supervised Neural Network is trained for regression between photometric values
and stellar population properties. For example, Surana et al. (2020) used
fully connected Artificial Neural Networks applied to data from the GAMA
survey to derive stellar masses, star-formation rates and dust properties of
galaxies. The training is performed on stellar population models and the
results are compared to standard approaches. The conclusions are also very
similar to other applications falling in the same category. Deep Learning
performs similarly to standard methods, but a factor of a few faster.
Similarly, Simet et al. (2019) used neural networks to estimate the stellar
population properties of high redshift galaxies from the CANDELS survey. The
training is in this case performed on semi-analytical models. The conclusion
is that galaxy physical properties can be measured with neural networks at a
competitive degree of accuracy and precision to template-fitting methods. It
is worth noticing that Neural Networks are not the only approach to address
this problem, although in this review we primarily focus on deep learning
techniques. As this is essentially a mapping between two sets of real numbers,
other Machine Learning techniques can be employed -Gilda et al., 2021;
Davidzon et al., 2019 used for example Boosting Trees and SOMs respectively
for a similar task.

In a recent work, Buck & Wolf (2021) pushed this idea further by trying to
predict resolved stellar population properties instead of integrated
quantities (Figure 23). In that case, the mapping is made between broad band
photometric images of galaxies and 2D maps of stellar mass, metallicity and
other stellar population properties. This is equivalent to a regression at the
pixel level. The architecture for this type of work is en encoder-decoder Unet
type of network as the ones used for segmentation (see section 3 and Figure
12) but with a mean square error loss to work in regression mode.

buck21

Figure 23: CNN applied to estimate resolved stellar populations of galaxies.
The input are images in different wavebands (2 left columns) and the output
are different stellar population maps (3 right columns). In the stellar
population maps, the top row is the ground truth, the middle one the
prediction and the bottom row shows the residual. Figure adapted from Buck &
Wolf (2021).

In addition to stellar population properties at the time of observation, one
can use the photometry of galaxies to infer the star formation histories
(SFHs), i.e. the star formation rate as a function of cosmic time. There are
several established approaches using either parametric or non-parametric
methods. However, the problem is known to be significantly degenerate and the
star formation activity at early times is in general poorly constrained.
Therefore the final estimation heavily relies on established priors. Lovell et
al. (2019) first attempted to use CNNs in a supervised regression setting to
estimate the SFHs of galaxies in the EAGLE cosmological simulation (Figure
24). One advantage of training on hydrodynamic simulations is that the prior
is learned in a data-driven way by using fairly realistic distributions from
the simulations. The authors of this first work show a reasonable
reconstruction of the SFH and a decent robustness to domain changes. Qiu &
Kang (2021) uses CNNs for the opposite task, i.e estimate the galaxy SED from
the galaxy SFH taken from simulations. In this case deep learning acts as an
emulator of radiative transfer codes.

As other applications of this kind, the results strongly rely on simulated
training sets and on the implicit assumption that the simulations properly
cover the observed parameter space. This is particularly critical here since
the mocking from numerical simulation is usually done with existing stellar
population models which are unavoidably a simplification of reality. Another
important limitation is that, up to now, none of the published works on this
front properly accounts for uncertainties, even though uncertainty estimation
is far from being a solved issue with traditional methods. Both uncertainty
estimation and domain shift are common challenges to many applications which
are discussed in section 7.

Figure 24: CNN applied to reconstruct the Star Formation Histories of
galaxies. The input are photometric images and the output the Star Formation
Rates in different bins of time. The different panels illustrate some examples
of predictions (red lines) together with the ground truth from cosmological
simulations (blue lines). Figure adapted from Lovell et al. (2019).

####  4.1.4 Strong lensing modeling

Another science case in which deep learning techniques have been extensively
tested over the past years is the modeling of strong gravitational lenses -
the formation of multiple images of distant sources due to the deflection of
their light by the gravity of intervening structures. In subsection 3.1 we
have discussed efforts done to find these lenses on large datasets. The goal
here is to characterize the lenses. This generally means quantifying image
distortions caused by strong gravitational lensing and estimating the
corresponding matter distribution of these structures (the ’gravitational
lens’). Similarly to the previous applications in this category, there exists
a method to perform this analysis, based on maximum likelihood modelling of
observations. The process is however time consuming requiring complex
dedicated software. Deep learning appears therefore as an appealing approach
for accelerating the inference. The first work in exploring this is by Hezaveh
et al. (2017). The authors test CNNs to estimate the lensing parameters from
the images - Einstein radius, complex ellipticity, and the coordinates of the
center of the lens. They show that CNNs can recover the parameters with
similar accuracy than standard methods but ten million times faster (Figure
25). An obvious caveat of the deep learning approach for inference is the lack
of reliable uncertainties. Perreault Levasseur et al. (2017) is one of the
first works exploring Bayesian Neural Networks to estimate uncertainties in
the modeling of strong lenses, and in astrophysics in general. They use the
technique of Monte Carlo dropout to approximate Bayesian Neural Networks (Gal
& Ghahramani, 2015; Charnock et al., 2020) and show that, in that particular
case, it results in accurate and precise uncertainty estimates, significantly
faster than Monte Carlo Markov Chains.

Figure 25: Estimation of strong lensing parameters with CNNs. Each panel shows
a different parameter (ground truth versus CNN estimation). Figure from
Hezaveh et al. (2017)

These two pioneer works have set the route for a fair amount of publications
exploring the use of deep learning for lens modeling along similar lines. The
most typical approach is the use of CNNs on images with an approximate
Bayesian component to estimate uncertainties. For example, Madireddy et al.
(2019) proposed a complete deep learning based pipeline including detection
and classification of lenses followed by a modeling phase. Bom et al. (2019)
apply Residual Neural Networks to simulated images of the Dark Energy Survey
to predict Einstein Radius, lens velocity dispersion and lens redshift within
$\sim 10-15\%$. See also Schuldt et al. (2021) for similar conclusions on
simulated Hubble Space Telescope and Hyper Suprime-Cam Survey images. Pearson
et al. (2019a) applied CNNs to simulated LSST and Euclid images. They find
that including color information results in a $\sim 20\%$ increase in accuracy
compared to single band estimates. In a follow up paper, Pearson et al. (2021)
perform a systematic comparison between CNN based estimation and conventional
parametric modelling on increasingly realistic datasets going from smooth
parametric profiles to real observations from the Hubble Ultra Deep Field. The
main conclusion is that CNNs outperform traditional methods not only in terms
of speed but also in accuracy by $\sim 20\%$. However, the work also concludes
that combining both approaches reduces further the errors. In addition, the
use of CNN priors reduces the computational time of parametric modelling by a
factor of $\sim 2$. Chianese et al. (2020) goes a step further by proposing a
fully differentiable framework for lensing modeling. The pipelines combines a
data-driven model based on a VAE for the source and a physical model for the
lens which allows a forward modeling of the system to be compared with the
observations. The main novelty, is that thanks to the differentiable
programming framework, it becomes possible to compute the derivatives of the
likelihood function with respect to both the parameters of the source and the
lens, allowing for fast inference (Figure 26). Along similar lines,
Morningstar et al. (2019) combines a physical model with a Recurrent Neural
Network to iteratively reconstruct the lens model which is then fed to a CNN
to estimate the lens parameters. Morningstar et al. (2018) applies the same
methodology to analyze interferometric data. The modeling of lenses can be
combined with a direct inference of cosmological parameters such as the Hubble
constant (Park et al., 2021). We will address these applications in more
detail in subsection 6.2. Maresca et al. (2021) proposed to use CNNs, not for
reconstruction, but to identify unphysical models from parametric fitting.

Figure 26: Lens modelling with a fully differentiable approach. The panels
show the posterior distribution of different parameters of the model for
different SNRs as labeled, together with the ground truth (black circle).
Figure adapted from Chianese et al. (2020)

####  4.1.5 Other properties

Deep learning has also been applied to measure other galaxy properties, in
addition to the major categories described in the previous subsections. Stark
et al. (2018) used a Generative Model to measure the photometry of AGN hosts.
The neural network is used in this case to separate the light of the quasar
from the emission of the host galaxy. In that respect it is similar to a
deblending problem discussed in section 3. In line with works under the same
category, the authors demonstrate that their approach is more than 40 times
faster than standard model fitting approaches. Yao-Yu Lin et al. (2021) also
addressed the issue of AGN quantification by inferring the mass directly
without going through photometry. The input in this case are quasar light time
series. The work shows that neural networks reach similar accuracy than
traditional methods that use SDSS spectra which are more time consuming to
obtain. Also within the time domain community, Stahl et al. (2020) developed a
deep learning framework to measure the phase and the light curve shape of Type
Ia supernova from an optical spectrum. Cabayol-Garcia et al. (2020) and
Cabayol et al. (2021) explored CNNs to systematically measure photometry on
the narrow band PAU survey. They find in this case that the deep learning
approach improves the photometric accuracy by up to $\sim 40\%$.

###  4.2 Galaxy physical properties from simulations

In the previous section we have discussed applications where deep learning is
used as an accelerator to replace existing methods. The main motivation of
these works is therefore efficiency for dealing with large amounts of data.
Because neural networks are universal approximators they can also be used to
unveil new correlations between observables and physical quantities. In this
case, deep learning is not replacing existing methods, but rather used as an
exploration tool to unveil new patterns in the data which can be informative
about underlying physical processes and/or about physical properties of
galaxies from simulations.

####  4.2.1 Physical processes of galaxy formation

Deep Learning offers a new way of establishing correlations between image
features and physical processes driving galaxy formation in general. The
general procedure is that simulations are used to identify a physical process
without ambiguity. For example, galaxy mergers are straightforward to identify
galaxies in a simulation but challenging to find in observations. Mock
observations can therefore be produced to train a CNN at identifying the
physical process. The advantage is that the network is let free to identify
the optimal features that characterize a physical process given the
observables. This way of proceeding is partly new. It has been driven by both
the emergence of deep learning techniques and large sets of numerical
simulations.

##### Galaxy mergers

A central example of this type of application which has caught the attention
of many groups in the past years is the characterization of galaxy mergers.
Mergers of galaxies are arguably a key driver of stellar mass assembly in
galaxies across cosmic time. Identifying and characterizing the properties of
large samples of mergers to assess their impact on diverse assembly processes
has remained a key open issue in the field of galaxy formation for many years.
The precise measurement of the merger rate - number of mergers per unit time
and unit volume - is also an indirect probe of the cosmological model. Since
galaxy mergers tend to disrupt the surface brightness distribution of galaxies
because of the gravitational interaction, it is an old idea to use
measurements of perturbations in the luminosity profiles of galaxies as
indicators of a merger activity. A popular approach in the early 2000’s was to
measure some moments of light that are sensitive to asymmetries in the light
distribution (e.g Conselice, 2003). The problem is that the link between these
perturbations and the actual merger activity is loose. The observability
timescale of a given feature such a tidal structure depends on the type of
merger, the cosmic epoch and other properties, which makes it very difficult
to establish a direct link between image features and merger status.

In that context, numerical simulations offer an attractive way of connecting
measurements on images to a phase in the merger since the dynamical process of
merger can be entirely tracked down in the simulation. Early efforts had
indeed tried to establish some first order calibration using numerical
simulations of galaxy pairs (e.g Lotz et al., 2008). The work by Lotz et al.
(2008) associated variations in the moments of light - concentration, Gini,
asymmetry also known as CAS parameters - with the merger phases. However this
was done manually and with a limited set of simulations. Snyder et al. (2019)
improved on these early works by exploring Random Forests applied to moments
of light on numerical simulations. More recently Whitney et al. (2021) used
galaxies from the TNG simulation to calibrate the observability timescale of
the so-called CAS parameters.

Deep learning offers however a new way of looking at this problem of detecting
and characterizing galaxy mergers by bypassing summary statistics and manually
engineered features. Pearson et al. (2019b) first applied a CNN to the
identification of galaxy mergers using a training set of mergers from the
EAGLE simulation mimicking the SDSS observational properties. The key
difference with previous works is that no manual features are extracted; the
images of the different mergers are provided as input. In this first work,
they found that CNNs did not achieve very high performance which is
interpreted as a signature that the images used did not present significant
perturbations. One possible reason for this poor performance is that mergers
were selected in an non homogeneous way over a large range of times. The
importance of the training set was carefully analyzed in the work by Bottrell
et al. (2019). The authors explored how the performance of CNNs changed
depending on the realism of the training set used for training. The main
conclusion is that it is more important that images used for training
reproduce the observational properties of the sample to be analyzed, i.e. PSF,
noise, background sources, than using full radiative transfer to improve the
conversion from stellar particles to light. Ferreira et al. (2020) followed up
on this idea and used deep learning for the first time to compute the merger
rate up to $z\sim 3$. They trained a supervised CNN to identify mergers
labeled in the TN300 simulation and selected over a fixed time window of $\sim
1Gyr$. They showed that the CNNs can distinguish mergers from non mergers
based on multi-wavelength imaging with an accuracy of $\sim 90\%$. When
applied to data from the CANDELS survey, their results reconcile measurements
of the merger rate performed with pair counts and photometry (see Figure 27).
These results confirm that the discrepancy between the two approaches was
caused mainly because of calibration issues. Using deep learning on simulation
allows one to properly calibrate the observability timescale based on the
simulation metadata and therefore obtain a more reliable measurement of the
merger rate. A similar approach was followed by Bickley et al. (2021), who
trained a CNN to focus only on post mergers identified in the TNG simulation.
They confirm that deep learning techniques outperform moment based
identifications of post mergers and applied their model to the CFIS survey.
However, the authors highlight a common problem with very unbalanced
classification problems as this one. Since the number of post merger galaxies
is very small compared to the global population of galaxies, even a small
fraction of false positives, strongly impacts the purity of the post merger
sample (see subsubsection 3.1.3).

Figure 27: Supervised deep learning trained on simulations used to infer
galaxy merger rates. The panels show the inferred merger fractions (left) and
merger rates (right) with deep learning as compared to other independent
techniques. Figure adapted from Ferreira et al. (2020)

Because neural networks are very flexible, they can be easily used to combine
the information of different types of inputs which is more difficult with
other techniques. Bottrell et al. (2021) recently explored the combined
information provided by photometry and kinematics to detect mergers. Both maps
are fed into deep learning network which combines the information in an
unsupervised way. Interestingly, the authors conclude that kinematics does not
bring significant additional information.

Other works have attempted to go beyond the classification and use neural
networks to regress on the properties of the mergers. Koppula et al. (2021)
used a residual deep neural network to estimate the time to / from the first
passage in a merger sequence. Using information from the Horizon-AGN
simulation, the authors produce a large sample of images of major mergers at
different stages. They show that, based on a single snapshot, the CNN is able
to recover the position of the stamp in the merger sequence with an error of
$\sim 40$ Myrs. This is interesting because deep neural networks are used to
provide temporal constraints on phases of galaxy formation, based on a single
snapshot. A similar approach was followed by Cai et al. (2020). They trained a
combination of Autoencoders and Variational Autoencoders to infer the
properties of galaxy mergers. They conclude as well that with a single image,
the dynamical status of the mergers can be inferred, bypassing dynamical
modelling. Even more recently Eisert et al. (2022) used an invertible neural
network to infer several mass assembly indicators of galaxies (i.e. mass of
accreted stars, time since the last major merger) using a variety of
observable quantities from the TNG simulation.

##### Other physical processes

Huertas-Company et al. (2018) first applied this idea to the identification of
a physical process called _compaction_ or _blue nugget_ phase. Several
observational works suggested that diffuse star forming galaxies become
compact star-forming galaxies called ’blue nuggets’ (BN) which subsequently
quench (’red nuggets’) following a sudden gas inflow towards their center. The
VELA zoom-in simulations (Ceverino et al., 2015) show also rapid gas inflows
leading to central starbursts, and several mechanisms were identified that
lead to this compaction phenomenon including major gas-rich mergers or disk
instabilities often triggered by minor. The authors tested whether deep
learning can detect the blue nugget phase of galaxy evolution purely
identified in the simulation metadata. Mock HST images from the simulated
galaxies were labelled according to their evolutionary stage from the
simulation metadata (before, during, or after the BN phase), i.e. the labeling
is exclusively done based on physics. The result is that galaxies can be
successfully classified into evolutionary stages without identifying specific
features, i.e. just using the pixel space (see Figure 28).

Figure 28: CNN applied to reconstruct to classify galaxies in different
evolutionary stages defined by cosmological simulations. Each column shows a
different phase of evolution as defined by the simulation. The top row is the
high resolution from the simulation. The middle row shows the same galaxy with
observational realism added. The bottom row shows real observed galaxies
identified by the CNN as being in one of the three different phases. Figure
adapted from Huertas-Company et al. (2018)

Diaz et al. (2019) applied a similar idea to the classification of formation
mechanisms of lenticular galaxies. They identified different evolutionary
tracks leading to the formation of S0 galaxies - isolated, tidal interaction
in a group halo, and spiral-spiral merger - and trained a CNN to identify them
based on stellar density and two-dimensional kinematic maps. They found that
the CNNs are able to distinguish the different formation scenarios and
conclude about the potential of deep learning to classify galaxies according
to their evolutionary phases. Schawinski et al. (2018) used a Generative
Adversarial Network to try constraining the physical processes leading to the
quenching of star formation in galaxies. Ginzburg et al. (2021) used a CNN
trained on zoom-in cosmological simulations to infer the longevity of star-
forming clumps in high redshift galaxies. Instead of relying on the conversion
between photometry and age of the stellar populations, they defined two types
of clumps - short and long lived - based on the information from the
simulation and trained a supervised neural network in a binary classification
mode to distinguish between the two types. This is yet another example where
neural networks are used as universal approximators to find not obvious links
between observables and physical properties.

##### Open issues

Using deep learning trained on simulations to constrain the phases of galaxy
formation is becoming increasingly popular in the community and the example of
mergers clearly illustrates this. All these approaches suffer however from an
obvious limitation, the so called domain gap between observations and
simulations. As this is a recurrent problem, more information is provided in
section 7. Since simulations do not perfectly reproduce observations, applying
a trained network on observations will induce some biases. Moreover, since
observations are generally not labeled at all, especially when trying to infer
physical properties, it is impossible to evaluate the effect of the domain gap
and the results need to be accepted blindly. As already mentioned in previous
sections, including uncertainty quantification in the neural networks is an
option to mitigate this effect. However the error induced by changes in the
domains is in general difficult to capture by uncertainty quantification
methods. Other options consist in performing the domain adaptation during
training to ensure that the features learned by the neural networks are not
specific to the simulation domain. There are different approaches to do so
since it is a problem that exists in many fields of application (see e.g. Wang
& Deng, 2018). In extragalactic astronomy there has been limited exploration
of these approaches. Ćiprijanović et al. (2021b) recently explored the impact
of domain adaptation during training for the identification of galaxy mergers.
They tested different domain adaptation techniques such as Maximum Mean
Discrepancy and Domain Adversarial Neural Networks and concluded that
including these leads to an increase of target domain classification accuracy
of up to $\sim 20\%$. This is a promising result for future applications of
neural networks trained on simulations. It is likely that future works will
start to incorporate these approaches more often.

####  4.2.2 Dark Matter

In a similar spirit of finding new correlations, deep learning has been
increasingly used in the past years to estimate the dark matter masses and
properties of galaxies and clusters, based on observable quantities.

##### Dynamical masses of clusters of galaxies

The earliest applications focused on galaxy clusters, which are the largest
gravitationally bound objects. This is because there exist alternative methods
to measure dynamical masses of clusters. However standard approaches are based
on simple scaling relations based on the virial theorem. The simplest approach
is to use a power law relation between the dispersion of the line of sight
velocities (LOS) and the cluster mass. This was indeed one of the first probes
of dark matter by Zwicky (1933). Other classical approaches consist in using
scaling relation between the X-ray luminosity of the gas and the mass of the
cluster or the Sunyaev-Zeldovich effect. However, these scaling laws are all
based on strong assumptions about the physical status of the cluster, the most
important being that the system is in virial equilibrium. This entails some
inherent biases in the estimated cluster masses.

Deep learning and machine learning in general, calibrated on simulations where
dark matter properties are known, offer an interesting approach to look for
additional correlations which could help in reducing the scatter in the
dynamical mass estimates. Early efforts were done before the emergence of deep
learning, especially in the works by Ntampaka et al. (2015, 2016). The authors
explored Support Distribution Machine class of algorithms to predict cluster
masses using LOS velocities and radial positions of cluster members. They
reported an improvement of a factor of two (see also Armitage et al. (2019)
for similar conclusions).

Ntampaka et al. (2019) applied CNNs to cluster mass estimation in which is the
first work using deep learning for this purpose. The training is performed
with mock 2D X-ray images of Chandra observations. There is indeed a well
known correlation between X-ray luminosity and cluster masses. They report a
$\sim 10\%$ smaller scatter than standard luminosity based methods, even
without using any spectral information. Interpretability techniques based on
attribution methods, reveal that the CNNs tend to ignore the cluster centers
because they likely lead to more biased estimates. This is another example of
CNNs used to find new correlations in the data and one of the few examples
were basic attribution techniques provide meaningful information. Similarly,
Yan et al. (2020) used a combination of feature maps (stellar mass, soft X-ray
flux, bolometric X-ray flux, and the Compton y parameter) and reach comparable
results. This work illustrates an other advantage of deep learning over
traditional approaches, which is the possibility of combining multiple
observables in a transparent way.

Along the same lines, Ho et al. (2019) explored CNNs to estimate cluster
masses. The training is based on LOS velocities and radial positions of
galaxies in the cluster. They explored both 1D and 2D CNNs. They also report a
factor of $\sim 2$ improvement with respect to power-law based estimates.
Interestingly CNNs also improve the results of more classical ML approaches
explored in previous works.

Convolutional Neural Networks have also been explored on the third observable
usually employed to infer cluster masses, the Sunyaev-Zeldovich effect. de
Andres et al. (2021) used CNNs on mock maps of the Planck satellite from
numerical simulations. The advantage of using deep learning is again that no
assumptions on the symmetry of the cluster’s gas distribution nor on the
cluster physical state are made.

These previous works, although promising, remain at the exploratory level and
suffer from the same limitations than other similar approaches. Namely, the
results are restricted by the prior inferred from the simulations and they
usually lack of uncertainty estimation with some exceptions. The works by Kodi
Ramanah et al. (2020); Kodi Ramanah et al. (2021) make a step forward to
address some of these issues. In these works, the authors explore, for the
first time, flow based neural networks (see Figure 19) trained of the phase
space of cluster galaxies to infer cluster masses (Figure 29). The key
addition of their approach is that the network provides therefore a full
probability distribution of the cluster mass instead of a single point
estimate, which can therefore be used to account for uncertainties. This is a
key step forward towards an application of deep learning based approaches for
estimation of cluster masses in large surveys. The authors claim a factor of 4
improvement compared to scaling relations based estimations. They then apply
their model to a sample of observed clusters with well calibrated dynamical
masses and show that the neural network provides both unbiased measurements
and well calibrated uncertainties. Ho et al. (2021) also investigate the use
of Bayesian CNNs to include uncertainty measurements. They show that BNNs
recover well calibrated 68% and 90% confidence intervals in cluster mass to
within 1% of their measured value.

Figure 29:  The figure shows the predicted dark matter mass as a function of
the true mass along with uncertainties using a Normalizing Flow. Figure
adapted from Kodi Ramanah et al. (2021) Figure 30: Illustration of a Neural
Flow for estimating posterior distributions. This type of approach is starting
to become common in simulation based inference approaches for estimating
cluster masses or cosmological parameters. A first Neural Network with a
standard $L_{2}$ loss is used to obtain some summary statistics of the data
which are then use as a condition for a Neural Flow mapping a simple
distribution into an approximation of the posterior.

##### Halos of galaxies

Deep learning can be also extended to estimate dark matter halo masses of less
massive galaxies than clusters. In low mass halos, there are also less
galaxies and therefore it is more challenging to use LOS velocities. They do
not present any X-ray emission either. The most standard way to proceed is by
using Abundance Matching techniques. Abundance Matching main assumption is -
with some variations - a monotonic relation between the stellar masses of
galaxies and dark matter halos. By using dark matter halo mass functions from
N-body simulations, one can then assign halo masses to galaxies. In this
context, deep learning can be used to look for additional correlations between
galaxy properties and dark matter beyond simple abundance matching
assumptions. Calderon & Berlind (2019) explored several machine learning
algorithms, including neural networks, for estimating the dark matter halo
masses of galaxies in the SDSS. They used ML to explore how much information
about halos is provided by several galaxy properties from synthetic catalogs.
They conclude that including more physical properties is translated into a
better accuracy than the one reached by abundance matching. A problem with
this approach, acknowledged by the authors, is that secondary dependencies of
halo masses on galaxy properties might be very model dependent. This can
therefore induce systematic biases in the inferred dark matter masses which do
not exist in simpler approaches. Shao et al. (2021) investigated how subhalo
masses can be estimated using neural networks trained on a number of physical
properties of galaxies (i.e. black hole mass, gas mass, stellar mass etc) from
numerical simulations. They used the CAMELS simulation suite which is a series
of numerical simulations performed with different codes and cosmologies,
specially designed for ML. We will describe the simulations into more detail
in subsection 6.2. They found that subhalo masses can be predicted accurately
($\sim 0.2$ dex) at any redshift from simulations with different cosmologies,
astrophysics models, subgrid physics, volumes, and resolutions. The authors
argue that the neural networks might have found a universal relation, which
turns out to be a generalized version of the virial theorem involving radius,
velocity dispersion and maximum circular velocity. This is a good example of
deep neural networks used to find hidden correlations which can be even
translated into analytical expressions. We will discuss this further in
section 5. In a recent work, Villanueva-Domingo et al. (2021a) explored the
use of Graph Neural Networks (GNNs) to estimate halo masses of galaxies. GNNs
are a special type of neural networks that are built on graphs, and therefore
allow one to account for the relations between neighbouring halos. The authors
used the halos from cosmological simulations as nodes of the graphs and
encoded the gravitational interaction between them in the edges of the graph.
The nodes include the positions of the halos, the relative velocities, the
stellar mass and the half-mass radius. They show that the model is able to
estimate halo masses with a $\sim 0.2$ dex uncertainty (Figure 31). The model
is also built to account for uncertainties and is shown to generalize
reasonably well between different simulated datasets.

Figure 31: The top panel illustrates the accuracy obtained on simulations,
when the training and testing is done on datasets coming from the same
underlying cosmological simulation. The bottom right panel is when two
different simulations are used for training and testing respectively. Figure
adapted from Villanueva-Domingo et al. (2021a)

In a follow-up work (Villanueva-Domingo et al., 2021b), the authors apply the
trained GNNs to measure the halo masses of the Milky-Way and Andromeda
galaxies. They show that the inferred constraints are in good agreement with
estimates from other traditional methods.

####  4.2.3 Deep learning generated observations

Some recent works have tried to push even further the ability of neural
networks to establish not obvious non-linear mappings between domains to
bypass telescope observations. Spectroscopic observations are more expensive
in terms of observing time than imaging ones. However, spectroscopy is
significantly richer in terms of available information about astrophysical
processes. Some works have therefore explored if deep neural networks can
infer spectra from images without the need for observing time. Wu & Peek
(2020) showed that deep neural networks efficiently learn a mapping between
galaxy imaging and spectra. They showed that SDSS spectra can be inferred from
5-band images with very high accuracy. The authors argue that this approach
could be used as an exploration tool for future surveys such as LSST. Holwerda
et al. (2021) followed up on this work by applying the same network to
estimate the spectrum of an AGN and compare it to existing spectra from the
literature. Although the neural network predicted spectrum is in overall good
agreement with the observed ones there are some differences in the strength of
emission lines. This suggests that the ML approach might be an interesting way
of identifying specific types of objects such as AGN and/or as a first order
exploration. It is difficult to use the inferred spectra to analyze the
physical properties though. This is somehow expected since spectra contain
more information than imaging, as previously stated. A similar experiment is
performed by Hansen et al. (2020) who infer kinematic information of galaxies
(velocity dispersion and rotation maps) from single band imaging.

Summary of Deep learning for inferring physical properties of galaxies
• Over the past years, deep learning has been tested as a tool to infer
physical properties of galaxies in large surveys. The most common applications
are: photometric redshift estimation, galaxy structure and stellar populations
and strong lensing modeling. • The main motivation for using deep learning for
these tasks is computational speed. Deep learning is used here as a fast
emulator for existing methods. Overall, the most common conclusion of these
tests is that deep learning approaches achieve state-of-the art performance,
several orders of magnitude faster. • The typical approach used is a
supervised regression (C)NN trained on simulated datasets, for which the
ground truth is known. • A major challenge of these applications is
robustness. Since the models are predominantly trained on simulations, the
representativity of the training set is a major issue. Extrapolation with
neural networks is in general problematic. Therefore, making sure that the
training samples properly cover the inference dataset is a key challenge. •
Uncertainty quantification is also particularly important for this type of
applications. Bayesian Neural Networks and Density estimators are among the
most commonly employed solutions. • As an extension to the derivation of
physical properties, deep learning has also been explored as a tool to
identify new correlations between observable and other physical properties of
galaxies, which are generally not accessible with existing methods. Examples
of this type of application include the inference of phases of galaxy
evolution such as interactions, or the estimation of the dark matter content
of galaxies. • The main motivation is that neural networks are universal
approximators. Deep learning is therefore employed to unveil hidden
correlations using simulation based inference. By definition, the training is
performed on (cosmological) simulations in which all the physical properties
and evolutionary phases of galaxies are accessible. • A key challenge for
these type of approaches is robustness against the representativity of
training sets and domain shifts. The effect of these issues is particularly
dramatic here since cosmological simulations are known to be approximations to
the observed universe. Moreover, in general, pre-deep learning approaches to
compare with do not exist, as opposed to the previous type of application. •
Because deep learning is typically used _blindly_ informed by simulations,
interpretability becomes a key limitation. Solutions based on saliency maps or
symbolic regression - when possible - have been explored. However these
approaches still present a limited informative power. There is significant
room for improvement in the future.

##  5 deep learning for discovery

In this section, we focus on efforts done by the community to use deep
learning as a discovery tool. Applications typically include dimensionality
reduction to visualize complex datasets and identify groups of objects,
anomaly detection to automatically find potentially interesting objects in
large datasets and some early efforts to automatically learn fundamental laws
of physics.

###  5.1 Visualization of large datasets

In addition of increasing in volume, datasets in astrophysics are becoming
increasingly complex and of high dimensionality. Machine Learning can be
employed to visualize datasets in a low dimension space to look for trends and
correlations in the data, which otherwise are difficult to extract. It can
also be used to identify classes of objects that share some properties which
can help with the scientific analysis. These applications use typically
unsupervised learning approaches, as opposed to what has been previously
discussed. In unsupervised learning, data is unlabeled and therefore we seek a
representation of the data instead of a mapping between data points and
labels. We emphasize again that the present review focuses on deep learning
applications and, therefore, we will not describe in detail works using other
ML approaches for data visualization. There exist however a large variety of
techniques which do not involve neural networks and that have been applied to
astronomy. For example, Baron & Ménard (2021) used graph representations to
find structures in imaging and spectroscopic data. The works by Hocking et al.
(2018) and Martin et al. (2020) also explore clustering coupled with graph
representations to group images of galaxies that look similar. Self Organizing
Maps have also been used to represent images of galaxies in the radio domain
(see e.g. Galvin et al., 2020) and for spectral classification (e.g. Rahmani
et al., 2018). Other non neural network based dimensionality reduction
techniques such as Principal Component Analysis (PCA), t-SNE (van der Maaten &
Hinton, 2008) or UMAP (McInnes et al., 2018) are also used in several works to
explore data.

Deep learning offers several approaches for dimensionality reduction and
visualization. The most standard and widely used are Autoencoders, a
particular type of encoder-decoder architectures (see section 3 and Figure 12)
in which the inputs and outputs are identical. The networks therefore simply
learn how to reproduce the input data. However, since these architectures -
which can be deterministic or probabilistic - typically present a bottleneck
at the junction between the encoder and the decoder, they naturally represent