11institutetext: GIR GCME. Departamento de Informática. Universidad de
Valladolid, 47011 Valladolid, Spain.
11email<EMAIL_ADDRESS>22institutetext:
Departamento de Física Teórica, Atómica y Óptica, Universidad de Valladolid,
47011 Valladolid, Spain. 33institutetext: Instituto de Astrofísica e Ciências
do Espaço, Universidade de Lisboa, OAL, Tapada da Ajuda, PT1349-018 Lisbon,
Portugal.

# Detecting the edges of galaxies with deep learning

Jesús Fernández 11 Fernando Buitrago 2233 Benjamín Sahelices 11

Galaxy edges or truncations are low-surface-brightness (LSB) features located
in the galaxy outskirts that delimit the distance up to where the gas density
enables efficient star formation. As such, they could be interpreted as a non-
arbitrary means to determine the galaxy size and this is also reinforced by
the smaller scatter in the galaxy mass-size relation when comparing them with
other size proxies. However, there are several problems attached to this novel
metric, namely, the access to deep imaging and the need to contrast the
surface brightness, color, and mass profiles to derive the edge position.
While the first hurdle is already overcome by new ultra-deep galaxy
observations, we hereby propose the use of machine learning (ML) algorithms to
determine the position of these features for very large datasets. We compare
the semantic segmentation by our deep learning (DL) models with the results
obtained by humans for HST observations of a sample of 1052 massive (Mstellar
¿ 1010 M⊙) galaxies at $z<1$. In addition, the concept of astronomic
augmentations is introduced to endow the inputs of the networks with a
physical meaning. Our findings suggest that similar performances than humans
could be routinely achieved, although in the majority of cases, the best
results are obtained by combining (with a pixel-by-pixel democratic vote) the
output of several neural networks using ensemble learning. Additionally, we
find that using edge-aware loss functions allows for the networks to focus
their optimization on the galaxy boundaries and, therefore, to provide
estimates that are much more sensitive to the presence of neighboring bodies
that may affect the shape of the truncation. The experiments reveal a great
similarity between the semantic segmentation performed by the AI compared to
the human model. For the best model, an average dice of 0.8969 is achieved,
while an average dice of 0.9104 is reached by the best ensemble, where the
dice coefficient represents the harmonic mean between the precision and the
recall. This methodology will be profusely used in future datasets, such as
that of Euclid, to derive scaling relations that are expected to closely
follow the galaxy mass assembly. We also offer to the community our DL
algorithms in the author’s github repository.

###### Key Words.:

Galaxies: evolution – Galaxies: structure – Galaxies: spiral – Galaxies:
fundamental parameters – Galaxies: statistics – Galaxies: general –

##  1 Introduction

Identifying the appropriate method to determine the size of a galaxy is a
longstanding problem in astrophysics since galaxies are diffuse objects
without sharply defined edges. Many different approaches have been adopted in
the last century (see Chamba 2020 for a thorough review). In recent decades,
the effective radius (re, the semi-major axis of the ellipse encompassing half
of the light from a galaxy) has been the most widely used proxy for the size.
The main reasons behind this decision are twofold: on the one hand, its
robustness against different values for signal-to-noise ratio (S/N) and
exposure time and, on the other hand, its connection with galaxy surface
brightness parametric fitting, namely, the Sérsic (1968) functions. However,
taking a half-light radius as the proxy for galaxy size is an arbitrary choice
and its calculation is tightly connected with the galaxy concentration (see
the discussion in the introduction of Trujillo et al. 2020).

The aforementioned Trujillo et al. (2020) suggested R1 (the isomass contour at
1 M⊙/pc2) as a physically motivated size proxy. Considering the two-phase
formation scenario for galaxies (see, e.g., Hilz et al., 2013; Pulsoni et al.,
2021), a first period of in situ star formation should be followed by the
subsequent accretion of galaxy satellites. Therefore, the cessation of that
first stage must be imprinted in the galaxy outskirts, as there is a spatial
limit for reaching the gas density threshold enabling star formation (Schaye,
2004). The gas density translates into a stellar density, assuming a certain
(low) efficiency into converting gas to stars and this is the reason why R1 is
connected with a sudden change in the galaxy properties (surface brightness,
color, and mass profiles) for Milky Way-like galaxies in the local Universe
(Martínez-Lombilla et al., 2019; Díaz-García, S. et al., 2022). However, the 1
M⊙/pc2 threshold seems to be dependent (at least) on the galaxy stellar mass
or (ultimately) on the physical conditions of the star formation episode. To
investigate in depth where these features appear, irrespective of their
surface mass density value, Chamba et al. (2022) and Buitrago & Trujillo
(2023) (with the former study focused in the low-z Universe and the latter at
intermediate, z $<$ 1, redshifts) detected these sudden drops or changes in
the galaxy outer profiles, studying the evolution on their location with
stellar mass and redshift. The series of observational tests conducted in
these articles, together with the reduction in the scatter in the mass-size
relation by a factor of 2-2.5, solidified this metric as a physically
motivated means to define galaxy sizes. Very interestingly, these edges could
be identified with the previously known galaxy truncations (van der Kruit,
1979) and, as such, we utilize these two names interchangeably in the present
work.

Galaxy truncations have initially been identified for edge-on galaxy disks
whose sizes did not seem to change despite integrating the images for longer
exposure times (van der Kruit & Searle, 1981a, b). It is to noteworthy that
this is not at variance with the fact of galaxies being fuzzy objects, because
there are always stars and light beyond this distance due to the existence of
a stellar halo or stellar migration. Nevertheless, they had not been
considered useful size proxies in the past because of the low surface
brightness (LSB) levels where they appear ($\mu$ ¿ 26-27 mag/arcsec-2 in
10$\times$10 arcsec apertures Martín-Navarro et al., 2012, 2014; Trujillo &
Fliri, 2016). This situation could be changed with the arrival of future
telescope facilities, especially the large-aperture synoptic telescopes that
will be able to routinely reach very deep surface brightness levels ($\mu$ ¿
28-30 mag/arcsec-2 Euclid Collaboration et al., 2022). This research has been
branded the LSB regime of galaxy formation and evolution (Duc et al., 2015;
Mihos, 2019). However, if we are to extract the maximum information from these
incoming images, our endeavor is not only about building the right telescopes
and other instruments, but also to develop the next generation software able
to deal with the millions and even billions of objects to be studied, while
simultaneously achieving high accuracy in our feature extraction.

As this is a pattern-recognition problem, machine learning (ML) is especially
suited for tackling these problems in spite of the diversity of galaxy shapes.
Deep learning (DL), and specifically convolutional neural networks (CNN), have
shown their power in similar big data projects focused on astrophysical and
extragalactic studies (Huertas-Company et al., 2015; Domínguez Sánchez et al.,
2018; Hausen & Robertson, 2020; Vega-Ferrero et al., 2021, to name a few).

The DL technique that we present here goes beyond the algorithmic segmentation
performed by other codes such as Source-Extractor (Bertin & Arnouts, 1996) or
NoiseChisel (Akhlaghi & Ichikawa, 2015; Akhlaghi, 2019) because of the fact
that several pieces of evidence (1D or 2D surface brightness and color and
mass profiles) must be taken into account in order to derive the truncation
position. Generative networks like U-Nets Ronneberger et al. (2015) or
variational autoencoders (VAE, Kingma & Welling 2013) are capable of
generating pixel level segmentation and separate the background noise from the
information of the galaxy. Specific axis and edge detection techniques have
been recently proposed (Soria et al., 2020) along with novel DL models such as
EfficientNet (Tan & Le, 2019), ResNet (He et al., 2016, 2019), DenseNet (Huang
et al., 2017), FastFCN (Wu et al., 2019), GatedSCNN (Takikawa et al., 2019) or
Mask RCNN (He et al., 2020) to perform this task. Other proposal (González et
al., 2018) rely on the use of object detection networks such as YOLO (Redmon
et al., 2016) for automated galaxy detection and classification. Other works
(Walmsley et al., 2020) have suggested leveraging bayesian DL techniques to
investigate morphological features of galaxies. Such an approach provides an
avenue to incorporate uncertainty into the process of edge estimation.

In the present work, we take a first step on the characterization of
truncations and galaxy edges with ML and with the relatively small ($>$1000)
albeit meaningful sample from Buitrago & Trujillo (2023), which will be the
base for the framework that will be developed for the forthcoming Euclid
satellite (Laureijs et al., 2011). Of course, this approach could also be
extended for other future enterprises such as the Vera Rubin Observatory’s
Legacy Survey of Space and Time (LSST, Ivezić et al., 2019) or the ARRAKIHS
satellite111https://www.cosmos.esa.int/documents/7423467/7423486/ESA-F2-ARRAKIHS-
Phase-2-PUBLIC-v0.9.2.pdf/61b363d7-2a06-1196-5c40-c85aa90c2113?t=1667557422996
among others. The tools to be created will become open-source in the following
repository222https://github.com/jesusferigl and our aim is that they could be
applied to different fits images and galaxy surveys than the ones that we took
as a basis for our study.

The paper structure is the following: Section 2 summarizes the parent imaging
and the adopted methodology in order to infer galaxy edges or truncations,
Section 3 describes the models we built to analyze our data, Section 4 details
our results and Section 5 reports our conclusions. Hereafter, our assumed
cosmology is $\Omega_{m}$=0.3, $\Omega_{\Lambda}$=0.7 and H0=70 km s-1 Mpc-1.
We used the Chabrier (2003) initial mass function (IMF), unless otherwise
stated. Magnitudes are provided in the AB system (Oke & Gunn, 1983).

##  2 Dataset

The parent sample comes from Buitrago & Trujillo (2023), where the reader can
find all the details from our galaxy sample and associated data. Here, we
summarize its most important characteristics, highlighting the necessary
changes to adapt our previous datasets to become the inputs for our ML
algorithms.

The images for our study come from the CANDELS fields (the best trade-off
between area and depth for HST observations), namely, the CANDELS survey
(Grogin et al., 2011; Koekemoer et al., 2011) for GOODS-N, UDS, EGS and
COSMOS, HLF (Illingworth et al., 2016) for GOODS-S, while for HUDF we used the
dataset in Beckwith et al. (2006) for the optical and ABYSS (Borlaff et al.,
2019) in the near-infrared (NIR) bands. We took the galaxy IDs, photometric
masses, and spectroscopic redshifts (only those flagged as the ones with top
quality) for all galaxies from the CANDELS public
catalogs333https://archive.stsci.edu/prepds/candels/ (Santini et al., 2015;
Stefanon et al., 2017; Nayyeri et al., 2017; Barro et al., 2019), whereas to
increase our sample, we also took galaxies with spectroscopic redshifts from
other sources (van der Wel et al., 2016; Straatman et al., 2018; Lilly et al.,
2009; Damjanov et al., 2019; van der Wel et al., 2021). Then, we selected
galaxies with Mstellar $>$ 1010 M⊙ and zspec $<$ 1.1 (in order to maximize the
S/N of our targets). As we are looking for features regarding the limits of
gas density enabling star formation, truncations should be more easily visible
in disk galaxies (Martín-Navarro et al., 2012) and, as such, we only selected
this type of galaxies according the criteria in Huertas-Company et al. (2015).
In this publication, these galaxies are later subdivided as pure disks (DISK),
disks with central spheroids (DISKSPH), and irregular disks (DISKIRR). We use
these subclasses in the present work. Lastly, any galaxy displaying an
observational artifact was also removed from our final sample. In summary, our
initial mass-selected sample of 3460 galaxies was reduced to 1052 galaxies
after applying all the aforementioned selection criteria with integrated
apparent magnitudes of 17.5 $<$ HAB $<$ 22.5.

The observed images for the galaxies in our sample were cut with a fixed stamp
size of 12$\times$12 arcsec, ergo 400$\times$400 pixels for the optical ACS
images, for the F606W (V) and F814W (I) filters, while 200$\times$200 pixels
for the near-infrared WFC3 images, for the F125W (J) and F160W (H) filters;
however, these latter ones were later resampled to the near-infrared
resolution to match the same pixel scale. A representative sky background
value was subtracted from each image (details in Buitrago & Trujillo 2023).
From these postage stamps we derived $g$, $r$, $i,$ and $z$ Sloan Digital Sky
Survey (SDSS)-restframe equivalent images in the same manner as the profiles
in Buitrago et al. (2017) or Buitrago & Trujillo (2023), but this time on a
pixel-by-pixel basis. Basically, we interpolated linearly between contiguous
observed bands converted to surface brightness using the formula:

$\mu=-2.5\log_{10}(\text{flux})+zeropoint+5\log_{10}(\rm\text{pixel\leavevmode\nobreak\
scale}),$

while taking as a reference the mean wavelength of each filter444See
http://svo2.cab.inta-csic.es/svo/theory/fps3/index.php. These SDSS-restframe
images were also corrected by galaxy inclination according to the model in
Trujillo et al. (2020), along with the extinction from our galaxy555Values
taken from https://ned.ipac.caltech.edu/extinction_calculator and cosmological
dimming. To avoid numerical errors, in the case of negative pixel values in
the images to be interpolated, we substituted their values with the noise
level of each image. This limiting magnitude was calculated utilizing 104
1$\times$1 arcsec2 boxes to match the characteristic spatial scales of the
galaxy in our sample, and then deriving the 3$\sigma$ clipped standard
deviations of those.

We obtained color images (g-i, g-r, g-z, i-z, r-i, r-z) from these SDSS-
restframe equivalent images by subtracting one from another. Overly high ($>$
5) or low values ($<$ -1) were trimmed out setting not-a-number (NaN) values
instead. Finally, these SDSS-restframe images and their color images were
combined pixel-by-pixel using the recipes in Roediger & Courteau (2015) for
retrieving mass images. We note that by using these methods, we obtained 24
mass images (one per combination of base filter and color) and thus our final
mass image is the median of all of them in units of M⊙ pc-2. Again, pixels
displaying too high ($>$ 105) or too low ($<$ 10-3) values were discarded.

Summarizing, our final inputs to our neural networks are all the previously
enumerated 200$\times$200 pixel images, namely, the HST observations, the
SDSS-restframe equivalent images, and the color and mass images. Our labels
are also 200$\times$200 pixel images set to 1 for all the pixels belonging to
the central galaxy and 0 otherwise. Any neighbor overlapping the galaxy has
also been set to 0 after a thorough masking. The borders between the central
galaxy and the rest of the pixels are our galaxy edges or truncations (see
Figure 1). They are defined as ellipses whose semi-major axis is the
truncation position or edge radius found in Buitrago & Trujillo (2023), with
the axis ratio and position angle being representative of those found for the
galaxy outer parts in the H-band. Mind this is approximate labelling, i.e. the
truncation might be found at slightly different radial distances azimuthally
or they might be erased in certain parts due to previous galaxy merging. These
aspects will be expanded in future publications (Raji et al. in prep.,
Fernández et al. in prep.). Finally, it is to note that our sample (both
galaxies and their associated truncation values) differs somewhat from the one
in Buitrago & Trujillo (2023) in terms of the number of objects (four more in
our analysis) and truncation values (96 reassessed different values, with a
median variation of 1.25 kpc). These changes are not important for our
objectives, since our purpose is to create a neural network that is able to
reproduce the decisions of human astronomers based on certain data and inputs;
thus, it is to be expected (and this is indicated by the preliminary results)
that with the new data, the DL models will also be able to adapt with a high
level of precision.

###  2.1 Astronomic augmentations (AAs)

The process of labeling galaxy edges in LSB areas is a complex task that
requires time for an experienced researcher to analyze all the input data. For
these reasons, we did not have access to a large number of galaxies available
for our study. It is well known that one of the main problems derived from the
scarcity of data is overfitting in the training phase. That is why the
application of data augmentation techniques is very relevant to achieve the
best possible learning., namely, performing slight modifications of the input
images to be fed into the networks. In this work, we have used some of the
most standard ones, such as random 90-degree-multiple rotations ($0^{\circ}$,
$90^{\circ}$, $180^{\circ}$, or $270^{\circ}$), flipping the images
(horizontally, vertically, or not at all) and adding Gaussian noise (standard
deviations of 0.01, 0.05, or 0.1).

Nevertheless, we wanted to increase the number of augmentations with the goal
of not only gaining access to more examples for learning features, but also to
change how the network learns as well. We combined the information that
researchers use when detecting truncations and we utilized this data in the
training phase, thus providing extra sets of physically meaningful inputs to
the networks. The use of multiple combinations of all the available data
maximizes the variability of the information used in the training phase,
always referring to the same physical reality, which greatly improves the
robustness of the training. We enumerate these new augmentations here, that we
call “astronomic” augmentations (given their origin), with the corresponding
names to each one of their types: 1) visual: creating RGB images by using all
possible range of observational bands sorted by wavelength. Consequently, the
different possibilities are IJH, VIJ, VIH, and VJH; 2) color: creating all
possible color images: g-i, g-r, g-z, i-z, r-i, and r-z; 3) sloan: creating
b/w and color images with the SDSS-restframe equivalent images: g, r, i, z,
gri, giz, riz, grz; 4) and mass: creating the mass per pixel images, as
detailed in Section 2.

##  3 Models and training

To detect the galaxy edges, we used a DL-based semantic segmentation approach.
Semantic segmentation is aimed at classifying all the pixels in an input,
grouping together those pixels that share the same characteristics or belong
to the same type of object. The aim of the DL methodology developed in this
work is to determine and isolate the bodies of the central galaxies from the
noise and other secondary galaxies. Therefore, each of the different
algorithms attempts to classify the pixels of the images into two categories,
main galaxy, and the rest of the pixels (where this latter category includes
both background noise and secondary galaxies) as seen in Figure 1. Using the
masks generated by the semantic segmentation model, the radius of the
associated truncations can be easily determined. Hence, it is crucial that the
forthcoming DL models exhibit a good performance in the pixels situated near
to the galaxy edges. The current section presents the architecture
configurations of our models along with the training specifications.

Figure 1: Image-mask-truncation equivalence. Left: Image created combining the
V, I, and H bands. Center: associated segmentation mask, where the only active
pixels are those belonging to the main galaxy. Right: Determination of the
truncation from the perimeter of the segmentation mask. Note: the jagged
profile is due to the superposition of a neighboring galaxy.

###  3.1 Architectural configurations

There are several CNN architectures that have been proposed over the years to
solve the problem of semantic segmentation. The best known examples are U-Nets
(Ronneberger et al., 2015), nested U-Nets (Zhou et al., 2018), or the DeepLab
(Chen et al., 2018) family of architectures. In this work, we decided to use
the U-Nets as the base scheme from which to derive the models for estimating
galactic truncations. This type of network is able to achieve reasonable
quality segmentation with a relatively small training sample size, which is a
very relevant restriction in this work. They are composed of two symmetrical
stages. The first one, the contraction stage, is carried out by an encoder and
tries to extract the most relevant features from the inputs (downsampling).
The second one, the expansion stage, is performed by a decoder and
reconstructs the segmentation mask using the previously obtained features. In
order to avoid a vanishing gradient problem (due to the concatenation of many
layers) and to enhance the retrieval of spatial information (that may be lost
during downsampling) skip connections are utilized. These links allow the
cross flow of information between the symmetrical layers of the encoder and
decoder.

Due to their morphology, U-Nets allow for different types of convolutional
neural networks to be used as encoders. In this work, three families of CNNs
are tested: ResNet (He et al., 2016), EfficientNet (Tan & Le, 2019) and
DenseNet (Huang et al., 2017). These families correspond to well-known models
that have obtained state-of-the-art results in a wide range of computer vision
tasks. Specifically, from the ResNet family, we used the 18-hidden layers and
the 50-hidden layers models. From the EfficientNet family, variants B1, B2,
and B6 have been used. Lastly, from the DenseNet family, the DenseNet-161 and
DenseNet-201 architectures have been used as encoders. These choices reflect
the requirement to include encoders of different sizes, considering the size
as the number of training parameters of the model. All the encoders have been
initialized with pretrained weights from the ImageNet dataset (Deng et al.,
2009). The use of pretrained deep neural networks usually improves the
performance and generalization capacity. The main reason is that they use
previously acquired knowledge related to high and low-level features (edges,
textures) that are useful for many computer vision tasks, such as the
identification of galaxy truncations.

AAs are represented as different images generated for each galaxy, as
explained in the Section 2.1. The simplest solution to combine the AAs in the
training phase is to concatenate the generated images in some dimension of the
input tensors. However, in this work, we decide to exploit the concept of
grouped convolutions as a way to solve the problem. Grouped convolutions were
primarily introduced in Krizhevsky et al. (2017), as part of the well-known
AlexNet architecture, with the main objective of reducing memory needs during
training. However, further studies showed that grouped convolutions have a
clear advantage in improving the representation of the training data than
classical convolutions, as for instance in Xie et al. (2017). Therefore, in
this work, grouped convolutions on the different inputs are used to extract
independently the most relevant spatial information from each image and,
subsequently, a second network (U-Net with pre-trained encoder) is used to
estimate the galactic truncations from all this information (Figure 2).

Figure 2: Example of the architecture designed for experiments with Mass-type
AAs. The architecture consists of two stages. In the first stage, grouped
convolutions process each image in isolation and then a U-Net with a pre-
trained encoder uses all the information from the different AAs to infer the
truncations.

###  3.2 Hyperparameters and training specifications

In order to estimate the true error rate of the DL methods, we split the
original dataset into two subsets, training and test, using the hold-out
technique. The number of observations making up the training set is 952, while
the number of instances reserved for the test set is 100. Furthermore, the
training set is divided into two subsets, training and validation, based on a
proportion of 80 % and 20 %, respectively. The validation set is used at the
end of each epoch to assess learning and make decisions about training
hyperparameters for the next epoch.

We train our DL models using a weighted categorical cross-entropy (CCE) loss
function. Weights are necessary in order to counterbalance the strong
imbalance of categories present in the images. Otherwise, there is a high risk
that models only predict the noise category for each pixel, due to the high
dominance of these pixels with respect to the ones representing the central
galaxy. Weights for the loss function have been estimated empirically. Their
values are $\omega_{n}$ = 0.075, for the class representing noise, and
$\omega_{g}$ = 0.925, for the class representing the main galaxy. This way,
hits and misses in pixels belonging to galaxies are weighted more heavily than
in pixels belonging to noise. The batch size was set to 32, learning rate was
fixed to $10^{-3}$, and the maximum number of training epochs is 1000. The
Adam optimizer, from Kingma & Ba (2015), was used to update the networks’
weights.

In order to mitigate the possible overfitting, which commonly occurs in the
domain of deep convolutional neural networks, three regularization techniques
were used: data augmentation, early stopping, and ridge penalty. For data
augmentation each training image, before being introduced into the network, is
randomly rotated $0^{\circ}$, $90^{\circ}$, $180^{\circ}$, or $270^{\circ}$
and, once this transformation is applied, the image is randomly kept without
further changes, mirrored around the abscissa axis or mirrored with respect to
the ordinate axis. Thus, there are 11 possible combinations that induce images
different from the original one, making it much more difficult for the network
to memorize the training examples. After this process, random noise from a
normal distribution is added to each training example. The noise added to each
image comes from a normal distribution with $\mu=0$ and $\sigma$ chosen
randomly (0.01, 0.05, or 0.1) for each electromagnetic band of each image.

The second regularization technique is the selection of the model that
maximizes the performance over the validation set, which is very similar in
interpretation to early stopping. Finally, ridge regularization, also known as
$l_{2}$ or the Tikhonov regularization, is used. This technique, originally
presented in Hoerl & Kennard (1970) for a linear regression problem, imposes a
penalty on the estimated model coefficients. This way, when applied to a DL
optimization problem, a penalty is added to the loss function. Equation 1
shows the application of the technique, where $J(y,h_{\theta}(x))$ represents
the traditional loss function between the labels ($y$) and the model outputs
($h_{\theta}(x)$) and $\lambda\sum_{i=1}^{p}w_{i}^{2}$ represents the penalty
(sum of the squared model weights driven by a $\lambda$ penalty factor). The
effect of applying this technique is to reduce proportionally the value of all
the model’s weights, without reducing them to zero. The result is that the
flexibility of the relationship between the model predictions and the target
variable is reduced, acting as a powerful technique for avoiding overfitting.

$L(y,h_{\theta}(x))=J(y,h_{\theta}(x))+\lambda\sum_{i=1}^{p}w_{i}^{2}$ (1)

For the performance evaluation of the model we used three metrics: precision,
recall, and Sørensen-dice or dice coefficient (Equation 2), also known as
$\mathcal{F}1$ score in some contexts. We did not use a pixel-level hit rate
because it is not sensitive to the cost of misclassification. The dice
coefficient takes into account the cost of misclassification and adapts to the
imbalance present in the images. It represents the harmonic mean between the
precision, which measures the influence of false positives, and the recall,
which measures the influence of false negatives. A perfect precision could be
achieved by correctly predicting a single pixel as a galaxy, predicting all
other pixels as noise. A perfect recall could be achieved by predicting all
pixels in the image as a galaxy. Therefore, it is necessary to perform a
weighting of both metrics. The harmonic mean, being more demanding and
pessimistic than its arithmetic and geometric counterparts, guarantees that it
accurately reflects the quality of the results obtained by our DL models.

$\text{Dice}=\frac{2\times\text{precision}\times\text{recall}}{\text{precision}+\text{recall}}=\frac{2\times
TP}{2\times TP+FP+FN}$ (2)

##  4 Segmentation results

This section shows the results obtained with the different DL models trained
according to the specifications described in the Section 3. All the results
shown in this section have been obtained on the test set of galaxies. This set
consists of 100 randomly chosen galaxies representative of the whole dataset
and which have not been used during the training phase. First of all, the
different encoders without AAs are evaluated in order to check the feasibility
of our proposal. Next, the impact of the AAs is evaluated and a complete
analysis of the best performing model is carried out. Finally, the results of
a complete ensemble analysis are shown.

###  4.1 Base experiment

The base experiment uses the seven models and the training specifications
described in Sections 3.1 and 3.2 without applying AAs, that is, the training
is done with the images of galaxies formed by the H, J, and I bands. Table 1
shows dice, precision, and recall for each of the seven encoders embedded in
the U-Net’s backbone. The general trend reveals that recall is higher than
precision for all experiments. This behavior indicates that networks tend to
give slightly optimistic estimates of galaxy size, placing the truncations
further away from the center of the galaxy than they actually are. The high
value for recall shows that the vast majority of the image regions belonging
to the galaxies of interest are correctly inferred, however, the slightly
lower value of precision shows that there are areas outside the galaxy that
have been incorrectly labeled. The dice index shows reasonably good
segmentation qualities (¿ 0.88 for all encoders), oscillating narrowly between
the different schemes. The worst value is reached by the Efficient-B2 encoder,
and the best one is reached by DenseNet-161.

Encoder | Dice | Precision | Recall
---|---|---|---
ResNet18 | 0.8849 | 0.8531 | 0.9438
ResNet50 | 0.8832 | 0.8340 | 0.9591
EfficientNetB1 | 0.8867 | 0.8543 | 0.9461
EfficientNetB2 | 0.8809 | 0.8437 | 0.9475
EfficientNetB6 | 0.8835 | 0.8604 | 0.9351
DenseNet161 | 0.8926 | 0.8768 | 0.9316
DenseNet201 | 0.8867 | 0.8592 | 0.9397
Average value | 0.8851 | 0.8545 | 0.9433
Table 1: Results for the base experiment. Last row shows the average values
for all encoders. Dice, precision, and recall values are represented in
reddish, bluish, and greenish tones respectively, where lighter colors
represent higher values and darker colors represent lower values.

###  4.2 Other experiments: Effects of AAs

In this section, we analyze the variability induced in the learning process by
the introduction of the different AAs which are classified into four groups:
1) the visual group is made up of RGB files created with the four possible
combinations of filters V, I, J, and H, always assigning the bluer channels to
shorter wavelength bands (IJH, VIJ, VIH, and VJH). In other words, the visual
group is formed by the base experiment plus the images formed by the VIJ, VIH
and VJH combinations; 2) the color group is composed of the visual group to
which six images are added, each one created by each of the six possible
colors (g-i, g-r, g-z, i-z, r-i, and r-z); 3) the sloan group is formed by the
visual group plus eight images formed both by the combination of the SDSS
filters ordered by wavelength (gri, giz, riz, and grz) and by each of the
filters individually (g, r, i, and z); 4) finally, the mass group is formed by
the visual group together with the mass image.

Table 2, analogous to Table 1, shows results over the test set in terms of
dice, precision, and recall indicators per encoder embedded in the U-Net’s
backbone and AA (visual, mass, color, and sloan). All average values of the
dice coefficient are higher than in the base experiment case, showing that the
introduction of the AAs leads to an improvement in the performance of the DL
models. The average recall takes lower values with AAs compared with the base
experiment. However, having obtained higher mean dice values means that the
precision values have to rise considerably. This behavior could suggest that
the estimates of the truncations with AAs are somewhat more conservative,
placing them closer to the centre of the galaxy than in the case of the base
experiment. In other words, AAs cause a substantial decrease in the number of
image regions misdetected as galaxy, maintaining a very good proportion of
pixels correctly categorized as a galaxy, leading to a marked improvement in
the overall quality of the inferences.

((a)) ResNet50 with base scheme.

((b)) EfficientNetB6 with visual AA.

((c)) DenseNet161 with sloan AA.

((d)) ResNet18 with color AA.

((e)) EfficientNetB2 with mass AA.

Figure 3: Learning curves showing the loss evolution over train and validation
datasets of some DL configurations using the base scheme (a) and different AAs
(b, c, d, e).

According to the results shown in Table 2, mass (visual+mass) displays the
best results, visual ranks second, while color and sloan have the worst
results. Given that both color and sloan include visual augmentations, it may
be assumed that these two sets of images do not improve the accurate
prediction of truncations by DL models. In any case, the performance offered
by all configurations, regardless of the inputs or architectures used, is
remarkably uniform. None of the experiments obtained notably worse results
than the rest. All configurations produce quality results, which are product
of a stable training. This is illustrated in Figure 3, which shows the
evolution of the loss over the epochs on the training and validation datasets.
Five combinations of DL models and AAs representative of the general behavior
have been selected. It can be seen that the learning patterns are consistent
and do not show large oscillations. Moreover, there is no obvious overfitting,
which is evidence of a good convergence towards optimal models. This provides
evidence of stable training processes and, therefore, quality of results
across all AAs and architectures.

Encoder | Visual | Sloan | Color | Mass
---|---|---|---|---
| Dice | Precision | Recall | Dice | Precision | Recall | Dice | Precision | Recall | Dice | Precision | Recall
ResNet18 | 0.8894 | 0.8769 | 0.9269 | 0.8876 | 0.8718 | 0.9293 | 0.8888 | 0.8530 | 0.9502 | 0.8949 | 0.9036 | 0.9080
ResNet50 | 0.8911 | 0.8803 | 0.9260 | 0.8858 | 0.8589 | 0.9401 | 0.8943 | 0.8898 | 0.9206 | 0.8897 | 0.8779 | 0.9258
EfficientNetB1 | 0.8884 | 0.8534 | 0.9490 | 0.8854 | 0.8797 | 0.9182 | 0.8819 | 0.8452 | 0.9451 | 0.8924 | 0.8616 | 0.9470
EfficientNetB2 | 0.8805 | 0.8549 | 0.9373 | 0.8836 | 0.8533 | 0.9420 | 0.8886 | 0.8625 | 0.9403 | 0.8890 | 0.8802 | 0.9230
EfficientNetB6 | 0.8856 | 0.8657 | 0.9324 | 0.8914 | 0.8590 | 0.9478 | 0.8845 | 0.8574 | 0.9397 | 0.8935 | 0.8867 | 0.9237
DenseNet161 | 0.8893 | 0.8955 | 0.9067 | 0.8867 | 0.8727 | 0.9264 | 0.8869 | 0.8834 | 0.9173 | 0.8959 | 0.8820 | 0.9304
DenseNet201 | 0.8969 | 0.8863 | 0.9274 | 0.8862 | 0.8434 | 0.9550 | 0.8830 | 0.8991 | 0.8937 | 0.8928 | 0.8701 | 0.9379
Average Value | 0.8887 | 0.8733 | 0.9294 | 0.8867 | 0.8627 | 0.9370 | 0.8869 | 0.8701 | 0.9296 | 0.8926 | 0.8803 | 0.9280
Table 2: Results with AAs over the test sample. Last row shows the average
values for all encoders. Color coding corresponds to the one used in Table 1,
i. e., lighter colors represent better results.

Nevertheless, beyond the fact that some AAs induce better or worse results,
the variability and singularities they introduce in the training process can
be used a posteriori to build a democratic system that takes advantage of the
majorities. It would therefore be interesting to demonstrate statistically
that the different AAs influence the learning process (and, thus, the
performance) of the different U-Nets. In order to demonstrate this assertion,
we carried out a Friedman two-way analysis of variance by ranks non-parametric
test, as used in Sheskin (2000). To this end, we transformed the dice results
obtained over the test set in Table 2 into a rank-order format for each model,
as can be seen in Table 3, having $n=7$ experiments and $k=5$ experimental
conditions in which every experiment is assessed.

Encoder | Base | Visual | Sloan | Color | Mass
---|---|---|---|---|---
ResNet18 | 5 | 2 | 4 | 3 | 1
ResNet50 | 5 | 2 | 4 | 1 | 3
EfficientNetB1 | 3 | 2 | 4 | 5 | 1
EfficientNetB2 | 4 | 5 | 3 | 2 | 1
EfficientNetB6 | 5 | 3 | 2 | 4 | 1
DenseNet161 | 2 | 3 | 5 | 4 | 1
DenseNet201 | 3 | 1 | 4 | 5 | 2
$\sum_{i}R_{i}$ | 27 | 18 | 26 | 24 | 10
$\overline{R}$ | 3.8571 | 2.5714 | 3.7143 | 3.4286 | 1.4286
Table 3: Rank-ordered results for AAs. Penultimate row shows the sum of all
the rankings obtained for each AA. Last row shows average rank for each AA.

$\chi_{r}^{2}=\frac{12}{nk\left(k+1\right)}\left[\sum_{j=1}^{k}\left(\sum_{i=1}^{n}R_{i,j}\right)^{2}\right]-3n\left(k+1\right)=11.4286.$
(3)

The null hypothesis represents the absence of a significant difference between
any of the $k=5$ population medians. To reject the null hypothesis the
computed value $\chi_{r}^{2}$ (see Equation 3) must be equal to or greater
than the tabled critical chi-square value at the prespecified level of
significance, 5 % in this work, with $k-1$ degrees of freedom ($df$). For
$df=4$, the reported critical 5 % chi-square value is 9.4877. Since the
computed value $\chi_{r}^{2}$ is greater than the chi-square value, the
alternative hypothesis is supported at the prespecified level, showing that
training with different AAs influence the U-Nets performance for at least two
of the five experimental conditions.

Differences | Result
---|---
$|\sum R_{\text{Base}}-\sum R_{\text{Visual}}|$ | —27 - 18— = 9
$|\sum R_{\text{Base}}-\sum R_{\text{Sloan}}|$ | —27 - 26— = 1
$|\sum R_{\text{Base}}-\sum R_{\text{Color}}|$ | —27 - 24— = 3
$|\sum R_{\text{Base}}-\sum R_{\text{Mass}}|$ | —27 - 10— = 17
$|\sum R_{\text{Visual}}-\sum R_{\text{Sloan}}|$ | —18 - 26— = 8
$|\sum R_{\text{Visual}}-\sum R_{\text{Color}}|$ | —18 - 24— = 6
$|\sum R_{\text{Visual}}-\sum R_{\text{Mass}}|$ | —18 - 10— = 8
$|\sum R_{\text{Sloan}}-\sum R_{\text{Color}}|$ | —26 - 24— = 2
$|\sum R_{\text{Sloan}}-\sum R_{\text{Mass}}|$ | —26 - 10— = 16
$|\sum R_{\text{Color}}-\sum R_{\text{Mass}}|$ | —24 - 10— = 14
Table 4: Difference scores between the pairs of sums of ranks of Table 3.
Green rows show the significantly different AAs.

To find out which of the AAs are significantly different from each other, it
is necessary to conduct comparisons contrasting specific AAs between them. For
this purpose, several sources (Daniel, 1990; Siegel & Castellan, 1988)
describe a comparison procedure based on the application of the Bonferroni-
Dunn method. Using Equation 4, we can compute the minimum required difference
(designated as $CD_{F}$) between the sums of the ranks of any two AAs at the
prespecified level of significance (5 % in this work):

$CD_{F}=z_{\text{adj}}\sqrt{\frac{nk(k+1)}{6}}=2.58\sqrt{35}=15.2635.$ (4)

When comparisons are not planned beforehand, the $z_{\text{adj}}$ value of
Equation 4 can be established by dividing the maximum tolerated familywise
Type I error by the total number of comparisons conducted (see Sheskin, 2000).
In this case, the error is 5 % and the total number of comparisons to be
performed is 10, giving the quotient of both 0.005 (directional hypothesis).
In the table of the normal distribution, the $z$ value that corresponds to
that quotient is 2.58. Therefore, the significant comparisons are those whose
ranks differences are equal to or greater than the value computed in Equation
4. The significant comparisons found are highlighted in green in Table 4.
Thus, we can state that there is a significant difference between the results
of the U-Nets when using base and mass AAs and when using sloan and mass AAs.
Once it can be concluded that learning processes differ due to the
introduction of the AAs, we can take advantage of this variability to develop
the concept of democratic ensemble learning in Section 4.5.

###  4.3 Best model performance

This section shows the results obtained by the best performing model, that is,
the one using a DenseNet-201 neural network as encoder and trained with visual
astronomical augmentations. We isolated the performance of the model in terms
of different characteristics into which the test sample can be grouped.
Specifically, these characteristics are the galaxy morphology (disk, diskirr,
and disksph), the exposure field (egs, goodsn, cosmos, uds, and goodss), the
redshift and the size. Figure 4 shows the results in terms of dice, precision
and recall estimators. Figure 5 shows the precision-recall (P-R) curves. We
did not use ROC curves due to the imbalance of our data. Each P-R curve
includes both the point where the 0.5 threshold is located (the one used to
give a binary classification between galaxy and noise) and the point where the
optimal dice index is reached. The results show that the morphological type
for which the worst performance is obtained is disksph, as expected given of
the fuzzier nature of these objects (Figure 4(a)). Ultimately, the disk and
diskirr morphologies perform best. Differences between precision and recall
are much more marked for the diskirr morphology than for the disk. This
indicates that the estimated truncations are much more balanced in the latter
case, leading to a higher number of false positives for diskirr galaxies. In
any case, the results obtained on diskirr galaxies are highly valuable. The
reason is that this morphology is characterized by the presence of asymmetric
features (Huertas-Company et al., 2015), which makes the precise estimation of
its edges more challenging than for disk-type galaxies. Although the dice
index for the 0.5 threshold is higher for the diskirr morphology, the quality
over the whole spectrum of thresholds is higher for inferences of disk
galaxies, as can be seen in Figure 5(a). For all morphologies, all thresholds
that induce an optimal dice index are greater than 0.5. This may indicate that
the model is overconfident, and it is necessary for the network to have a high
certainty that a pixel belongs to the galaxy class.

((a))

((b))

((c))

((d))

Figure 4: Metrics over the test set by galaxy morphology (a), field of view
(b), redshift (c), and size (d) for our best performing model. The first plot
(a) shows mean dice, precision, and recall for diskirr, disksph, disk, and all
morphologies. The second plot (b) displays the same metrics for all our galaxy
fields, namely egs, goodsn, cosmos, uds, and goodss. The third and fourth
plots shows the same metrics for the five redshift bins and the five size
quintiles the galaxy sample is split into.

((a))

((b))

((c))

((d))

Figure 5: Precision-recall curves, along with the area under the curves (AUC),
over the test set by galaxy morphology (a), galaxy field (b), redshift (c),
and size (d). The categories for each plot are the same as in Figure 4. The
dot symbol represents the location of the threshold 0.5 (the one used for
binary classification). The star symbol shows the optimal threshold that
maximizes the dice index.

((a))

((b))

Figure 6: Greatest (a) and least (b) accurate inferences by morphology for the
best-performing model on the test dataset. First row shows the images of the
galaxies. Second row displays the overlap of the predicted and real masks,
where yellow, red and green zones represent true positives, false positives,
and false negatives, respectively. The third row shows the residuals, where
reddish and greenish tones represent negative and positive residuals, and the
value (from light to dark) indicates their magnitude (from higher to lower).
The last row shows the predicted and real truncations superimposed on the
images.

Assessing the metrics by field of exposure (Figure 4(b)), the performance is
very similar for all of them except for the goodss field where we retrieve the
worst values. The images in this field are, on average, the deepest (i.e.,
with the least noise) that we are analyzing. Therefore, our network better
discerns the central galaxy from any other neighbor, and so our network
becomes more conservative in detecting pixels belonging to the central body.
This leads to a smaller recall, which, in turn, produces a reduction in the
dice index. Looking at the P-R curves (Figure 5(b)), it has been observed that
the threshold inducing an optimal dice coefficient is higher than 0.5 for egs,
goodsn, cosmos, and uds. However, in the case of goodss, this threshold is
below the threshold used for binary classification. This result suggests that
while the model exhibits a high level of confidence, with respect to the first
fields it is cautious when categorizing pixels as galaxies in the goodss
field. This may be another reason that explains the worse behavior of the
model on the goodss field than on the rest of the fields.

There is no marked variation in the quality of the estimated truncations for
the different redshift values (Figure 4(c)). The dice index does not decrease
uniformly as galaxies become more distant, which is reasonable considering the
high quality of the whole data sample for all redshifts. Nevertheless, it can
be seen that the trends in precision and recall are opposite. While the former
increases along with the redshift, the latter gradually decreases. This means
that as galaxies become more distant, the DL model infers masks with fewer
false positives and more false negatives, namely, the truncations are tighter,
but some areas that are actually galaxies are not detected as such. To explain
this, it should be noted that the higher the redshift, the higher the effect
of the cosmological dimming decreasing the contrast between galaxy and
background noise pixels, thereby making the 2D galaxy surface brightness
profiles appear fainter. Consequently, the number of false positive pixels is
progressively smaller and thus increasing the precision. The P-R curves and
their inner area also reflect the same results for the different redshift
levels (Figure 5(c)). Similar to what was observed for the different
morphologies, all thresholds inducing an optimal dice index are greater than
0.5, suggesting some overconfidence.

Finally, when the results are analyzed by galaxy size, it can be found that
the performance varies considerably (Figure 4(d)). The pattern seems to
indicate that, on average, the dice index improves with increasing galaxy
size. Furthermore, while the recall evolution appears to be slightly more
stable, the main improvement lies in the precision. One of the possible
reasons may be that as a galaxy gets larger, it is easier for the DL model not
to make false positive errors, which leads to an increase in the precision.
For very large galaxies (Q5) the results worsen. This may be because in such
galaxies, the amount of noise that appears in the stamp is very small, with
most of the image occupied by the galaxy and the galaxy-noise transition. As a
result, it is more difficult for the model to locate the truncation and the
performance worsens. The information provided by the P-R curves aligns with
this pattern, as galaxies of larger sizes consistently yield to better results
when using different thresholds (Figure 5(d)). Again, all thresholds inducing
an optimal dice index are greater than 0.5, reflecting the ease of the model
to be very confident in its predictions about the galaxy class.

Figure 6 shows the most and least accurate inferences by galaxy morphology.
From the first row to the last one, it can be appreciated the images, the
superimposed real and predicted masks, the residuals and the superimposed real
and predicted truncations. For the masks, true positives, false positives and
false negatives are represented by yellowish, reddish, and greenish tones,
respectively. The information provided by the residuals is richer. Reddish
tones represent those areas where the real mask is 0, greenish tones represent
those areas where the real mask is 1 and black tones indicate residuals very
close to 0, that is, there are almost identical behaviors seen between the
predictions and labels. Analyzing the best cases (Figure 6(a)) it can be seen
that for the three types of morphologies, the predictions reveal a high degree
of similarity with respect to the ground truths (dice $>0.94$). Regarding the
masks, reddish and greenish tones are barely visible, leading to very high
values of both precision and recall. Concerning the residuals, it can be seen
that the largest ones (less dark colors) appear in the boundary regions of the
galaxies, that is, the zones close to the truncation. This event reflects that
the model has more doubts about the category of these pixels, which is a
desirable behavior. For the worst cases (Figure 6(b)), the neural network
infers the shape of the truncation remarkably well. However, the penalty
suffered in the dice index is caused by an optimistic estimation of the
limits. This can be easily seen by analyzing both the overlap of the masks and
the residuals. For the former, reddish tones predominate around the yellowish
areas, causing an increase in the number of false positives and penalizing the
precision. For the latter, residuals in bright red clearly indicate an
overconfidence of the model about the presence of galaxy, estimating much
larger truncations than those reflected in the masks.

Lastly, we trained this best performance model with the final masks in
Buitrago & Trujillo (2023) as explained toward the end of Section 2.1. The
numbers are very similar to those already quoted in the paper: the mean dice
is 0.8933, the mean precision is 0.8685, and the mean recall is 0.9411. We
also conducted this experiment with other architectures and AAs. The outcome
of all these tests is that there is not a substantial change by using or not
these new labels, reassuring us of the stability of our results.

###  4.4 Prioritizing boundary proximity: A close look at edge-aware loss
functions

The DL models presented so far have been optimized using a loss function that
equally weights all pixels belonging to the same class ($\omega_{g}=0.925$ for
galaxy class and $\omega_{n}=0.075$ for noise class). Thus, a pixel located at
the center of the galaxy will have the same importance as a pixel placed just
inside the truncation. Similarly, a noise pixel that is far away from the
central body will be worth the same as a noise pixel just outside the
truncation. This situation might potentially induce the network to minimize
the loss by focusing its attention on those areas that are easy to learn, so
that precise behavior is not necessary in the most doubtful areas to
categorize (those close to the truncations). In order to overcome this
situation, in this section we explore the performance of some loss functions
that aim to ensure that DL models give more importance to those regions that
are close to the boundaries of the galaxies.

We propose to use a variant of the CCE loss, where the weighting is performed
per pixel rather than per class. Two different formulas are suggested for
assigning the weight of each pixel. The first one ($\text{Pixel-CCE}_{1}$ from
now on) is in Equation 5, where $d((i,j),c)$ represents the Euclidean distance
from the pixel $(i,j)$ to the point $c$, while $\mathcal{C}$ represents the
set of all pixels that make up the truncation contour and $\alpha$ acts as a
weighting factor. This factor marks the weight intensity in the neighboring
areas of the truncation compared to the more distant zones, as can be seen in
(Figure 7). Weights belonging to the internal and external parts of the
truncation are scaled independently.

$W_{i,j}=1-D(i,j)^{\alpha}\text{, where }D(i,j)=\min_{\forall
c\in\mathcal{C}}d((i,j),c)$ (5)

The second one ($\text{Pixel-CCE}_{2}$ hereafter) is in Equation 6. The terms
$d((i,j),c)$ and $\mathcal{C}$ have the same meaning as in the previous
equation, while $\delta$ acts as an offset. That offset determines how broad
the truncation-surrounding area that is highly weighted is (Figure 7).

$W_{i,j}=\log{\frac{1}{\min_{\forall c\in\mathcal{C}}d((i,j),c)+\delta}}$ (6)

Figure 7: IJH image with its associated mask (left) and different weight
matrices using the $\text{pixel-CCE}_{1}$ (center) and $\text{pixel-CCE}_{2}$
(right) losses.

((a))

((b))

Figure 8: Probability maps obtained by class-CCE, $\text{pixel-CCE}_{1}$ and
$\text{pixel-CCE}_{2}$ losses for two galaxies (a, b) of the test set. Dark
red tones show confidence values close to 1 (high probability of galaxy),
while dark blue tones reflect values close to 0 (high probability of noise).

In order to assess the performance of the presented loss functions, we used
the configuration that yielded the best-performing model (DenseNet-201 encoder
with visual AAs). We trained the model using different configurations for both
pixel CCE approaches. In the first case, we launched six experiments, varying
the values of the hyperparameter $\alpha$. In the second case, we ran nine
experiments, using different values for the hyperparameter $\delta$. The
results obtained on the test set can be seen in Table 5. In the case of
$\text{pixel-CCE}_{1}$, none of the six configurations obtained a better dice
index than the obtained using class-CCE. For $\text{pixel-CCE}_{2}$, two of
the nine configurations were able to improve the results. In both cases, and
for all configurations tested, no clear pattern is observed indicating that
these functions are a better alternative to class-CCE. When the $\text{pixel-
CCE}_{1}$ $\alpha$ hyperparameter increases, the results seem to worsen
markedly. This suggests that excessive weighting of the zones adjacent to the
truncation, in relation to the rest of the parts, leads to a decrease in
performance. On the other side, when the $\text{pixel-CCE}_{2}$ $\delta$
hyperparameter is varied, the results do not appear to follow a linear trend.
The only two configurations that lead to better results than class-CCE are
achieved with $\delta=0.9$ and $\delta=1.7$. However, the intermediate values
between these deltas show an oscillating and irregular behavior.

Although the average performance of both losses does not seem to improve the
behavior of class-CCL, sometimes their use may be worthwhile. In Figure 8, we
illustrate a significant example of the difference between the different loss
functions. We chose two representative galaxies from the test dataset in which
differences between the approaches can be seen. It can be observed the
probability maps obtained by class-CCE and by all the tested $\text{pixel-
CCE}_{1}$ and $\text{pixel-CCE}_{2}$ variants. In the first case (Figure
8(a)), we can notice the presence of a foreign body on the right side of the
galaxy. In the respective mask, it can be seen the impact this object has on
the truncation. The class-CCE configuration reflects some doubt about whether
that zone should be categorized as a main galaxy or as noise (yellowish
tones). However, several $\text{pixel-CCE}_{1}$ ($\alpha=1,2,3,4$) and
$\text{pixel-CCE}_{2}$ ($\delta=0.5,0.7,1.1,1.3,1.5,1.9$) configurations
reflect more clearly the notion that this zone is not a central galaxy and it
corresponds to noise (blue-greenish and bluish colors). Furthermore, although
it does not appear to affect the truncation since it is more distant, the
galaxy also has a neighboring object on its left side. The class-CCE
configuration interprets this object as an obvious area of noise (very dark
blue tone), which does not impact the shape of the truncation. However,
certain configurations (primarily $\text{pixel-CCE}_{1}$ with $\alpha=2$, and
to a lesser extent $\text{pixel-CCE}_{1}$ with $\alpha=3,5$ and $\text{pixel-
CCE}_{2}$ with $\delta=0.7,0.9,1.7$) seem to be more sensitive to this object,
reflecting lower probabilities of belonging to the central galaxy and thus
affecting the shape of the truncation. This behavior, which, in terms of the
dice coefficient, leads to a decrease in performance (due to the absence of
this object in the mask), may be more interesting and reliable when estimating
truncations in images with numerous clustered bodies. In the second case
(Figure 8(b)), the galaxy also has a neighboring object, this time in the
upper-left part. This object affects its boundaries and, consequently, the
shape of its truncation. Once again, the probability map provided by Class-CCE
is uniform in that area and is not affected by the presence of the object.
However, several configurations of the edge-aware alternatives (primarily
$\text{pixel-CCE}_{1}$ with $\alpha=2,3,4,5,6$ and $\text{pixel-CCE}_{2}$ with
$\delta=0.5$) detect the presence of this object. Consequently, they give much
lower probabilities of galaxy membership (greenish tones) to the pixels in
that area. In a general sense (for both galaxies), it can be observed that for
the majority of configurations, edge-aware alternatives produce probability
maps that are less pronounced. This can be verified by observing fewer sharp
transitions between intense reddish areas and dark blue areas. More
uncertainty may be beneficial in situations where a range of predictions needs
to be considered, ranging from scenarios where high confidence in the pixels
being galaxies is required (resulting in smaller truncations) to scenarios
where greater flexibility in dealing with model false positives is allowed
(resulting in more lenient truncations).

Loss | Config | Dice | Precision | Recall
---|---|---|---|---
Class-CCE |  | 0.8969 | 0.8863 | 0.9274
$\text{Pixel-CCE}_{1}$ | $\alpha=1$ | 0.8819 | 0.8939 | 0.8980
$\alpha=2$ | 0.8930 | 0.9023 | 0.9048
$\alpha=3$ | 0.8911 | 0.9023 | 0.9028
$\alpha=4$ | 0.8887 | 0.8986 | 0.9034
$\alpha=5$ | 0.8850 | 0.8659 | 0.9293
$\alpha=6$ | 0.8804 | 0.8622 | 0.9256
$\text{Pixel-CCE}_{2}$ | $\delta=0.3$ | 0.8824 | 0.8796 | 0.9159
$\delta=0.5$ | 0.8960 | 0.8957 | 0.9176
$\delta=0.7$ | 0.8967 | 0.8836 | 0.9309
$\delta=0.9$ | 0.8988 | 0.8891 | 0.9291
$\delta=1.1$ | 0.8901 | 0.8894 | 0.9148
$\delta=1.3$ | 0.8931 | 0.9103 | 0.8973
$\delta=1.5$ | 0.8926 | 0.8826 | 0.9257
$\delta=1.7$ | 0.9022 | 0.9086 | 0.9137
$\delta=1.9$ | 0.8880 | 0.8830 | 0.9172
Table 5: Results over the test set using the edge-aware loss functions and the
encoder DenseNet-201 with visual AAs. Color-coding corresponds to the one used
in Table 1.

###  4.5 Approaching human expert behavior via democratic ensemble learning

In this section, the use of ensemble learning is proposed in order to improve
the quality of the segmentation performed by DL models. Ensemble learning
combines more than one base algorithm to obtain, through an aggregation
mechanism, a unitary prediction. Intuitively, decision-making improves when
diverse opinions are contrasted. Thus, the algorithms to be combined need to
be diverse, that is, they need to make errors on different instances. The
experimental grid, in which different AAs are combined with different
encoders, results in the derivation of highly diverse models. Therefore, the
35 segmentation networks that have been presented throughout the work are used
as the base algorithms. To keep the computation within reasonable limits, it
was decided to combine the base models in groups of three. This gives rise to
6545 unique ensembles, which are combined by the democratic criterion of the
majority vote (mode), as can be seen in Figure 9 We note that this way of
aggregating predictions forces to take an odd number of base models to form
the ensemble.

Figure 9: Designed ensemble architecture. Predictions from the individual
models are combined using a majority vote criterion to provide a final output.

In the Figure 10(a), it can be seen how 64.75 % of the constructed ensembles
exceed the highest dice index obtained with any of the individual models
(namely DenseNet-201 trained with visual AAs). These data show the
effectiveness of the ensemble learning approach to approximate human behavior
in finding the galaxy truncations. In fact, the approach can be linked to the
creation of a committee of expert astronomers in which the opinion of the
majority prevails. The best configuration found is made up of the models
EfficinetNet-B6, ResNet-18, and DenseNet-161, all of them trained with mass-
type AAs, reaching a dice index of 0.9104, a precision of 0.9016, and a recall
of 0.9353. On the other hand, Figure 10(b) shows that the AA that, on average,
induces ensembles with a higher dice index is mass. The visual, color, and
sloan type AAs (in that order) are placed later. At the end, we have the
ensembles consisting of at least one base model. In order to quantify the
improvement caused by the participation of each AA in the ensemble, we order
the results from lowest to highest (base $<$ sloan $<$ color $<$ visual $<$
mass) and compute the improvement ratio caused by each AA. This way, the sloan
improvement over base is 1.1 per ten thousand, the color improvement over
sloan is 3.3 per ten thousand, the visual improvement over color is 2.2 per
ten thousand, and the mass improvement over visual is 1.8 per thousand. The
improvement induced by the mass-type AA is one order of magnitude greater than
any of the other improvements, which indicates that this type of AA is clearly
the one that induces the best performance. These results are coherent with
those shown in the Sections 4.1 and 4.2.

((a))

((b))

Figure 10: Metrics summarizing the behavior of the ensembles. The left plot
(a) shows the sorted dice index values for all built ensembles. The right plot
(b) displays, for each AA, the average dice index of all assemblies in which
that AA participates at least one time.

Analogously to the best individual model, Figure 11 shows the dice, precision,
and recall estimators by galaxy morphology, field of view, redshift, and size
for the best ensemble configuration. In order to simplify comparisons with the
best individual model, its results have also been plotted in Figure 11 with
dashed more-transparent lines. Analyzing the morphology, in terms of the dice
index, disk performs the best, followed by disksph and diskirr, respectively.
It is worthwhile to analyze the oscillation of precision and recall for the
different morphologies. In the case of disk and disksph, the recall is higher
than the precision, however, for diskirr the opposite happens. The inferred
masks for diskirr type galaxies show far fewer false positives than for the
other morphologies, while the inferred masks for disk and disksph type
galaxies show fewer false negatives. Comparing the results with those of the
best single model, the main change occurs over diskirr galaxies. While the
individual model had a high recall and a lower precision, the ensemble model
greatly increases the recall and decreases the precision. This means that the
truncations offered by the ensemble for these types of galaxies are much more
optimistic than those offered by the individual alternative. For disksph and
disk type galaxies, the ensemble improves the results proportionally,
improving both precision and recall. Regarding the field of view, uds achieves
the best results, as already found for the best individual model in Section
4.3. Similarly, the worst performance is also obtained for the goodss type
galaxies. goodsn, cosmos, and egs are in the mid-range, with a dice index that
goes from best to worst, respectively. While for the goodsn, cosmos, and
goodss fields, the recall is much higher than the precision, for the egs and
uds fields, the predictions are much more balanced in terms of false positives
and negatives. Performance, as in the case of the best single model, shows
signs of invariance to redshift. Again, the precision seems to increase as the
galaxies become more distant, while the recall decreases. In terms of size,
the conclusions are similar to the case of the best individual model. The best
results are achieved for medium-size galaxies (in the third and fourth
quintiles). For the smallest and largest galaxies, lower quality inferences
are obtained.

((a))

((b))

((c))

((d))

Figure 11: Metrics over the test set by galaxy morphology (a), field of view
(c), redshift (c), and size (d) for the best ensemble configuration. The
categories for each plot are the same as in Figure 4. Dashed (more
transparent) lines show the results for the best individual model.

In order to explore the performance of ensemble learning when combining more
than three models, we have selected some relevant cases. The results can be
seen in Table 6. First, we built the ensembles consisting of all the models
that have been trained with the same astronomic augmentation (Table 12(a)).
Thus, for each AA, the results of the seven encoders were combined. The
results are consistent with those obtained in Tables 1 and 2. The best
ensemble is the one that uses all seven encoders trained with mass-type AAs.
The worst results were still obtained when no AA is used (base case). Between
them, ranked from best to worst, we have: visual, color, and sloan AAs.
Nevertheless, all five ensembles improve the results of the best individual
model, showing the ability of this technique to enhance and complement the
results of models that make diverse errors. We have also calculated the
ensemble consisting of all available individual models (35). Although the
result is excellent, achieving a dice of 0.9054 and surpassing the best
individual model, it does not overcome the best ensemble formed by three
models (dice of 0.9104). In order to explore more possibilities, we also
constructed the ensembles formed by all the models using the same encoder
(Table 12(b)). Thus, for each encoder, the five versions that have been
trained using different AAs are combined. All ensembles, except the one formed
by the EfficientNetB2 encoder, outperform the best single model. The best
results are obtained by DenseNet encoder ensembles (DenseNet201 and
DenseNet161), followed by the ResNet ones (ResNet18 and ResNet50) and,
finally, by the EfficientNet ones (EfficientNetB1, EfficientNetB6, and
EfficientNetB2).

Appendix A shows the truncations predicted by the best combination of ensemble
learning for the 100 galaxies in the test set.

Combination | Dice | Precision | Recall
---|---|---|---
All base | 0.8973 | 0.8631 | 0.9537
All visual | 0.9026 | 0.8857 | 0.9401
All sloan | 0.8997 | 0.8737 | 0.9479
All color | 0.9006 | 0.8790 | 0.9438
All mass | 0.9097 | 0.8941 | 0.9424
All | 0.9054 | 0.8823 | 0.9484
((a))

Combination | Dice | Precision | Recall
---|---|---|---
All ResNet18 | 0.9029 | 0.8823 | 0.9437
All ResNet50 | 0.9008 | 0.8781 | 0.9441
All EfficientNetB1 | 0.8985 | 0.8674 | 0.9515
All EfficientNetB2 | 0.8951 | 0.8677 | 0.9467
All EfficientNetB6 | 0.8972 | 0.8725 | 0.9445
All DenseNet161 | 0.9049 | 0.8927 | 0.9353
All DenseNet201 | 0.9035 | 0.8829 | 0.9432
((b))

Table 6: Metrics over the test set obtained by the ensembles, grouping the
individual models by AA (a) and by encoder (b). In Table 12(a), each row
collects the results of the ensemble formed by the seven models (one for each
encoder) that have been trained with the corresponding AA. Last row shows the
ensemble made up of the 35 trained models. In Table 12(b), each row shows the
results of the ensemble formed by the five models (one for each AA) that have
been trained with the same encoder. Color coding corresponds to the one used
in Table 1.

##  5 Conclusions

Galaxies are thought to be diffuse objects without clear borders and, as a
result, their nature impedes measurements of physically motivated galaxy
sizes. However, LSB observations have shown that galaxy outskirts display
sudden drops in their mass density profiles, which also display a counterpart
in their surface brightness profiles. These features, which trace a perimeter
much closer to our intuitive idea of the end of a galaxy, are linked with the
radial location of the gas density threshold enabling star formation and
actually produce size-related scaling relations with a much smaller scatter
(Trujillo et al., 2020). Chamba et al. (2022) and Buitrago & Trujillo (2023)
also identified them with the galaxy truncations detected from the seminal
works by Piet van der Kruit (van der Kruit, 1979). Current ultra-deep
observations and next-generation synoptic telescopes will reveal these
features for millions of galaxies. Because of the numerous pieces of evidence
needed to derive the positions of these edges, and considering the inherent
pattern-recognition nature of this problem, using DL (in general) and CNNs (in
particular) appears to be the only way forward to address these measurements
for the swarm of data to come in the near future.

Based on the detections of these edges or truncations for a sample of 1052
massive (Mstellar $>$ 1010 M⊙) disk galaxies at zspec $<$ 1.1 in the CANDELS
fields (Buitrago & Trujillo, 2023), we built a series of U-Nets that make use
of convolutional neural networks as encoders. Our final objective is both to
construct an algorithm able to detect efficiently these LSB features as well
as understanding the best inputs to achieve our aims. This last point has to
do with the fact that our neural networks do not only analyze the HST parent
sample images, but they are also feed with ancillary data, namely, SDSS-
restframe equivalent images, as well as color and mass images (see Section 2).
In this way, we are providing the same information as the one given to humans
when inferring the positions of these edges or truncations. We called these
transformations of the original images “astronomic augmentations,” as they are
similar to standard machine learning augmentations, but they are based on our
astronomical knowledge. All average values of the dice coefficient improve in
comparison with those of the base experiment when making use of such
“astronomic augmentations.”

The several experiments that we carried out, namely, visual (HST RGB images),
sloan (visual + SDSS bands and SDSS RGB images), color (visual + SDSS colors),
and mass (visual + mass images), display an inherent variability in their
outcome metrics for the various U-Net architectures we utilize (see Section 4
and Table 1 and Table 2). This ensures that our neural networks learn in a
different manner based on different inputs (see Section 3.1). Among all the
AAs, the best results are recurrently obtained when using the mass-type ones,
which actually makes physical sense as it is the channel that directly refers
to the drop in the galaxies’ stellar mass density profiles. Even though we
highlight the best model performance (see Section 4.3), which we obtained for
the encoder DenseNet-201 trained with visual AAs (dice: 0.8969, precision:
0.8863, recall: 0.9274; see Table 2), we want to emphasize that the best
results are retrieved for ensembles where different neural networks
democratically vote the best solution in a pixel-by-pixel mode (see Figure
10). Using this method, 64.75 % of the ensembles improve the highest dice
value in comparison with our best individual model.

This best ensemble is the combination of EfficientNet-B6, ResNet-18, and
DenseNet-161, all of them (very remarkably, as it is similar to what human
astronomers need) trained with mass AAs. The metrics in this case are dice:
0.9104, precision: 0.9016, and recall: 0.9353. Finally, we also performed a
series of tests checking our results versus morphological disk type, depth
(i.e., galaxy field), redshift, and size, in order to ensure their robustness
(see Section 4.3).

Galaxy edges or truncations will permit us to obtain physically motivated
galaxy sizes for images fulfilling the adequate depth and data reduction
requirements. Our study shows that ML is able to reach a similar performance
to that of humans and it will be mandatory to apply this technique for the
billions of galaxies and tens of thousands of square degrees to be imaged in
the future. In addition, the extraction of these features could be beneficial
to other studies of galaxy morphology, as they will help identify any excess
of light in the galaxy outer parts, thus indicating the presence of stellar
haloes or minor merging. In summary, analyses with state-of-the-art tools as
the one we are presenting here, will pave the way for solidifying galaxy edges
or truncations as a viable physically motivated proxy for galaxy sizes.

###### Acknowledgements.

We acknowledge the referee for the insightful comments and feedback on the
paper. We have used extensively the following software packages: TOPCAT
(Taylor, 2005), ALADIN (Bonnarel et al., 2000), SEGMENTATION MODELS
(Iakubovskii, 2019), MATPLOTLIB (Hunter, 2007) and PYTORCH (Paszke et al.,
2019). JF and FB acknowledge the support from the grant PID2020-116188GA-I00
by the Spanish Ministry of Science and Innovation, while FB also acknowledges
PID2019-107427GB-C32. This work has made use of the Rainbow Cosmological
Surveys Database, which is operated by the Centro de Astrobiología (CAB/INTA),
partnered with the University of California Observatories at Santa Cruz
(UCO/Lick,UCSC). Based on zCOSMOS observations carried out using the Very
Large Telescope at the ESO Paranal Observatory under Programme ID:
LP175.A0839. We also thank Jesús Vega-Ferrero, Ignacio Trujillo, Nushkia
Chamba and Helena Domínguez-Sánchez for their advice.

## References

  * Akhlaghi (2019) Akhlaghi, M. 2019, arXiv e-prints, arXiv:1909.11230
  * Akhlaghi & Ichikawa (2015) Akhlaghi, M. & Ichikawa, T. 2015, ApJS, 220, 1
  * Barro et al. (2019) Barro, G., Pérez-González, P. G., Cava, A., et al. 2019, ApJS, 243, 22
  * Beckwith et al. (2006) Beckwith, S. V. W., Stiavelli, M., Koekemoer, A. M., et al. 2006, AJ, 132, 1729
  * Bertin & Arnouts (1996) Bertin, E. & Arnouts, S. 1996, A&AS, 117, 393
  * Bonnarel et al. (2000) Bonnarel, F., Fernique, P., Bienaymé, O., et al. 2000, A&AS, 143, 33
  * Borlaff et al. (2019) Borlaff, A., Trujillo, I., Román, J., et al. 2019, A&A, 621, A133
  * Buitrago & Trujillo (2023) Buitrago, F. & Trujillo, I. 2023, Strong size evolution of disc galaxies since z = 1: Readdressing galaxy growth using a physically-motivated size indicator
  * Buitrago et al. (2017) Buitrago, F., Trujillo, I., Curtis-Lake, E., et al. 2017, MNRAS, 466, 4888
  * Chabrier (2003) Chabrier, G. 2003, PASP, 115, 763
  * Chamba (2020) Chamba, N. 2020, Research Notes of the American Astronomical Society, 4, 117
  * Chamba et al. (2022) Chamba, N., Trujillo, I., & Knapen, J. H. 2022, A&A, 667, A87
  * Chen et al. (2018) Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence, 40
  * Damjanov et al. (2019) Damjanov, I., Zahid, H. J., Geller, M. J., et al. 2019, ApJ, 872, 91
  * Daniel (1990) Daniel, W. 1990, Applied Nonparametric Statistics, Duxbury advanced series in statistics and decision sciences (PWS-KENT Pub.)
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., et al. 2009, in 2009 IEEE conference on computer vision and pattern recognition, Ieee, 248–255
  * Domínguez Sánchez et al. (2018) Domínguez Sánchez, H., Huertas-Company, M., Bernardi, M., Tuccillo, D., & Fischer, J. L. 2018, MNRAS, 476, 3661
  * Duc et al. (2015) Duc, P.-A., Cuillandre, J.-C., Karabal, E., et al. 2015, MNRAS, 446, 120
  * Díaz-García, S. et al. (2022) Díaz-García, S., Comerón, S., Courteau, S., et al. 2022, A&A, 667, A109
  * Euclid Collaboration et al. (2022) Euclid Collaboration, Borlaff, A. S., Gómez-Alvarez, P., et al. 2022, A&A, 657, A92
  * González et al. (2018) González, R. E., Muñoz, R. P., & Hernández, C. A. 2018, Galaxy detection and identification using deep learning and data augmentation
  * Grogin et al. (2011) Grogin, N. A., Kocevski, D. D., Faber, S. M., et al. 2011, ApJS, 197, 35
  * Hausen & Robertson (2020) Hausen, R. & Robertson, B. E. 2020, ApJS, 248, 20
  * He et al. (2020) He, K., Gkioxari, G., Dollár, P., & Girshick, R. 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence, 42
  * He et al. (2016) He, K., Zhang, X., Ren, S., & Sun, J. 2016, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778
  * He et al. (2019) He, T., Zhang, Z., Zhang, H., et al. 2019, in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2019-June (IEEE), 558–567, bag of tricks usado desde fast.ai
  * Hilz et al. (2013) Hilz, M., Naab, T., & Ostriker, J. P. 2013, MNRAS, 429, 2924
  * Hoerl & Kennard (1970) Hoerl, A. E. & Kennard, R. W. 1970, Technometrics, 12
  * Huang et al. (2017) Huang, G., Liu, Z., Maaten, L. V. D., & Weinberger, K. Q. 2017, in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Vol. 2017-January, denseNet
  * Huertas-Company et al. (2015) Huertas-Company, M., Gravet, R., Cabrera-Vives, G., et al. 2015, ApJS, 221, 8
  * Huertas-Company et al. (2015) Huertas-Company, M., Pé rez-González, P. G., Mei, S., et al. 2015, The Astrophysical Journal, 809, 95
  * Hunter (2007) Hunter, J. D. 2007, Computing In Science & Engineering, 9, 90
  * Iakubovskii (2019) Iakubovskii, P. 2019, Segmentation Models Pytorch, https://github.com/qubvel/segmentation_models.pytorch
  * Illingworth et al. (2016) Illingworth, G., Magee, D., Bouwens, R., et al. 2016, arXiv e-prints, arXiv:1606.00841
  * Ivezić et al. (2019) Ivezić, Ž., Kahn, S. M., Tyson, J. A., et al. 2019, ApJ, 873, 111
  * Kingma & Ba (2015) Kingma, D. P. & Ba, J. 2015, CoRR, abs/1412.6980
  * Kingma & Welling (2013) Kingma, D. P. & Welling, M. 2013
  * Koekemoer et al. (2011) Koekemoer, A. M., Faber, S. M., Ferguson, H. C., et al. 2011, ApJS, 197, 36
  * Krizhevsky et al. (2017) Krizhevsky, A., Sutskever, I., & Hinton, G. E. 2017, Communications of the ACM, 60, alexNet
  * Laureijs et al. (2011) Laureijs, R., Amiaux, J., Arduini, S., et al. 2011, ArXiv e-prints [arXiv:1110.3193]
  * Lilly et al. (2009) Lilly, S. J., Le Brun, V., Maier, C., et al. 2009, ApJS, 184, 218
  * Martín-Navarro et al. (2012) Martín-Navarro, I., Bakos, J., Trujillo, I., et al. 2012, MNRAS, 427, 1102
  * Martín-Navarro et al. (2014) Martín-Navarro, I., Trujillo, I., Knapen, J. H., Bakos, J., & Fliri, J. 2014, MNRAS, 441, 2809
  * Martínez-Lombilla et al. (2019) Martínez-Lombilla, C., Trujillo, I., & Knapen, J. H. 2019, MNRAS, 483, 664
  * Mihos (2019) Mihos, J. C. 2019, arXiv e-prints, arXiv:1909.09456
  * Nayyeri et al. (2017) Nayyeri, H., Hemmati, S., Mobasher, B., et al. 2017, ApJS, 228, 7
  * Oke & Gunn (1983) Oke, J. B. & Gunn, J. E. 1983, ApJ, 266, 713
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., et al. 2019, in Advances in Neural Information Processing Systems 32 (Curran Associates, Inc.), 8024–8035
  * Pulsoni et al. (2021) Pulsoni, C., Gerhard, O., Arnaboldi, M., et al. 2021, A&A, 647, A95
  * Redmon et al. (2016) Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. 2016, You Only Look Once: Unified, Real-Time Object Detection
  * Roediger & Courteau (2015) Roediger, J. C. & Courteau, S. 2015, MNRAS, 452, 3209
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., & Brox, T. 2015, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9351, uNet
  * Santini et al. (2015) Santini, P., Ferguson, H. C., Fontana, A., et al. 2015, ApJ, 801, 97
  * Schaye (2004) Schaye, J. 2004, ApJ, 609, 667
  * Sérsic (1968) Sérsic, J. L. 1968, Atlas de galaxias australes
  * Sheskin (2000) Sheskin, D. J. 2000, Handbook of PARAMETRIC and NONPARAMETRIC STATISTICAL PROCEDURES. SECOND EDITION (London: CHAPMAN & HALL/CRC)
  * Siegel & Castellan (1988) Siegel, S. & Castellan, N. 1988, Nonparametric Statistics for the Behavioral Sciences, McGraw-Hill international editions statistics series (McGraw-Hill)
  * Soria et al. (2020) Soria, X., Riba, E., & Sappa, A. 2020, in Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020
  * Stefanon et al. (2017) Stefanon, M., Yan, H., Mobasher, B., et al. 2017, ApJS, 229, 32
  * Straatman et al. (2018) Straatman, C. M. S., van der Wel, A., Bezanson, R., et al. 2018, ApJS, 239, 27
  * Takikawa et al. (2019) Takikawa, T., Acuna, D., Jampani, V., & Fidler, S. 2019, in Proceedings of the IEEE International Conference on Computer Vision, Vol. 2019-October
  * Tan & Le (2019) Tan, M. & Le, Q. V. 2019, in 36th International Conference on Machine Learning, ICML 2019, Vol. 2019-June
  * Taylor (2005) Taylor, M. B. 2005, in Astronomical Society of the Pacific Conference Series, Vol. 347, Astronomical Data Analysis Software and Systems XIV, ed. P. Shopbell, M. Britton, & R. Ebert, 29
  * Trujillo et al. (2020) Trujillo, I., Chamba, N., & Knapen, J. H. 2020, MNRAS, 493, 87
  * Trujillo & Fliri (2016) Trujillo, I. & Fliri, J. 2016, ApJ, 823, 123
  * van der Kruit (1979) van der Kruit, P. C. 1979, A&AS, 38, 15
  * van der Kruit & Searle (1981a) van der Kruit, P. C. & Searle, L. 1981a, A&A, 95, 105
  * van der Kruit & Searle (1981b) van der Kruit, P. C. & Searle, L. 1981b, A&A, 95, 116
  * van der Wel et al. (2021) van der Wel, A., Bezanson, R., D’Eugenio, F., et al. 2021, ApJS, 256, 44
  * van der Wel et al. (2016) van der Wel, A., Noeske, K., Bezanson, R., et al. 2016, ApJS, 223, 29
  * Vega-Ferrero et al. (2021) Vega-Ferrero, J., Domínguez Sánchez, H., Bernardi, M., et al. 2021, MNRAS, 506, 1927
  * Walmsley et al. (2020) Walmsley, M., Smith, L., Lintott, C., et al. 2020, Monthly Notices of the Royal Astronomical Society, 491
  * Wu et al. (2019) Wu, H., Zhang, J., Huang, K., Liang, K., & Yu, Y. 2019, arXiv e-prints, arXiv:1903.11816
  * Xie et al. (2017) Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. 2017, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 5987–5995
  * Zhou et al. (2018) Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J. 2018, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11045 LNCS

##  Appendix A Test set inferences

Here, we present the inferences for all the galaxies in the test set (one row
per galaxy) using the best ensemble configuration. The first column shows the
RGB color image created from the concatenation of the I, J and H bands. The
second one represents the overlap between the label and the output of the
neural networks (true positives in yellow, true negatives in dark blue, false
positives in red, and false negatives in green). The third column depicts the
RGB image with the human-created (cyan) and the model-created (coral)
truncations superimposed. The fourth column shows the output of the ensemble
in a probabilistic way. Thus, the pixels represented in dark blue indicate
that the three models that make up the ensemble have categorized them as
noise. The violet pixels indicate that while two of the models have
categorized them as noise, one has categorized them as a galaxy. Pixels
represented in dark green indicate that while one model has categorized them
as noise, two have categorized them as galaxies. Finally, the pixels
represented in light green show total agreement for the categorization as a
galaxy. Therefore, we note that the mask predicted by the ensemble corresponds
with the area colored in light green plus the area colored in dark green.

Figure 12: RGB images (first column), superimposed predicted and real masks
(second column), superimposed predicted and real truncations (third column),
and probabilistic ensemble outputs for galaxies 1 to 4 in the test set. Figure
13: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 5 to 8 in the test set. Figure 14:
RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 9 to 12 in the test set. Figure
15: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 13 to 16 in the test set. Figure
16: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 17 to 20 in the test set. Figure
17: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 21 to 24 in the test set. Figure
18: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 25 to 28 in the test set. Figure
19: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 29 to 32 in the test set. Figure
20: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 33 to 36 in the test set. Figure
21: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 37 to 40 in the test set. Figure
22: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 41 to 44 in the test set. Figure
23: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 45 to 48 in the test set. Figure
24: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 49 to 52 in the test set. Figure
25: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 53 to 56 in the test set. Figure
26: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 57 to 60 in the test set. Figure
27: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 61 to 64 in the test set. Figure
28: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 65 to 68 in the test set. Figure
29: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 69 to 72 in the test set. Figure
30: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 73 to 76 in the test set. Figure
31: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 77 to 80 in the test set. Figure
32: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 81 to 84 in the test set. Figure
33: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 85 to 88 in the test set. Figure
34: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 89 to 92 in the test set. Figure
35: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 93 to 96 in the test set. Figure
36: RGB images (first column), superimposed predicted and real masks (second
column), superimposed predicted and real truncations (third column), and
probabilistic ensemble outputs for galaxies 97 to 100 in the test set.