# PDiscoNet: Semantically consistent part discovery for fine-grained
recognition

Robert van der Klis1 Stephan Alaniz2 Massimiliano Mancini3 Cassio F. Dantas4,6
Dino Ienco4,6 Zeynep Akata2 Diego Marcos5,6
1WUR 2University of Tübingen 3University of Trento 4INRAE/UMR TETIS 5Inria
6University of Montpellier

###### Abstract

Fine-grained classification often requires recognizing specific object parts,
such as beak shape and wing patterns for birds. Encouraging a fine-grained
classification model to first detect such parts and then using them to infer
the class could help us gauge whether the model is indeed looking at the right
details better than with interpretability methods that provide a single
attribution map. We propose PDiscoNet to discover object parts by using only
image-level class labels along with priors encouraging the parts to be:
discriminative, compact, distinct from each other, equivariant to rigid
transforms, and active in at least some of the images. In addition to using
the appropriate losses to encode these priors, we propose to use part-dropout,
where full part feature vectors are dropped at once to prevent a single part
from dominating in the classification, and part feature vector modulation,
which makes the information coming from each part distinct from the
perspective of the classifier. Our results on CUB, CelebA, and PartImageNet
show that the proposed method provides substantially better part discovery
performance than previous methods while not requiring any additional hyper-
parameter tuning and without penalizing the classification performance. The
code is available at https://github.com/robertdvdk/part_detection

##  1 Introduction

Commonly used approaches to inspect a deep learning model’s inner workings
yield a saliency map that indicates which regions contributed the most to the
output [5, 29]. If the model seems to focus on image regions that are known to
be irrelevant, (e.g. the background or the wrong object), it becomes clear
that the model has picked up on spurious correlations and cannot be trusted.
This observation could then be used to improve future iterations of the model,
for instance, by eliminating or compensating for the detected spurious
correlations. However, this type of approach offers little information when
the model provides an incorrect answer but the saliency map suggests that it
is attending to the correct image regions.

Figure 1: Our PDiscoNet extracts semantically consistent parts, without any
part annotations, and reasons on these parts before combining the results into
a final fine-grained classification output.

Other approaches aim at modifying the model architecture itself in order to
ensure that the provided explanation actually reflects the decision process of
the model [10, 6]. In particular, the saliency map explanation can be enriched
by dividing it in multiple semantically interpretable parts, mimicking the
traditional approaches of tackling fine-grained visual categorization (FGVC),
in which image-level part annotations were leveraged [22] in order to help the
model differentiate between similar classes by helping it focus on the
relevant parts. In this manner, we have more information to judge the adequacy
of the model’s reasoning: even if the correct object is highlighted, we will
be suspicious of the result if the part map that the model generally
associated to the head of a bird seems to highlight the feet in one particular
image. We thus posit that a model that classifies images based on just a few
discriminative regions that are semantically consistent across images would be
more interpretable than one which highlights the whole object, as one can
immediately visualise the parts of the image that have been attended to and
interpret their semantics across images. By inspecting a few images and their
corresponding detected parts, we can easily assign semantic meaning to each
part (e.g., bird beak, vehicle wheel) and judge whether the correct parts are
being detected in a new image.

Even if the model correctly assigns high saliency to the object of interest,
we will know to mistrust the result in case the discovered part semantics are
not respected. This way of interpreting the models has an additional advantage
over post-hoc methods in that we can be more certain that the model only uses
information from the indicated regions. Such models have also been shown to be
more robust; irrelevant parts are filtered out by only looking at the
discovered discriminative regions, which can have a positive impact on
generalization capability and thus robustness to occlusion [37] and
adversarial attacks [30].

Discovering meaningful and discriminative parts using only image-level class
labels requires the use of additional priors that encode our expectations on
the characteristics of these parts along with a model architecture that allows
for these priors to be implemented. We design a model, based on a
Convolutional Neural Network (CNN) backbone, which discovers discriminative
parts of objects by being forced to use the discovered parts as a bottleneck
for fine-grained classification. The fine-grained setting ensures a high level
of similarity between classes, enabling the possibility of discovering
semantic parts that are shared by multiple classes. In our part bottleneck,
class logits are independently extracted from each of the discovered parts
before being combined for the final classification, along with a dropout layer
that affects whole parts at a time, ensures that all discovered parts are
relevant to classification.

##  2 Related work

Fine-grained recognition FGVC is a classification setting in which objects of
multiple sub-classes of the same super-class are present, thus constituting a
challenging task where subtle intra-class and large inter-class variation need
to be simultaneously addressed [33]. Solving fine-grained tasks usually
requires one to closely inspect the object for the telltale differences
between closely related classes. Traditional methods exploit shared keypoints
[21, 22], parts [16, 35], attributes [25, 24], or a pre-segmentation of the
object of interest [4] in order to effectively discriminate between similar
sub-classes, although deep learning approaches using large quantities of data
have since also proved effective [23]. Our PDiscoNet belongs to a family of
approaches that facilitate injecting some of the structure provided by part-
based [14, 17] or attribute-based [13] reasoning via weakly-supervised
learning without part or attribute annotations.

Interpretability via attribution maps In saliency-based attribution methods,
the goal is to highlight important regions of the image that are used by the
network to form its decision. Examples include perturbation-based [27, 28],
activation-based [20, 39], and gradient-based [29, 31] explanations methods.
Despite their popularity, these model-agnostic methods often cannot guarantee
that their explanations are faithful to the model [2]. In contrast, inherently
interpretable models aim to directly expose the decision process of the
network [7, 10]. In this work, we focus on incorporating interpretable
components into the network architecture to reveal the learned structure
transparently. Attention rollout [1, 9] is a popular way to understand whether
attention modules can provide such explanations [19, 34]. However, deep
transformer architectures model complex functions such that reliable
interpretation is often limited to inspecting single self-attention layers [8,
3]. Based on this observation we employ a shallow attention structure into our
network that allows to directly explain the attention maps with the
correspondence to object parts.

Unsupervised part discovery Some previous works discover parts by using image
reconstruction [36, 11], where a landmark bottleneck is used to discover
object parts. However, the model having no learning signal indicating which
parts of the image may represent an object of interest limits the
applicability of these approaches to cases where the objects of interest are
either dominating the image and depicted in similar poses [36] or are endowed
with foreground segmentation masks that can be used as an additional training
signal [11]. On datasets where most parts are common to all images, pre-
trained Vision Transformer [3] is typically able to find the parts of the most
relevant object in a semantically consistent manner. However, it breaks down
when the assumption that salient parts occur in almost all images does not
hold, since parts tend to become polysemous in such a setting. Unlike these
approaches, PDiscoNet is able to leverage the class labels, requiring no
additional annotations in fine-grained classification datasets, to learn parts
that are specific to similar classes, making them more semantically consistent
and suitable for interpretation.

Figure 2: Diagram of the proposed method. The part discovery process is driven
by the fine-grained classification loss and the losses that applied on the
part attention maps (red boxes).

Weakly-supervised part discovery via FGVC MA-CNN [38] and ProtoPNet [10]
propose to directly enforce that the CNN activation maps develop a part-like
behaviour, showing that an architecture with enhanced interpretability does
not result in a loss of performance. However, their focus is more on
downstream fine-grained classification than on evaluating the discovered
parts. SCOPS [18], a model for part co-segmentation, puts more emphasis on the
quality of the discovered parts by adding several losses on the part maps that
encourage them to be compact and distinct, the latter via decorrelation of the
learned part prototypes. It also encourages part maps to be equivariant under
geometric transforms of the image. Taken together, these incentives ensure
that the discovered parts are semantically consistent across images. This
method assumes that all the parts should be active in every image of the
dataset. Huang and Li[17] aim to solve this issue by encouraging the presence
of each part across a batch of images to follow a beta distribution with
manually defined parameters. Depending on the chosen parameters, this
encourages a pre-defined proportion of images in a batch to display the part,
while it is discouraged in the rest of the images in the batch.

##  3 PDiscoNet Method

We design an approach to discover $K$ discriminative parts that are relevant
to a fine-grained classification task, based solely on the image-level class
labels. Let $\mathbf{X}\in\mathbb{R}^{3\times A\times B}$ denote an image in
the dataset, and let $y\in\\{1,2,...,C\\}$ be its corresponding label. Using a
CNN base model $f_{\theta}$ we obtain a feature tensor
$\mathbf{Z}=f_{\theta}(\mathbf{X})$ with $\mathbf{Z}\in\mathbb{R}^{D\times
H\times W}$. Following [18] and [17], from this tensor we compute $K+1$ ($K$
parts plus one background element) attention maps
$\mathbf{A}^{k}=[0,1]^{H\times W},\ k\in\\{1,\dots,K+1\\}$ by applying a
negative squared Euclidean distance function between feature vectors
$\mathbf{z}_{ij}$ (with $\mathbf{z}_{ij}\in\mathbb{R}^{D}$,
$i\in\\{1,...,H\\}$, $j\in\\{1,...,W\\}$) and $K$ part prototypes
$\mathbf{p}^{k}\in\mathbb{R}^{D}$ in a $1\times 1$ convolutional manner,
followed by a softmax across the $K+1$ channels:

$a_{ij}^{k}=\frac{\exp(-\|\mathbf{z}_{ij}-\mathbf{p}^{k}\|^{2})}{\sum_{k}\exp(-\|\mathbf{z}_{ij}-\mathbf{p}^{k}\|^{2})},$
(1)

Each attention map is then used to compute its corresponding part vector
$\textbf{v}^{k}\in\mathbb{R}^{D}$ by using the attention values to calculate a
weighted average over the feature vectors in $\mathbf{Z}$:

$\mathbf{v}^{k}=\frac{\sum_{i}\sum_{j}\mathbf{z}_{ij}a_{ij}^{k}}{HW}$ (2)

Each of these part vectors could then be used to obtain a vector of class
scores $\mathbf{s}^{k}\in\mathbb{R}^{C}$ calculated as

$\mathbf{s}^{k}=\mathbf{W}_{class}\mathbf{v}^{k}$

by applying the same linear classifier
$\mathbf{W}_{class}\in\mathbb{R}^{C\times D}$ to all part feature vectors, but
we use the modification in Eq. (3). The scores are then averaged into a single
score vector $\mathbf{s}=\frac{1}{K}\sum_{k}\mathbf{s}^{k}$ on which a softmax
is applied to obtain the final classification probabilities
$\mathbf{\hat{y}}$.

Part vector modulation In the above formulation, all parts share the same
classifier weights $W_{\text{class}}$. This poses the problem that, from the
perspective of the classifier, all parts are equivalent, meaning that the
classifier could be encouraging all parts of the same object to result in the
similar feature representation. The classifier can also profit from part
misdetection, since a wrongly detected part would still provide a useful
feature vector. Although it would, in principle, be possible to learn part-
specific classifiers, this would not scale well to fine-grained classification
scenarios where the classification head already contains the majority of
learnable weights. As an alternative, we propose to keep a modulation vector
$\mathbf{m}_{k}\in\mathbb{R}^{D}$ per landmark that multiplies element-wise
each part vector before classification:

$\textbf{s}^{k}=W_{\text{class}}\cdot(\textbf{m}_{k}\odot\textbf{v}_{k}).$ (3)

Part dropout We would like the learned parts to be as discriminative as
possible. In order to prevent the most discriminative parts (such as the head
in birds) to discourage other parts from becoming discriminative by rendering
them unnecessary, we propose to randomly drop out a proportion of all parts
during training. This encourages the model to find a variety of discriminative
parts.

Loss functions The main learning signal for our model comes from fine-grained
classification, for which we use cross-entropy on the output classification
probabilities $\mathcal{L}_{\text{class}}(\mathbf{y},\hat{\mathbf{y}})$.
Although this signal itself would suffice for the model to perform well in the
classification task, it does not guarantee that the learned attention maps
will be interpretable as parts. There are several desirable properties we wish
to enforce in the learned parts. First, parts must be discriminative. This is
taken care of by the classification as described previously. However, we also
wish parts to be:
Compact ($\mathcal{L}_{\text{conc}}$): We would like each detected part to
consist of a compact and contiguous image region.
Distinct ($\mathcal{L}_{\text{orth}}$): We want to avoid overlap between
parts. This is encouraged by decorrelating part feature vectors.
Consistent ($\mathcal{L}_{\text{equiv}}$): The same parts should be detected
under translation, rotation or scaling of the image. This can be enforced via
a loss that encourages the equivariance of the attention maps to random rigid
transforms.
Present in the dataset ($\mathcal{L}_{\text{pres}}$): All parts should be
present in some of the images of the dataset. For this, we penalize the
absence of a part across a whole batch during training.

To enforce these priors, we use as many loss functions. Our concentration loss
over the attention maps $\mathbf{A}^{k}$:

$\displaystyle\mathcal{L}_{\text{conc}}=\frac{\sum_{k=1}^{K}\sigma_{v}^{2}(\mathbf{A}^{k})+\sigma_{h}^{2}(\mathbf{A}^{k})}{K},$
(4)

where $\sigma_{v}$ and $\sigma_{h}$ represent the vertical and horizontal
spatial variance respectively.

We calculate an orthogonality loss over the part vectors by applying the
cosine distance between all pairs:

$\mathcal{L}_{\text{orth}}=\sum_{k}\sum_{l\neq
k}\frac{\mathbf{v}^{k}\cdot\mathbf{v}^{l}}{\|\mathbf{v}^{k}\|\cdot\|\mathbf{v}^{l}\|}.$
(5)

Our equivariance loss creates a transformed image by applying a random rigid
transformation $T$ to the input image. We then pass both the original and the
transformed image through the model and invert the transformation on the
attention maps from the transformed ones. If $A^{k}(\mathbf{X})$ is a function
that returns the $k^{th}$ attention map for image $\mathbf{X}$, the
equivariance loss is computed using the cosine distance between the attention
maps from the original image and the transformed image:

$\mathcal{L}_{\text{equiv}}=1-\frac{1}{K}\sum_{k}\frac{\big{\|}A^{k}(\mathbf{X})\odot
T^{-1}(A^{k}(T(\mathbf{X})))\big{\|}}{\|A^{k}(\mathbf{X})\|\cdot\|A^{k}(T(\mathbf{X}))\|}.$
(6)

Lastly, a presence loss encourages each part to be present at least once per
batch. Given a batch $\\{\mathbf{X}_{1},\dots,\mathbf{X}_{B}\\}$:

$\mathcal{L}_{\text{pres}}=1-\frac{1}{K}\sum_{k}\max_{b,i,j}\text{avgpool}(a_{ij}^{k}(\mathbf{X}_{b})),$
(7)

where $\text{avgpool}()$ is a 2D average pooling with a small kernel size and
a stride of 1. This operator is applied to prevent encouraging single pixel
attention maps. A weighted combination of these losses is used as the final
loss.

##  4 Experiments

We compare our method, for different values of $K$, against the results
obtained by the most closely related methods in the recent literature [17]. We
also compare our method to a few other methods, among which the most recent
method on part discovery [3], which is not aimed at fine-grained
classification but showcases high quality part discovery by using self-
supervised pretraining with a visual transformer architecture.

Datasets Our aim is to perform part discovery with the only assumption being
that we have image-level class labels where parts are shared by some of the
classes, which is typically the case in FGVC tasks. In order to investigate
this, we have chosen three datasets with a varying proportions of shared parts
across images: a face image dataset where the vast majority of images display
all relevant parts (_i.e._ facial landmarks), a bird species recognition
dataset, where the assumption of the presence of all parts is limited due to
the effects of pose and occlusion, and a more challenging dataset in which
several fine-grained class categories (_e.g._ birds and cars) are mixed
together, resulting in specific parts only being shared by a small subset of
the images in the dataset. To assess the quality of the discovered parts, we
have selected datasets for which semantic part annotations are available.

| CUB | CelebA | PartImageNet
---|---|---|---
| Kp.$\downarrow$ | NMI$\uparrow$ | ARI$\uparrow$ | Class.$\uparrow$ | Kp. reg. $\downarrow$ | NMI$\uparrow$ | ARI$\uparrow$ | NMI$\uparrow$ | ARI$\uparrow$ | Class. $\uparrow$
Choudhury [11]∗ | 9.20 | 43.50 | 19.60 | - | - | - | - | - | - | -
SCOPS [18]∗∗ | 12.60 | 24.40 | 7.10 | - | 15.00 | - | - | - | - | -
DFF [12]∗∗ | - | 25.90 | 12.40 | - | 31.30 | - | - | - | - | -
Dino [3]∗∗ (K=4) | - | 31.18 | 11.21 | - | 11.36 | 1.38 | 0.01 | 19.17 | 7.59 | -
Dino [3]∗∗ (K=8) | - | 47.21 | 19.76 | - | 10.74 | 1.12 | 0.01 | 31.46 | 14.16 | -
Dino [3]∗∗ (K=16) | - | 50.57 | 26.14 | - | - | 3.29 | 0.06 | 37.81 | 16.50 | -
Huang [17] (K=4) | 11.51 | 29.74 | 14.04 | 87.30 | 8.75 | 56.69 | 34.74 | 5.88 | 1.53 | 74.22
Huang [17] (K=8) | 11.60 | 35.72 | 15.90 | 86.05 | 7.96 | 54.80 | 34.74 | 7.56 | 1.25 | 73.56
Huang [17] (K=16) | 12.60 | 43.92 | 21.10 | 85.93 | 7.62 | 62.22 | 41.01 | 10.19 | 1.05 | 73.20
PDiscoNet (K=4) | 9.12 | 37.82 | 15.26 | 86.17 | 11.11 | 75.97 | 69.53 | 27.13 | 8.76 | 88.58
PDiscoNet (K=8) | 8.52 | 50.08 | 26.96 | 86.72 | 9.82 | 62.61 | 51.89 | 32.41 | 10.69 | 89.00
PDiscoNet (K=16) | 7.67 | 56.87 | 38.05 | 87.49 | 9.46 | 77.43 | 70.48 | 41.49 | 14.17 | 86.06
Table 1: Part discovery results on CUB, CelebA and PartImageNet. * Methods use
foreground masks in training. ** Methods do not use class supervision. In the
case of PartImageNet the number of parts are $K=[8,25,50]$ instead of
$K=[4,8,16]$ used in CUB and CelebA. A ResNet101 baseline trained in the same
setting as our model results in accuracies of 85.35% on CUB and 90.81% on
PartImageNet.

CUB [32] contains 11,788 images of 200 bird species that include manual part
annotations of 15 body parts. The images are split approximately in half for
training and half for evaluating. During the development phase of this work we
used a 90%-10% split of the training set of CUB in order to find a good set of
hyperparameters for our model and used the same across all experiments on all
datasets.

CelebA [26] is a dataset of face images of 10,177 celebrities. We follow
earlier approaches [18, 17] and use the unaligned training set of 45,609
images to train our models and use the 283 MAFL test images to evaluate the
part detection, and the 5,379 images of the MAFL training set were used for
training the keypoint regressor. We use identity classification as the
downstream task.

PartImageNet [15] consist of 158 classes split among a diverse set of
categories (e.g., 10 species of fish, 14 of birds, 15 of snakes, 23 types of
car). We train all models on 14,876 images of the train set, which is limited
to 109 classes, and test on 1,664 images.

Evaluation metrics The part annotations in CUB and CelebA are in the form of
points, meant to represent part centroids in CUB and facial landmarks in
CelebA. We first evaluate the quality of the part discovery methods by
performing part location regression based on the centroids of the discovered
parts. However, as noted by [11], keypoint regression may not be a good
indicator of overall part quality. We therefore employ also the Normalized
Mutual information (NMI) and the Adjusted Rand Index (ARI), metrics commonly
used for evaluating clustering quality. In PartImageNet the part annotations
are in the form of semantic segmentation masks, from which we extract the
centroids to compute NMI and ARI. Note that NMI and ARI are computed on the
annotation/prediction correspondences across the whole datasets, meaning that
they capture part semantic consistency (i.e. a perfect score can only be
obtained if the same discovered part matches exactly with the same annotated
part). In CUB and PartImageNet we report, in addition, the classification
score on the same test set used for part quality evaluation. In the case of
CelebA, the classes on the test set do not overlap with those in the training
set.

Implementation details We trained all our models with Adam, with a starting
learning rate of $10^{-4}$ for the ResNet-101 backbone, $10^{-3}$ for the new
layers, and $10^{-2}$ for the modulation vectors. We apply 5 reductions by 0.5
every 5 epochs for CUB and PartImageNet and every 3 for CelebA. The loss
weights were all set to 1 except for $\mathcal{L}_{\text{conc}}$, where a
weight of 1000 was used because of its much lower magnitude. This setting was
decided based on the results on the CUB validation set and kept constant on
all experiments afterwards. For [17], we used $\alpha=1$ on CUB and CelebA and
$\alpha=0.002$ on PartImageNet.

###  4.1 Quantitative results

The results in Table 1 show that, on CUB with $K=4$ parts, our method already
performs comparably to [11], with 9.12% keypoint regression error vs. 9.20% in
[11], even though [11] use spatially explicit foreground masks at train time.
Our method obtains better results than all other methods in all settings and
on all metrics, improving over the second best method [3] from 50.57 to 56.87
NMI and 26.14 to 38.05 ARI for $K=16$, all while improving the classification
accuracy over [17] and a ResNet101 trained in the same setting. Interestingly,
increasing the number of parts not only results in a substantial improvement
on the part quality metrics, but also in classification accuracy, from 86.17%
with $K=4$ to 87.49% with $K=16$, unlike for [17], with which the
classification accuracy is reduced as the number of parts increases.

On CelebA, our method obtains the best clustering scores on all settings,
improving for $K=4$ over [17] from 56.69 to 75.97 NMI and from 34.74 to 69.53
ARI, thus doubling the result of the best competing method. However, [17] does
result in lower keypoint regression errors. We also obtain better keypoint
regression errors than [3], 11.11% vs. 11.36%, although this method completely
fails when evaluated in terms of the clustering metrics. As can be seen in
Section 4.4, this is related to the fact that this method is task agnostic and
focuses on elements not related to facial landmarks, such as clothing and
hair, which are not as useful for locating facial landmarks. With a single
part being assigned to the face, Dino ViT [3] obtains much lower clustering
scores than the other methods.

In the case of PartImageNet, a more challenging dataset in terms of class
diversity, Table 1 shows that both our method and Dino ViT [3] are
competitive, with our method taking the lead in terms of NMI: 41.49 with
PDiscoNet vs. 37.81 with Dino ViT, and Dino ViT in terms of ARI: 14.17 with
PDiscoNet and 16.50 with Dino ViT. The method by Huang and Li [17] fails to
capture the diversity in terms of part semantics, resulting in very low NMI
and ARI, 10.19 and 1.05, and much lower classification scores, with a maximum
of 74.22% with $K=4$, while our model reaches 89.00% with $K=25$, close to the
90.81% obtained by ResNet101 in the same training settings.

| CUB | PartImageNet
---|---|---
| Kp. $\downarrow$ | NMI $\uparrow$ | ARI $\uparrow$ | Class. $\uparrow$ | NMI $\uparrow$ | ARI $\uparrow$ | Class. $\uparrow$
Full model | 7.67 | 56.87 | 38.05 | 87.49 | 27.13 | 8.76 | 88.58
No $\mathcal{L}_{\text{orth}}$ | 10.29 | 36.12 | 19.41 | 86.17 | 16.25 | 4.59 | 89.12
No $\mathcal{L}_{\text{equiv}}$ | 10.31 | 40.22 | 21.32 | 86.60 | 17.55 | 4.22 | 89.90
No $\mathcal{L}_{\text{pres}}$ | 7.72 | 55.18 | 35.69 | 87.21 | 12.22 | 3.84 | 88.52
No $\mathcal{L}_{\text{conc}}$ | 8.58 | 52.44 | 32.17 | 86.77 | 19.71 | 7.39 | 90.32
No modulation | 8.05 | 53.45 | 35.90 | 86.36 | 19.83 | 6.02 | 89.42
No part dropout | 8.48 | 46.37 | 25.36 | 86.93 | 19.97 | 4.65 | 89.72
Table 2: Ablation studies on CUB with $K=16$ and PartImageNet with $K=8$.

###  4.2 Ablation studies

The ablation results in Table 2 confirm that all ingredients in the method are
important to obtain competitive results on part discovery. On CUB,
$\mathcal{L}_{\text{orth}}$, $\mathcal{L}_{\text{equiv}}$ and part dropout
seem to individually contribute the most to part quality in terms of the
clustering metrics, while $\mathcal{L}_{\text{conc}}$ seems to play an
important role to improve the keypoint regression results. On the other hand,
$\mathcal{L}_{\text{orth}}$ and part feature vector modulation are the
elements with the highest impact on classification performance, which on CUB
is positively impacted by all terms. $\mathcal{L}_{\text{pres}}$ seems to only
have a very marginal impact on both part discovery and classification
performance on CUB. However, on PartImageNet it is the most important of all
terms and the only one that does not hurt the classification accuracy. This is
likely to stem from the very different distribution of parts in each dataset.
On CUB, on the one hand, all parts are shared by all objects in the dataset,
since they are all birds, and a majority of them is visible in all images. On
PartImagenet, on the other hand, the different categories of classes do not
naturally share the exact same parts (e.g. snakes, vehicles and birds).
Forcing all parts to be present in each batch would prevent one single part
prototype from dominating and becoming an object detector rather than a part
detector, as happens in PartImageNet with the method of [17], as seen in Fig.
4.

###  4.3 Sensitivity studies

We have performed a part-dropout rate sensitivity analysis, shown in Table 3.
In general, increasing the dropout rate improves the part discovery
performance, with the best results obtained with the highest tested rate of
0.9, with NMI going up to 54.42 from 49.22 when a part dropout rate of 0.3 is
used. Such a high rate means that every part needs to capture enough
information to be able to classify the image, since very often all parts
except one are dropped-out. This poses a very strict constraint on the part
discovery process that prevents the appearance of spurious part prototypes.
However, this tends to negatively impact the classification performance, which
drops from 87.31% to 83.34%, probably due to the fact that, by trying to learn
parts that are able to perform classification on their own, there is a lack of
incentives for the model to learn the complementarities between parts. A value
between 0.3 and 0.7 provides a good compromise between the two tasks.

Dropout rate | 0.1 | 0.3 | 0.5 | 0.7 | 0.9
---|---|---|---|---|---
NMI $\uparrow$ | 45.30 | 49.22 | 50.08 | 49.63 | 54.42
ARI $\uparrow$ | 22.78 | 27.27 | 26.96 | 29.15 | 34.56
Class. $\uparrow$ | 86.90 | 87.31 | 86.72 | 86.26 | 83.34
Table 3: Part-dropout rate sensitivity on CUB with $K=8$.

We also investigate the behaviour of our method with respect to the presence
of noise at test time. The results in Table 4 show that the classification
accuracy of our method is higher than the most closely related method [17]
when the input images are subjected to Gaussian noise. Apart from the absolute
accuracy of our method being higher, the percentual decrease between each
increase in noise is lower, suggesting that an improved ability in part
localization also carries advantages in terms of robustness to noise.

Noise SD | 0.03 | 0.07 | 0.15 | 0.30 | 0.75 | 1.50
---|---|---|---|---|---|---
Huang [17] | 84.2 | 82.1 | 78.2 | 71.4 | 50.5 | 22.1
PDiscoNet | 85.6 | 85.6 | 82.2 | 77.9 | 63.3 | 38.2
Table 4: Class. acc. with Gaussian noise on CUB with $K=16$.

###  4.4 Qualitative results

In Figs. 3 and 4 we showcase the effect of increasing the number of parts for
the compared methods on CUB ($K=4$ and $K=16$) and PartImageNet ($K=8$ and
$K=25$). For the method in [17], we show the part assignment maps for all
parts and for those with an assigned attention value higher than 0.1, in order
to highlight only the image regoins that contribute substantially towards the
classification output. We can see that [3] and PDiscoNet are able to correctly
assign most parts to the foreground objects in the shown examples, even with
the increased number of parts (bottom three rows), with [3] resulting in the
best adherence to object boundaries. [17], on the other hand, assigns only a
few parts to the foreground objects, even when more parts are available. In
the third column ([17] (all)), we see how the rest of parts are assigned to
the background in ways that do not follow any of the objects in the image.
When showing only parts that are actually attended to by the classifier ([17]
$>0.1$), we confirm that only two or three parts are used in CUB (Fig. 3) with
both $K=4$ and $K=16$. In the case of PartImageNet (Fig. 4), [17] assigns one
single part to the foreground object with $K=8$ and none with $K=16$, while
again both [3] and PDiscoNet tend to assign a diverse set of parts to the
foreground. Also in this case we observe that Dino ViT [3] results in better
boundary adherence than PDiscoNet, since we did not explicitly add any element
towards this objective.

Image | Dino [3] |  [17] (all) |  [17] $>0.1$ | PDiscoNet
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 3: Qualitative results on CUB for our method, [17] w/ and w/o part map thresholding, and [3]. Top rows: all methods with $k=4$. Bottom rows: $K=16$. Image | Dino [3] |  [17] all |  [17] $>0.1$ | PDiscoNet
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 4: Qualitative results on PartImageNet for our method, [17] w/ and w/o part map thresholding, and [3]. Top rows: all methods with $k=8$. Bottom rows: $K=25$. Original |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
Dino [3] |  |  |  |  |  |  |  |  |  |
Huang [17] |  |  |  |  |  |  |  |  |  |
PDiscoNet |  |  |  |  |  |  |  |  |  |
Figure 5: Discovered part segmentation with $K=4$ on CUB for [3] (top), [17] (middle) and our method (bottom).  Original |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
Dino [3] |  |  |  |  |  |  |  |  |  |
Huang [17] |  |  |  |  |  |  |  |  |  |
PDiscoNet |  |  |  |  |  |  |  |  |  |
Figure 6: Discovered part segmentation with $K=4$ on CelebA for [3] (top), [17] (middle) and our method (bottom). The ground truth facial landmarks appear as black dots. Original |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
Dino [3] |  |  |  |  |  |  |  |  |  |
Huang [17] |  |  |  |  |  |  |  |  |  |
PDiscoNet |  |  |  |  |  |  |  |  |  |
Figure 7: Discovered part segmentation with $K=8$ on PartImageNet for [3]
(top), [17] (middle) and our method (bottom).  Figure 8: Histograms of part
presence per PartImageNet supercategory for Dino ViT [3] (top) and our method
(bottom), with $K=8$. Same color code as the figure above.

In Figs. 5 to 6 we show the assignment maps for the three methods, with an
attention threshold of 0.1 for [17], across ten images for each dataset in
order to explore the semantic consistency of the discovered parts across a
diverse set of examples. As can be seen in Fig. 6, [17] tends to find
symmetric part assignment maps, while our method finds independent parts for
the areas around each eye. This figure also explains why [3] fails in the
clustering metrics: the whole face tends to be assigned to a single part,
making all the facial landmarks indistinguishable from each other. This
phenomenon showcases that, although the self-supervised approach of [3]
provides remarkable results in terms of semantics and boundary adherence, it
may also miss the relevant partitions due to not making use of the fine-
grained recognition signal. This drawback of the Dino ViT approach is also
visible in the CUB results in Fig. 5, where in some examples some parts either
mix with background elements (first column) or are completely missed (last
column), while our method is consistent across all samples. The method by
Huang and Li [17] also displays a problem with mixing in background parts in
CUB and, even more markedly, in PartImageNet. Fig. 3 shows that a majority of
the available parts tend to be used on background areas that ultimately
receive a low attention weight, leading to only two or three parts being used
even in the case of $K=16$. In Fig. 4 we can see that [17] assigns one single
part to foreground objects with $K=8$ and fails to assign any parts to
foreground objects for $K=25$, which explains the low part discovery and
classification scores in Table 1. Both our method and [3] are generally able
to identify the object of interest in PartImagenet (Figs. 4 and 7), with [3]
often providing better boundary adherence, while our method tends to provide
better semantic consistency. For instance, notice how our method uses the same
part (in cyan) for the head of mammals and birds, while another one (in
purple) is used for the head of reptiles and amphibians.

This better semantic consistency is further reinforced by Fig. 8, where we
show the histograms of part presence per PartImageNet supercategory for Dino
ViT [3] and our method with $K=8$. We omit the results on [17] because only
one part tended to be active in high attention areas (see Fig. 7). This figure
confirms the notion that PDiscoNet discovers parts with strong semantic
consitency. We can see that similar supercategories (such as _Aeroplane_ and
_Boat_ , or _Biped_ and _Quadruped_) tend to share the same parts, and parts
tend to specialize on only a subset of supercategories. For instance, we note
that the cyan part (number 5) is indeed mostly present in _Biped_ ,
_Quadruped_ and _Bird_ , while the orange part (number 7), is only present in
_Fish_ , _Reptile_ and _Snake_. On the other hand, all parts are almost
equally shared by all supercategories in the case of Dino ViT, indicating that
parts are less semantically consistent across the dataset and acquire multiple
semantic interpretations.

The quantitative and qualitative results indicate that recent methods for part
discovery seem to be tailored to datasets with specific characteristics: Dino
ViT [3] thrives with a diverse set of natural images belonging to different
supercategories such as PartImageNet but fails to provide the sought after
results on the more narrow CelebA, where the parts of interest are restricted
to facial landmarks, and the opposite is true for [17]. Our proposed method,
on the other hand, is able to extract semantically consistent parts on all
tested datasets without the need for any dataset-specific adjustment, showing
its potential for out-of-the-box application to datasets with different
characteristics.

##  5 Conclusion

We propose a method for fine-grained visual categorization that uses part
representations as an information bottleneck and thus learns to detect
semantically consistent parts that are useful for that task. Our method
requires no additional annotation effort and leverages the fine-grained class
labels as the sole supervision signal. The quantitative and qualitative
comparisons against recent part discovery methods shows that our approach
improves upon the state-of-the-art in part localization and semantic
consistency, with parts specializing in certain categories and consistently
overlapping with the same semantic elements of the objects of interest,
without sacrificing accuracy on the down-stream classification task.

There are several directions in which more work is needed to improve this
approach. The first relates to the fact that, by applying a mask to a high-
level feature map in a deep model, we have no guarantee that only the
underlying regions of the image influence the corresponding part feature
representation. Information from the background or neighboring parts can leak
into the feature representation of a part due to the large receptive field of
most modern architectures, limiting the interpretability of the approach. In
addition to this, our results show that PDiscoNet displays a lower level of
contour adherence than a method trained with a very large dataset with self-
supervision. This, in turn, could affect the interpretability of the part maps
and allow background information to substantially affect the part feature
representation.

We hope that this approach will contribute towards making models for fine-
grained visual categorization more interpretable by facilitating inspection of
some aspects of the model’s internal reasoning, thus allowing a much richer
interaction between the model and its end users.

## Acknowledgements

This work was supported by the French National Research Agency under the
Investments for the Future Program, referred as ANR-16-CONV-0004 (DigitAg), by
the ERC (853489 - DEXIM), by the DFG (2064/1 – Project number 390727645), by
the Tübingen AI Center (BMBF, FKZ: 01IS18039A), and by the MUR PNRR project
FAIR - Future AI Research (PE00000013) funded by the NextGenerationEU.

## References

  * [1] Samira Abnar and Willem H. Zuidema.  Quantifying attention flow in transformers.  In ACL. Association for Computational Linguistics, 2020.
  * [2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim.  Sanity checks for saliency maps.  In NeurIPS, 2018.
  * [3] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel.  Deep ViT Features as Dense Visual Descriptors.  arXiv preprint arXiv:2112.05814, 2022.
  * [4] Anelia Angelova and Shenghuo Zhu.  Efficient object detection and segmentation for fine-grained recognition.  In CVPR, 2013.
  * [5] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation.  PloS one, 10(7):e0130140, 2015.
  * [6] Moritz Bohle, Mario Fritz, and Bernt Schiele.  Convolutional dynamic alignment networks for interpretable classifications.  In CVPR, 2021.
  * [7] Moritz Böhle, Mario Fritz, and Bernt Schiele.  B-cos networks: Alignment is all we need for interpretability.  In CVPR, 2022.
  * [8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In ICCV, 2021.
  * [9] Hila Chefer, Shir Gur, and Lior Wolf.  Transformer interpretability beyond attention visualization.  In CVPR, 2021.
  * [10] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su.  This looks like that: deep learning for interpretable image recognition.  In NeurIPS, 2019.
  * [11] Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi.  Unsupervised Part Discovery from Contrastive Reconstruction.  In NeurIPS, 2021.
  * [12] Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk.  Deep feature factorization for concept discovery.  In ECCV, 2018.
  * [13] Kun Duan, Devi Parikh, David Crandall, and Kristen Grauman.  Discovering localized attributes for fine-grained recognition.  In CVPR, 2012.
  * [14] Weifeng Ge, Xiangru Lin, and Yizhou Yu.  Weakly supervised complementary parts models for fine-grained image classification from the bottom up.  In CVPR, 2019.
  * [15] Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xiaoding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille.  PartImageNet: A Large, High-Quality Dataset of Parts.  In ECCV, 2022.
  * [16] Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang.  Part-stacked CNN for fine-grained visual categorization.  In CVPR, 2016.
  * [17] Zixuan Huang and Yin Li.  Interpretable and Accurate Fine-grained Recognition via Region Grouping.  In CVPR, 2020.
  * [18] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang, and Jan Kautz.  SCOPS: Self-Supervised Co-Part Segmentation.  In CVPR, 2019.
  * [19] Sarthak Jain and Byron C. Wallace.  Attention is not explanation.  In NAACL-HLT. Association for Computational Linguistics, 2019.
  * [20] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres.  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV).  In ICML. PMLR, 2018.
  * [21] Jonathan Krause, Timnit Gebru, Jia Deng, Li-Jia Li, and Li Fei-Fei.  Learning features and parts for fine-grained recognition.  In ICPR, 2014.
  * [22] Jonathan Krause, Hailin Jin, Jianchao Yang, and Li Fei-Fei.  Fine-grained recognition without part annotations.  In CVPR, 2015.
  * [23] Jonathan Krause, Benjamin Sapp, Andrew Howard, Howard Zhou, Alexander Toshev, Tom Duerig, James Philbin, and Li Fei-Fei.  The unreasonable effectiveness of noisy data for fine-grained recognition.  In ECCV, 2016.
  * [24] Xiao Liu, Jiang Wang, Shilei Wen, Errui Ding, and Yuanqing Lin.  Localizing by describing: Attribute-guided attention localization for fine-grained recognition.  In AAAI, 2017.
  * [25] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In CVPR, 2016.
  * [26] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In ICCV, 2015.
  * [27] Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In NeurIPS, 2017.
  * [28] Vitali Petsiuk, Abir Das, and Kate Saenko.  RISE: randomized input sampling for explanation of black-box models.  In BMVC, 2018.
  * [29] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In ICCV, 2017.
  * [30] Chawin Sitawarin, Kornrapat Pongmala, Yizheng Chen, Nicholas Carlini, and David Wagner.  Part-based models improve adversarial robustness.  In NeurIPS ML Safety Workshop, 2022.
  * [31] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In ICML, 2017.
  * [32] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie.  The caltech-ucsd birds-200-2011 dataset.  2011\.
  * [33] Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge J. Belongie.  Fine-grained image analysis with deep learning: A survey.  IEEE Trans. Pattern Anal. Mach. Intell., 44(12):8927–8948, 2022.
  * [34] Sarah Wiegreffe and Yuval Pinter.  Attention is not not explanation.  In EMNLP-IJCNLP. Association for Computational Linguistics, 2019.
  * [35] Jiahang Yin, Ancong Wu, and Wei-Shi Zheng.  Fine-grained person re-identification.  Int. J. Comput. Vis., 128(6):1654–1672, 2020.
  * [36] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee.  Unsupervised discovery of object landmarks as structural representations.  In CVPR, 2018.
  * [37] Zhishuai Zhang, Cihang Xie, Jianyu Wang, Lingxi Xie, and Alan L Yuille.  Deepvoting: A robust and explainable deep network for semantic part detection under partial occlusion.  In CVPR, 2018.
  * [38] Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo.  Learning multi-attention convolutional neural network for fine-grained image recognition.  In CVPR, 2017.
  * [39] Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In CVPR, 2016.


# Linking high-$z$ and low-$z$: Are We Observing the Progenitors of the Milky
Way with JWST?

Elka Rusta Stefania Salvadori Dipartimento di Fisica e Astronomia, Università
degli Studi di Firenze, Largo E. Fermi 1, 50125, Firenze, Italy
INAF/Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, 50125, Firenze,
Italy  Viola Gelli Cosmic Dawn Center (DAWN), Denmark  Niels Bohr Institute,
University of Copenhagen, Jagtvej 128, 2200 Copenhagen N, Denmark  Ioanna
Koutsouridou Alessandro Marconi Dipartimento di Fisica e Astronomia,
Università degli Studi di Firenze, Largo E. Fermi 1, 50125, Firenze, Italy
INAF/Osservatorio Astrofisico di Arcetri, Largo E. Fermi 5, 50125, Firenze,
Italy

###### Abstract

The recent JWST observation of the Firefly Sparkle at $z=8.3$ offers a unique
opportunity to link the high- and the low-$z$ Universe. Indeed, the claim of
it being a Milky Way (MW) type of assembly at the cosmic dawn opens the
possibility of interpreting the observation with locally calibrated galaxy-
formation models. Here, we use the MW-evolution model NEFERTITI to perform
forward modeling of our Galaxy’s progenitors at high-$z$. We build a set of
mock spectra for the MW building blocks to make predictions for JWST and to
interpret the Firefly Sparkle observation. First, we find that the most
massive MW progenitor becomes detectable in a deep survey like JADES from
$z\approx 8.2$, meaning that we could have already observed MW-analogs that
still need interpretation. Second, we provide predictions for the number of
detectable MW progenitors in lensed surveys like CANUCS, and interpret the
Firefly Sparkle as a group of MW building blocks. Both the number of
detections and the observed NIRCam photometry are consistent with our
predictions. By identifying the MW progenitors whose mock photometry best fits
the data, we find bursty and extended star-formation histories, lasting
$>150-300$ Myr, and estimate their properties: $M_{h}\approx
10^{8-9}\,M_{\odot}$, $M_{\star}\approx 10^{6.2-7.5}\,M_{\odot}$, $SFR\approx
0.04-0.20\,M_{\odot}yr^{-1}$ and $Z_{gas}\approx 0.04-0.24\,Z_{\odot}$.
Uncovering the properties of MW-analogs at cosmic dawn by combining JWST
observations and locally-constrained models, will allow us to understand our
Galaxy’s formation, linking the high- and low-$z$ perspectives.

galaxies: high-redshift — galaxies: evolution — galaxies: formation

##  1 Introduction

The galaxies that populate the present-day Universe, such as our own Milky Way
(MW), are predicted to result from the hierarchical assembling of lower-mass
progenitor galaxies, some of which formed $>13$ billion years ago (e.g. White
& Rees, 1978; Tumlinson, 2010; Salvadori et al., 2010). During the last
decade, our knowledge of the MW in its present state has tremendously advanced
thanks to local stellar surveys (e.g., Gaia, Gaia Collaboration et al. 2023,
GALAH, Buder et al. 2021). Furthermore, observations of ancient metal-poor
stars (e.g. Bonifacio et al., 2021) are providing key insights on the early
stages of the MW evolution, which can be used to indirectly study the
formation of the MW and constrain cosmological models (e.g. Hartwig et al.,
2022; Koutsouridou et al., 2023). However, we lack an understanding of how to
link our knowledge of present-day galaxies with those observed at high-z.

The exceptional capability of the James Webb Space Telescope (JWST) has opened
the possibility of looking directly at the early stages of galaxy formation.
Several surveys (e.g. GLASS, Treu et al. 2022; JADES, Eisenstein et al. 2023;
CEERS, Finkelstein et al. 2023) have been designed to observe galaxies at the
so-called cosmic dawn (redshift $z\approx 15-6$) and many bright galaxies have
been already discovered and spectroscopically confirmed at $z>10$ (e.g.
Harikane et al., 2023, 2024; Carniani et al., 2024). Moreover, gravitational
lensing enabled to catch the light of fainter and lower-mass galaxies that
would otherwise be inaccessible in the early Universe (e.g. Roberts-Borsani et
al., 2023; Vanzella et al., 2023).

Ultimately, we are accumulating large amounts of data for both present-day and
high-$z$ galaxies. Still, we need to connect these two regimes to understand
the overall galaxy-formation process.

Very recently, the JWST CAnadian NIRISS Unbiased Cluster Survey program
(CANUCS, Willott et al. 2022) has provided a unique opportunity to fill this
gap by reporting the discovery of a strongly lensed system at $z=8.3$, the
Firefly Sparkle (Mowla et al., 2024). The Firefly Sparkle comprises 10 systems
identified as “star clusters” and it has two neighboring galaxies at $\approx
2$ kpc and $\approx 13$ kpc. By comparing the stellar masses derived for these
systems with those predicted by abundance matching methods, the authors
suggest to have observed a MW-like assembly.

However physical properties of distant galaxies, such as the Firefly Sparkle
systems, are inferred from photometric data through spectral energy
distribution (SED) fitting. Inferring the star-formation histories (SFH) of
galaxies becomes particularly challenging at high-$z$ due to their bursty
nature. Indeed, both simulations (e.g. Sun et al., 2023; Pallottini & Ferrara,
2023; Gelli et al., 2023) and recent JWST observations (e.g. Endsley et al.,
2023; Langeroodi & Hjorth, 2024; Looser et al., 2023) are revealing that in
the first billion years of the Universe, galaxies undergo complex and highly
time-variable evolution, difficult to capture through SED fitting techniques.
Observing the building blocks of the MW at high-$z$ might allow us to solve
this issue. In this case, indeed, we can interpret the observed SEDs by
exploiting locally-calibrated MW-formation models to obtain physically-founded
SFHs.

In this letter, we interpret the Firefly Sparkle observation using a forward
modeling approach based on the predictions of NEFERTITI, a data-calibrated
model for the MW formation (Koutsouridou et al., 2023). We aim to answer the
following questions: Is the Firefly Sparkle consistent with a MW-like analog
at $z\approx 8$? And if so, what are the properties of the observed MW
progenitors? By comparing our predicted SEDs with those observed in the
Firefly Sparkle we will link the low- and high-$z$ Universe, also providing
key predictions to unveil other infant MW-like galaxies with the JWST.

##  2 Methods

Figure 1: Left: Synthetic SED evolution for a MW major branch simulated by the
NEFERTITI model, color-coded for redshift. For each SED, we show the
corresponding photometries in the JWST NIRCam filters. Right: Star-formation
(top panel) and metal-enrichment history (bottom panel) of the same MW major
branch. The vertical lines highlight the ages for which the SEDs on the left
are shown, with the same color coding. On the bottom-right, we report the
$M_{\star}$ at these redshifts.

The forward modeling approach consists of two steps: i) predicting the star-
formation and chemical evolutionary histories of the high-$z$ building blocks
of the MW, i.e. the MW progenitors, using a data-constrained model; and ii)
constructing the synthetic SEDs. In 2.1 we summarize the model employed and in
2.2 we explain how we infer the mock SEDs.

###  2.1 The Milky Way evolution model: NEFERTITI

We employ NEFERTITI (NEar FiEld cosmology: Re-Tracing Invisible TImes,
Koutsouridou et al. 2023), a state-of-the-art semi-analytical model for the
formation and evolution of a MW-like galaxy that can be coupled with N-body
simulations of dark matter (DM) only or with merger trees produced via Monte
Carlo algorithms. The model is calibrated to reproduce the present-day
observational properties of the MW, like the stellar/gas mass and the
metallicity distribution function of halo stars (Bonifacio et al., 2021).

NEFERTITI was developed from earlier semi-analytical models for the Local
Group formation Salvadori et al. (2007, 2015); Pagnini et al. (2023). Its main
innovations are the treatment of the incomplete sampling of the stellar
Initial Mass Function (IMF, see Rossi et al. 2021), the exploration of the
unknown IMF of the first (Pop III) stars and the energy distribution function
of the first supernovae (SNe, see Koutsouridou et al., 2024).

NEFERTITI tracks the evolution of baryonic matter inside the DM halos by
accounting for star formation and feedback processes. Stars form with a rate
proportional to the available gas mass in DM halos that reach a minimum mass
value, which changes over time according to the evolution of the ionizing and
photo-dissociating radiation (Salvadori & Ferrara, 2009). Pop III stars
initially form in the pristine gas, which is then enriched with metals from
stellar winds and SNe. SNe explosions can eject a fraction of gas/metals mass
outside of the MW progenitors, thus decreasing their star formation and
enriching the intergalactic medium (IGM). Metals are assumed to be
instantaneously mixed and when the metallicity of the interstellar medium
reaches a value $Z_{gas}>10^{-4.5}\,Z_{\odot}$ (de Bennassuti et al., 2017),
normal (Pop II) stars form according to a Larson IMF (Larson, 1998) with
$m_{\star}=[0.1,100]\,M_{\odot}$ and a peak at $m_{ch}=0.35\,M_{\odot}$.

Here we use NEFERTITI coupled with 50 different merger histories, to have a
valid statistical sample. For each merger tree, there is a most massive
progenitor (i.e. the MW major branch) and other $\rm 10^{2-4}$ smaller
systems. For example, a typical MW major branch at $z=11$ has stellar mass
$M_{\star}\approx 10^{7}\,M_{\odot}$ and gas metallicity $Z_{gas}\approx
0.1\,Z_{\odot}$, while the other progenitors can go down to $M_{\star}\approx
10^{2}\,M_{\odot}$ and $Z_{gas}\approx 10^{-4}\,Z_{\odot}$. For each of these
halos, we have access to the star-formation and metal-enrichment histories
sampled every Myr from $z=18$ to $z=6$. Here we adopt a Pop III Larson IMF
with $m_{\star}=[0.8,1000]\,M_{\odot}$ and $m_{ch}=10\,M_{\odot}$,
consistently with stellar archaeology observations (see Koutsouridou et al.,
2023, 2024). However, our findings are independent of the assumed Pop III IMF.

###  2.2 Building Up Galaxy Spectra

To make accurate predictions for the emission of the MW progenitors at
high-$z$, we build their synthetic SEDs using the spectral synthesis code
Yggdrasil (Zackrisson et al., 2011) in the following manner. For each timestep
of $1$ Myr, consistently with the NEFERTITI code, we assume an instantaneous
star-formation burst. Both the $M_{\star}$ formed and the $Z_{gas}$ from which
the stars form, are taken from the output of the galaxy-formation model. With
these properties as input, the Yggdrasil code provides the rest-frame spectrum
of a simple stellar population (SSP) at different ages (see Zackrisson et al.
2011 for more details). For Pop II stars, the available IMF is the Kroupa 2001
in the interval $m_{\star}=[0.1-100]\,M_{\odot}$ and the SSP are from
Starburst99 (Leitherer et al., 1999; Vázquez & Leitherer, 2005). For Pop III
stars we choose the lognormal IMF option in the range
$m_{\star}=[1-500]\,M_{\odot}$ with the SSP from Raiter et al. 2010. The SEDs
shown in this work will have a maximal nebular contribution, thus no escape of
Lyman continuum photons, and no dust corrections.

To model the synthetic SED of the progenitor galaxy at a chosen redshift, we
assemble the emitted rest-frame spectra of all the star-formation bursts that
make up its evolutionary history and shift into the observed frame of
reference. Fig. 1 (left) shows the SED evolution of one illustrative MW major
branch and the corresponding synthetic photometries in the JWST NIRCam
filters. On the right, we show the star-formation rate ($SFR$) and the
$Z_{gas}$ as a function of the age of the galaxy (i.e. time since the first
star-formation event). The colors mark the ages at which the SED is shown.

##  3 Results

###  3.1 Detectability of the MW Major Branch

Figure 2: Top: Predictions for the number of detectable progenitors in the
virial volume of the MW at different $z$, as a function of the magnification
factor. From the 50 merger histories provided by the NEFERTITI model, we
calculate the mean number of detections (solid lines) and the standard
deviation (shaded areas). Bottom: Comparison between the number of systems
detected in the Firefly Sparkle (grey) and the number of progenitors predicted
to be detectable in the same conditions (i.e. $z$, volume, lensing). The
dashed grey lines highlight the lensing range of the central region of the
Firefly Sparkle.

We investigate the detectability of the most massive progenitor of the MW at
different redshifts with JWST NIRCam photometry. To this end, we select the
major branch (MB) in each of the 50 merger trees simulated with NEFERTITI (see
Sec. 2.1). More details on the possible evolution of a MB are in Appendix A.
By modeling their SED evolution as shown in the example in Fig. 1, we can
analyze the possible behavior of a typical MW-like galaxy at high-$z$. We
determine the detectability by comparing the synthetic photometry of the MW
major branches at different ages (using steps of $\Delta z=0.1$) to the
detection limits of typical deep JWST surveys in the NIRCam wide filters.
Since we do not include the IGM attenuation in our spectral synthesis, we
ignore the detectability in the filters at $\lambda<\lambda_{obs}({\rm
Ly\alpha})\approx 1215\,(1+z)\,\rm\AA$.

Considering the detection limits of the JADES survey (assuming the $5\sigma$
flux depths from Rieke et al. 2023), we predict that out of the 50 MW major
branch realizations, one is detectable at $z=11$, half at $z=8.2$ and all at
$z=7.1$. From this simple analysis, we conclude that the probability for a MW-
like progenitor to become detectable in a deep JWST survey like JADES is
$P_{det}^{MB}(z<11)\geq 2\%$, $P_{det}^{MB}(z<8.2)\geq 50\%$, and
$P_{det}^{MB}(z<7.1)=100\%$

###  3.2 Detectability of a MW-type Assembly

Figure 3: Star-formation (gold) and metal-enrichment (blue) histories of the
progenitors that best fit the emission of each system of the Firefly Sparkle,
in order of increasing estimated $M_{\star}$. The synthetic SED is shown in
the top-right inset (red), together with the Mowla et al. 2024 observations
(black). The grey areas are not included in fit. The histories of the minor
branches (dotted lines) that have merged with the most massive progenitor
(solid lines) to form the final one at $z=8.3$ are also shown.

We have shown that a typical MW progenitor can become detectable and appear in
JWST observations from $z\lesssim 8$. However, to interpret an observed galaxy
as a MW-like progenitor we need to make further predictions on its expected
environment and properties. Indeed, we expect the most massive MW progenitor
at high-$z$ to be surrounded by hundreds of smaller galaxies, which will
accrete and merge over time. Due to their lower masses and weaker emission, we
likely need to rely on gravitational lensing to be able to observe such a
group of smaller MW building blocks. Here, we investigate their detectability
in a typical JWST lensing survey.

By applying the method used in Sec. 3.1 for the most massive progenitor to all
the MW progenitors predicted by NEFERTITI, we predict the number of MW
building blocks that we expect to be detectable for different observational
conditions, i.e. as a function of redshift ($z$), observed volume ($V_{obs}$),
and magnification factor ($\mu$). The threshold for detectability is set
according to the JWST NIRCam detection limits of the CANUCS program (Willott
et al., 2022).

Fig. 2 (top panel) shows our predictions at different $z$ for the number of MW
progenitors detectable in a JWST lensing survey as a function of $\mu$, within
the MW volume, which can be estimated as:

$V_{MW}(z)=\frac{4}{3}\pi\left(\frac{R_{MW}^{vir}(0)}{(1+z)}\right)^{3}$ (1)

where $R_{MW}^{vir}(0)\approx 240\,\rm kpc$. Without gravitational lensing
(i.e. $\mu=1$) we can detect only one progenitor at $z=8$, as estimated in
Sec. 3.1, and up to $\approx 5$ MW progenitors at $z=6$. In general, we see an
increasing number of detections both for bigger $\mu$ and for lower $z$,
understanding the importance of lensing when aiming to observe the MW assembly
at high-$z$.

Our predictions can be easily applied and compared to specific JWST
observations to understand if they can be interpreted as MW-like systems, by
simply rescaling the MW volume to the one observed:

$N_{det}(z)=N_{det,MW}(z)\frac{V_{obs}(z)}{V_{MW}(z)}.$ (2)

Let us apply this to the Firefly Sparkle observation. For the observed region
$V_{obs}$ we assume a sphere with $R_{obs}=13\leavevmode\nobreak\ \rm kpc$,
corresponding to the estimated distance of the second-closest galaxy to the
Firefly Sparkle.

In Fig. 2 (bottom) we show the comparison between the number of systems
detected by Mowla et al. 2024 (grey), consisting of 10 systems in the central
region and two nearby galaxies, and our predictions for the same observational
conditions (orange). We find that in the magnification range observed in the
Firefly Sparkle, $\mu=16-26$, our predicted number of detectable MW
progenitors is consistent with those observed. Although this is just a rough
estimate, the remarkable consistency of our predictions with the observed
number of sources provides a solid groundwork for their interpretation as MW-
like building blocks, i.e. galaxies. In the next section, we will compare the
systems of the Firefly Sparkle one by one to our modeled galaxies.

###  3.3 Interpreting the Firefly Sparkle

Figure 4:  $SFR$ (top panel) and $Z_{gas}$ (bottom panel) as a function of $M_{\star}$ for all the MW progenitors predicted by the NEFERTITI model at $z=8.3$, colored by density of points. The best-fitting progenitors to the Firefly Sparkle are highlighted with star markers. On top and the sides, we show the probability distribution functions (PDF) of the properties of all the progenitors (beige) and the detectable ones (red), assuming $\mu=24$ and the detection limits of CANUCS (Willott et al., 2022) Table 1: Properties of the Firefly Sparkle systems inferred via forward modeling with NEFERTITI.  Name | $log(M_{\star})$ | $log(SFR)$ | $log(Z_{gas})$ | $log(M_{h})$ | $t_{50}$ | $min\,\chi^{2}_{r}$
---|---|---|---|---|---|---
| $[M_{\odot}]$ | $[M_{\odot}\,yr^{-1}]$ | $[Z_{\odot}]$ | $[M_{\odot}]$ | $[Myr]$ |
FF-1 | $6.64\pm 0.09$ | $-1.06\pm 0.13$ | $-0.97\pm 0.15$ | $8.35\pm 0.08$ | $169\pm 45$ | 0.232
FF-2 | $6.26\pm 0.10$ | $-1.25\pm 0.16$ | $-1.38\pm 0.14$ | $8.14\pm 0.08$ | $150\pm 47$ | 0.340
FF-3 | $6.95\pm 0.06$ | $-0.70\pm 0.10$ | $-0.98\pm 0.11$ | $8.60\pm 0.04$ | $267\pm 37$ | 1.367
FF-4 | $7.16\pm 0.07$ | $-0.80\pm 0.06$ | $-0.84\pm 0.06$ | $8.74\pm 0.05$ | $309\pm 31$ | 0.900
FF-5 | $7.08\pm 0.08$ | $-1.24\pm 0.14$ | $-0.79\pm 0.14$ | $8.68\pm 0.07$ | $220\pm 31$ | 0.407
FF-6 | $7.48\pm 0.03$ | $-0.81\pm 0.04$ | $-0.62\pm 0.03$ | $8.98\pm 0.02$ | $288\pm 31$ | 0.809
FF-7 | $6.93\pm 0.09$ | $-0.77\pm 0.09$ | $-0.98\pm 0.07$ | $8.57\pm 0.07$ | $320\pm 43$ | 0.092
FF-8 | $6.96\pm 0.07$ | $-1.13\pm 0.16$ | $-0.90\pm 0.21$ | $8.64\pm 0.06$ | $241\pm 35$ | 0.002
FF-9 | $6.78\pm 0.07$ | $-1.37\pm 0.27$ | $-0.88\pm 0.27$ | $8.48\pm 0.05$ | $199\pm 37$ | 0.027
FF-10 | $6.80\pm 0.07$ | $-1.29\pm 0.24$ | $-0.92\pm 0.24$ | $8.50\pm 0.05$ | $180\pm 35$ | 0.057

Here we present the properties of the individual Firefly Sparkle systems
inferred through forward modeling with the MW galaxy-formation model
NEFERTITI. We construct the synthetic SEDs of the MW building blocks according
to their star-formation and metal-enrichment histories (see Sec. 2.2).

With this method, we build $\approx 35000$ mock JWST NIRCam photometric data
sets, representing 50 different possible populations of MW progenitors at
$z=8.3$. Among them, we select the 10 galaxies whose synthetic photometry best
fits the observational data of the 10 Firefly Sparkle systems. To this end, we
assume the $\mu$ estimated in Mowla et al. 2024 for each “star cluster” to
retrieve its unlensed photometry and compare it to our synthetic SEDs,
selecting the one with the minimum reduced $\chi^{2}$ value. We only consider
the central Firefly Sparkle, since we don’t have the photometric data of the
two nearby galaxies.

Fig. 3 shows the star formation and metal enrichment histories for the ten
best-fitting MW progenitors, from the least (top-left) to the most (bottom-
right) massive. The insets show the correspondent synthetic SEDs, compared to
the photometric JWST data. The emission of the MW building blocks predicted by
NEFERTITI at $z=8.3$ is in good agreement with the photometric data of the
Firefly Sparkle, with $\chi_{r}^{2}<1.4$. We notice that the SFHs of the MW
progenitors are not smooth and regular but rather stochastic and characterized
by multiple bursts and drops of star formation due to feedback processes (see
Sec. 2.1). These realistic complex star formation and metal enrichment
histories can fit remarkably well the observed emission of all the observed
$z=8.3$ systems simultaneously, proving that we can interpret the Firefly
Sparkle as a MW-like assembly.

Table 1 reports the physical properties inferred for the 10 Firefly Sparkle
systems, which in our interpretation are MW-progenitor galaxies. Their stellar
masses are $M_{\star}=10^{6.2-7.5}\,M_{\odot}$, their DM halos have
$M_{h}=10^{8.1-9}\,M_{\odot}$, the instantaneous
$SFR=0.04-0.20\,M_{\odot}yr^{-1}$, and $Z_{gas}=0.04-0.24\,Z_{\odot}$. We also
report $t_{50}$, defined as the time needed to form half of the total stellar
mass, which is of the order of $150-300$ Myr. As a result, these galaxies have
old stellar populations that have formed more than 100 Myr before the
observation.

In Fig. 4 we show the properties of all the $z=8.3$ MW building blocks in 50
realizations of NEFERTITI. On the top and right panels, we can visualize the
probability distribution of their properties, where we have highlighted in red
those of the galaxies that can be detected in the observational conditions of
Mowla et al. 2024 and assuming $\mu=24$ (for different $\mu$ see Appendix B).
The detectable MW progenitors are the most massive, star-forming, and metal-
rich systems, meaning that we are observing only a biased sample. The star
markers show the galaxies that best fit the individual Firefly Sparkle
systems, which are scattered across the observable regions.

Here we show all the 50 merger tree realizations to visualize the possible
properties of a MW-assembly environment at $z=8.3$. If we focus on a single
assembling history, only a couple of galaxies are more massive and star-
forming than our best fits. They could be the two nearby galaxies, but we need
further photometric observations to conclude their interpretation.

##  4 Discussion and Conclusions

The recent JWST CANUCS observation of the Firefly Sparkle claimed to be a MW-
like galaxy assembly at $z\approx 8$, presented us with the unique opportunity
to connect the still debated early stages of galaxy formation to our knowledge
of the Local Universe. In this Letter, we performed forward modeling of the MW
progenitors with the galaxy-formation model NEFERTITI to make both predictions
and interpretations for JWST.

First, analyzing multiple realizations of possible MW assembly histories, we
find that the most massive MW progenitor can appear within JADES detection
limits as early as $z\approx 11$ ($2\%$ probability), it is very likely
detectable at $z\approx 8.2$ ($50\%$), and always detectable at $z\leq 7.1$.
As a result of this prediction, in both completed and ongoing deep surveys of
JWST, we could have observed the major branch of MW-like galaxies that still
require interpretation.

Second, we confirm the interpretation of the Firefly Sparkle as a MW analog.
The number of systems detected in Mowla et al. 2024 is indeed consistent with
the predicted number of detectable MW progenitors assuming the same
observational conditions. Moreover, the NIRCam photometric data of the 10
Firefly Sparkle “star clusters” are in good agreement with the synthetic
photometries of the MW progenitors at the same redshift, with
$\chi^{2}_{r}<1.4$. By identifying the MW building blocks in NEFERTITI that
best fit the observations, we estimate the physical properties of each of
these systems, which are not predicted to be star clusters but rather MW
progenitor galaxies with $M_{h}=10^{8.1-9}\,M_{\odot}$,
$M_{\star}=10^{6.2}-10^{7.5}\,M_{\odot}$, $SFR=0.04-0.20\,M_{\odot}yr^{-1}$
and $Z_{gas}=0.04-0.24\,Z_{\odot}$.

Our $SFRs$ are consistent with those inferred by Mowla et al. 2024 with
spectrophotometric fitting techniques (Iyer & Gawiser, 2017; Iyer et al.,
2019), while their derived $M_{\star}$ are on average lower than our
predictions. This is not surprising given the bursty and long SFHs of our MW
progenitors, with $\rm t_{50}=150-300$ Myr. Indeed, reconstructing and
resolving old stellar populations through SED fitting is particularly
difficult due to the outshining of their stellar emission by the dominating
nebular emission from young stars (e.g. Whitler et al., 2023; Giménez-Arteaga
et al., 2024).

As for the gas metallicity, Mowla et al. (2024) derive a single value for the
entire system, $Z_{gas}=0.02\pm 0.01\,Z_{\odot}$. We find the $Z_{gas}$ and
its evolution for each MW progenitor, with values that are higher than their
estimation but still subsolar. The reason for this discrepancy could be due to
the direct $T_{e}$ method used to infer $Z_{gas}$ from the observed spectrum.
By fitting the observations with the HOMERUN code (Marconi et al., 2024),
which uses a weighted combination of multiple single-cloud photoionization
models, we found a total metallicity value that is consistent with those
predicted by the NEFERTITI model.

In conclusion, our results show that the Firefly Sparkle observation is
consistent with what we predict for a MW-like environment at $z=8.3$. Our
forward modeling approach has allowed us to estimate the properties of the
individual MW building blocks with physically motivated and complex
evolutionary histories. To help ongoing and future JWST surveys understand if
they are witnessing high-$z$ MW-like environments in lensed fields, we provide
predictions for the number of detectable MW progenitors at various redshifts
and for different magnification factors. Catching other MW-like systems at
cosmic dawn will indeed allow the community to collect different pictures of a
MW-analog during its infancy. Ultimately, more JWST observations combined with
our predictions might allow us to link the high- and the low-$z$ perspectives
to understand the early assembly stages of our Galaxy.

## Acknowledgements

This project received funding from the ERC Starting Grant NEFERTITI
H2020/804240 (PI: Salvadori).

## References

  * Bonifacio et al. (2021) Bonifacio, P., Monaco, L., Salvadori, S., et al. 2021, A&A, 651, A79, doi: 10.1051/0004-6361/202140816
  * Buder et al. (2021) Buder, S., Sharma, S., Kos, J., et al. 2021, MNRAS, 506, 150, doi: 10.1093/mnras/stab1242
  * Carniani et al. (2024) Carniani, S., Hainline, K., D’Eugenio, F., et al. 2024, arXiv e-prints, arXiv:2405.18485, doi: 10.48550/arXiv.2405.18485
  * de Bennassuti et al. (2017) de Bennassuti, M., Salvadori, S., Schneider, R., Valiante, R., & Omukai, K. 2017, MNRAS, 465, 926, doi: 10.1093/mnras/stw2687
  * Eisenstein et al. (2023) Eisenstein, D. J., Willott, C., Alberts, S., et al. 2023, arXiv e-prints, arXiv:2306.02465, doi: 10.48550/arXiv.2306.02465
  * Endsley et al. (2023) Endsley, R., Stark, D. P., Whitler, L., et al. 2023, arXiv e-prints, arXiv:2306.05295, doi: 10.48550/arXiv.2306.05295
  * Finkelstein et al. (2023) Finkelstein, S. L., Bagley, M. B., Ferguson, H. C., et al. 2023, ApJL, 946, L13, doi: 10.3847/2041-8213/acade4
  * Gaia Collaboration et al. (2023) Gaia Collaboration, Vallenari, A., Brown, A. G. A., et al. 2023, A&A, 674, A1, doi: 10.1051/0004-6361/202243940
  * Gelli et al. (2023) Gelli, V., Salvadori, S., Ferrara, A., Pallottini, A., & Carniani, S. 2023, ApJL, 954, L11, doi: 10.3847/2041-8213/acee80
  * Giménez-Arteaga et al. (2024) Giménez-Arteaga, C., Fujimoto, S., Valentino, F., et al. 2024, A&A, 686, A63, doi: 10.1051/0004-6361/202349135
  * Harikane et al. (2023) Harikane, Y., Ouchi, M., Oguri, M., et al. 2023, ApJS, 265, 5, doi: 10.3847/1538-4365/acaaa9
  * Harikane et al. (2024) Harikane, Y., Inoue, A. K., Ellis, R. S., et al. 2024, arXiv e-prints, arXiv:2406.18352.  https://arxiv.org/abs/2406.18352
  * Hartwig et al. (2022) Hartwig, T., Magg, M., Chen, L.-H., et al. 2022, ApJ, 936, 45, doi: 10.3847/1538-4357/ac7150
  * Iyer & Gawiser (2017) Iyer, K., & Gawiser, E. 2017, ApJ, 838, 127, doi: 10.3847/1538-4357/aa63f0
  * Iyer et al. (2019) Iyer, K. G., Gawiser, E., Faber, S. M., et al. 2019, ApJ, 879, 116, doi: 10.3847/1538-4357/ab2052
  * Koutsouridou et al. (2024) Koutsouridou, I., Salvadori, S., & Skúladóttir, Á. 2024, ApJL, 962, L26, doi: 10.3847/2041-8213/ad2466
  * Koutsouridou et al. (2023) Koutsouridou, I., Salvadori, S., Skúladóttir, Á., et al. 2023, MNRAS, 525, 190, doi: 10.1093/mnras/stad2304
  * Kroupa (2001) Kroupa, P. 2001, MNRAS, 322, 231, doi: 10.1046/j.1365-8711.2001.04022.x
  * Langeroodi & Hjorth (2024) Langeroodi, D., & Hjorth, J. 2024, arXiv e-prints, arXiv:2404.13045, doi: 10.48550/arXiv.2404.13045
  * Larson (1998) Larson, R. B. 1998, Monthly Notices of the Royal Astronomical Society, 301, 569
  * Leitherer et al. (1999) Leitherer, C., Schaerer, D., Goldader, J. D., et al. 1999, ApJS, 123, 3, doi: 10.1086/313233
  * Looser et al. (2023) Looser, T. J., D’Eugenio, F., Maiolino, R., et al. 2023, arXiv e-prints, arXiv:2306.02470, doi: 10.48550/arXiv.2306.02470
  * Marconi et al. (2024) Marconi, A., Amiri, A., Feltre, A., et al. 2024, arXiv e-prints, arXiv:2401.13028, doi: 10.48550/arXiv.2401.13028
  * Mowla et al. (2024) Mowla, L., Iyer, K., Asada, Y., et al. 2024, arXiv e-prints, arXiv:2402.08696, doi: 10.48550/arXiv.2402.08696
  * Pagnini et al. (2023) Pagnini, G., Salvadori, S., Rossi, M., et al. 2023, MNRAS, 521, 5699, doi: 10.1093/mnras/stad912
  * Pallottini & Ferrara (2023) Pallottini, A., & Ferrara, A. 2023, A&A, 677, L4, doi: 10.1051/0004-6361/202347384
  * Raiter et al. (2010) Raiter, A., Schaerer, D., & Fosbury, R. A. E. 2010, A&A, 523, A64, doi: 10.1051/0004-6361/201015236
  * Rieke et al. (2023) Rieke, M. J., Robertson, B., Tacchella, S., et al. 2023, ApJS, 269, 16, doi: 10.3847/1538-4365/acf44d
  * Roberts-Borsani et al. (2023) Roberts-Borsani, G., Treu, T., Chen, W., et al. 2023, Nature, 618, 480, doi: 10.1038/s41586-023-05994-w
  * Rossi et al. (2021) Rossi, M., Salvadori, S., & Skúladóttir, Á. 2021, Monthly Notices of the Royal Astronomical Society
  * Salvadori & Ferrara (2009) Salvadori, S., & Ferrara, A. 2009, MNRAS, 395, L6, doi: 10.1111/j.1745-3933.2009.00627.x
  * Salvadori et al. (2010) Salvadori, S., Ferrara, A., Schneider, R., Scannapieco, E., & Kawata, D. 2010, Monthly Notices of the Royal Astronomical Society: Letters, 401, L5
  * Salvadori et al. (2007) Salvadori, S., Schneider, R., & Ferrara, A. 2007, Monthly Notices of the Royal Astronomical Society, 381, 647
  * Salvadori et al. (2015) Salvadori, S., Skúladóttir, Á., & Tolstoy, E. 2015, MNRAS, 454, 1320, doi: 10.1093/mnras/stv1969
  * Sun et al. (2023) Sun, G., Faucher-Giguère, C.-A., Hayward, C. C., & Shen, X. 2023, MNRAS, 526, 2665, doi: 10.1093/mnras/stad2902
  * Treu et al. (2022) Treu, T., Roberts-Borsani, G., Bradac, M., et al. 2022, ApJ, 935, 110, doi: 10.3847/1538-4357/ac8158
  * Tumlinson (2010) Tumlinson, J. 2010, ApJ, 708, 1398, doi: 10.1088/0004-637X/708/2/1398
  * Vanzella et al. (2023) Vanzella, E., Claeyssens, A., Welch, B., et al. 2023, ApJ, 945, 53, doi: 10.3847/1538-4357/acb59a
  * Vázquez & Leitherer (2005) Vázquez, G. A., & Leitherer, C. 2005, ApJ, 621, 695, doi: 10.1086/427866
  * White & Rees (1978) White, S. D. M., & Rees, M. J. 1978, MNRAS, 183, 341, doi: 10.1093/mnras/183.3.341
  * Whitler et al. (2023) Whitler, L., Stark, D. P., Endsley, R., et al. 2023, MNRAS, 519, 5859, doi: 10.1093/mnras/stad004
  * Willott et al. (2022) Willott, C. J., Doyon, R., Albert, L., et al. 2022, Publ. Astr. Soc. Pac., 134, 025002, doi: 10.1088/1538-3873/ac5158
  * Zackrisson et al. (2011) Zackrisson, E., Rydberg, C.-E., Schaerer, D., Östlin, G., & Tuli, M. 2011, ApJ, 740, 13, doi: 10.1088/0004-637X/740/1/13

##  Appendix A Properties of the MW major branches

The 50 merger tree realizations of the NEFERTITI model provide us with 50
different possible formation histories of the MW major branches. Fig. 5 shows
the halo mass ($M_{h}$) accretion history and the star-formation history for
each merger tree’s most massive MW progenitor, from $z\approx 16$ to $z\approx
6$. The wide range of possible evolutions showcases the importance of using
the DM merger tree method (e.g. see Salvadori et al., 2007, for more details),
which allows us to have a significant statistical sample. We can notice how
the $SFR$ evolves in a stochastic and bursty manner due to the stellar
feedback (see Sec. 2.1 and Koutsouridou et al., 2023, for more details on
NEFERTITI) and to the galaxy merger process, clearing once more the importance
of using physically-motivated SFHs when interpreting JWST observations at
high-$z$.

Figure 5: Halo mass accretion history (top panel) and SFH (bottom panel) for
all the MW major branches simulated by NEFERTITI, color-coded for the halo
mass at $z=6$. The black error bars represent the mean value and the
corresponding standard deviation at $z=14,12,10,9,8,7,6$.

##  Appendix B Detectable MW progenitors with different lensings

In Sec. 3.3 we found that the MW-progenitor galaxies that can be detected in a
JWST lensing survey like CANUCS (Willott et al., 2022) are in the high-tail of
the $M_{\star}$, $SFR$, and $Z_{gas}$ distributions. To better visualize the
detectable galaxies, in Fig. 6 we show the final properties of all the MW
progenitors at $z=8.3$ for 50 merger histories, with each point representing a
progenitor galaxy. In this way, we can highlight the single galaxies that can
be detected at different $\mu$. We notice that there are $SFR-M_{\star}$ and
$Z_{gas}-M_{\star}$ relations for the detectability:

$log(SFR)\gtrsim-log(M_{\star})+a(\mu)$ (B1) $log(Z_{gas})\lesssim
2\,log(M_{\star})+b(\mu).$ (B2)

These relations quantify how we can observe lower-mass systems if they are
more star-forming or metal-poor, and how a higher magnification factor can
help us access more typical MW progenitor galaxies.

Figure 6:  $SFR$ (top panel) and $Z_{gas}$ (bottom panel) as a function of $\
M_{\star}$ for all the MW progenitors predicted by the NEFERTITI model at
$z=8.3$. Assuming the detection limits of CANUCS (Willott et al., 2022), the
grey points represent the non-detectable systems, while the colored ones are
detectable with different magnification factors (i.e. orange for $\mu=30$,
yellow for $\mu=24$, light green for $\mu=18$ and dark green for $\mu=12$).
The best-fitting progenitors to the Firefly Sparkle are highlighted with star
markers. On top and the sides, we show the probability distribution functions
(PDF) of the properties of all the progenitors and of the detectable ones,
again color-coded for $\mu$.