# Stellar: Systematic Evaluation of Human-Centric
Personalized Text-to-Image Methods

Panos Achlioptas Alexandros Benetatos11footnotemark: 1 Iordanis
Fostiropoulos11footnotemark: 1 Dimitris Skourtis11footnotemark: 1
Steel Perlot Management LLC
Los Angeles, California, USA All authors made very significant and distinct
contributions. Email Correspondence<EMAIL_ADDRESS>
###### Abstract

In this work, we systematically study the problem of personalized text-to-
image generation, where the output image is expected to portray information
about specific human subjects. E.g., generating images of oneself appearing at
imaginative places, interacting with various items, or engaging in fictional
activities. To this end, we focus on text-to-image systems that input a single
image of an individual to ground the generation process along with text
describing the desired visual context. Our first contribution is to fill the
literature gap by curating high-quality, appropriate data for this task.
Namely, we introduce a standardized dataset (Stellar) that contains
personalized prompts coupled with images of individuals that is an order of
magnitude larger than existing relevant datasets and where rich semantic
ground-truth annotations are readily available. Having established Stellar to
promote cross-systems fine-grained comparisons further, we introduce a
rigorous ensemble of specialized metrics that highlight and disentangle
fundamental properties such systems should obey. Besides being intuitive, our
new metrics correlate significantly more strongly with human judgment than
currently used metrics on this task. Last but not least, drawing inspiration
from the recent works of ELITE [73] and SDXL [49], we derive a simple yet
efficient, personalized text-to-image baseline that does not require test-time
fine-tuning for each subject and which sets quantitatively and in human trials
a new SoTA. For more information, please visit our project’s website:
https://stellar-gen-ai.github.io.

##  1 Introduction

There is no doubt that the recent progress in terms of developing and publicly
sharing multimodal deep learning models trained with internet scale data,
comprised of billions of textual tokens and images (e.g., [51, 56, 49]), have
given rise to an unprecedented explosion of novel solutions for many long-
standing problems at the intersection of computer vision and natural language
[80, 8]. Applications ranging from creating de novo fictional images [53, 58,
55] to editing them via text [11, 83], or similarly manipulating higher-
dimensional visual representations (e.g., video [74, 22, 67, 42] or 3D objects
[60, 6, 52, 82]); have vastly pushed the envelope for computer-aided
creativity and further democratized this field given the accessibility of
their linguistic interfaces.

Among the plethora of text-to-modality methods, we focus in this work on text-
to-image (T2I) models that can personalize their output by considering input
text that specifies imaginative renditions of specific human individuals. Such
personalized models have recently started to emerge [73, 19, 57, 66] as a
natural and powerful extension of de novo image generators [56, 49], inspiring
radically novel applications. These applications include personalized story-
telling via character animation of specific individuals, aiding people to
attend otherwise inaccessible real-world events virtually, or simply lifting
the burden to capture photographs physically [17, 21, 68]. Despite these
encouraging prospects, two noticeable limitations currently hinder future
developments in this field.

First, there is a need for carefully curated, standardized data, explicitly
targeting personalized T2I. Current efforts make it easy to find vast
quantities of objective captioning data for images (e.g., [62, 33]); however,
curating images of individuals performing fictional actions accompanied by
textual descriptions is practically impossible given the open-ended and
imaginary nature of this task [57]. Even if one is to lift the constraint of
coupling the descriptions of images to specific individuals [57, 81, 32],
there is yet no common ground for building large-scale collections of textual
prompts in natural language that can be deemed applicable for generating i)
interesting fictional depictions of humans, being ii) diverse, and iii),
carrying semantically meaningful meta-annotations that can be used to evaluate
the output quality of different personalized T2I approaches.

Second, more rigorous evaluation protocols are needed for comparing methods
and tangibly measuring progress on personalized T2I tasks. Despite that, many
recent and indispensable metrics for de novo text-to-image systems exist [51,
62, 23]; such metrics are typically blind to fundamental aspects that affect
the overall performance and quality of personalized, subject-driven
generators. For instance, the ability of a model to maintain the distinctive
facial characteristics of the input subject, i.e., its ‘identity’ at its
output, while essential for a successful generation, is typically left
unchecked. To make matters worse, recent works (e.g., [49, 30]) have shown
that widely used metrics, such as CLIP [51] or FID [24], correlate poorly with
human preferences, even for subject-agnostic de novo image generations.

In this work, we aim to address the above shortcomings and set robust
foundations to assist future developments in personalized T2I generation. In
summary, we introduce 1 a large-scale multimodal dataset, Stellar, with
prompts of imaginary human-centric depictions, grounding publicly available
images of celebrities [36]. 2 an extensive evaluation framework comprised of
intuitive metrics capturing complementary and fundamental aspects determining
the efficacy of modern personalized T2I models. We use these metrics to make a
thorough evaluation of existing SoTA personalized neural methods and, lastly,
we develop 3 a simple, yet efficient, T2I baseline which sets a new SoTA both
per algorithmic evaluations and in human judgment assessed via extensive human
evaluation trials.

Table 1: Comparison of Stellar with existing datasets focusing on personalized, or de novo (Parti [81]), image generations. Stellar has more prompts and identities than any existing dataset, focuses explicitly on human-centric depictions, and provides rich meta-annotations for both the paired prompts & image-based identities. Dataset |  # Prompts |  # Subjects |  Human- Centric |  Meta- Annot.
---|---|---|---|---
DreamBooth [57] | $35$ | $40$ | ✗ | ✗
CustomConcept101 [32] | $617$ | $101$ | ✗ | ✗
Parti [81] | $1.6k$ | N/A | ✗ | ✓
Stellar (Ours) | $\mathbf{20k}$ | $\mathbf{400}$ | ✓ | ✓
Table 2: Representative prompts of Stellar-$\mathcal{H}$ along with template-
based examples of Stellar-$\mathcal{T}$. The Template column showcases the
underlying production rule followed to generate the Stellar-$\mathcal{T}$
example of each row. The right-most column captures the object-centric
categories we assign to each row example of Stellar-$\mathcal{T}$. Different
colors indicate object-based categories named by their corresponding part in
the Template column, e.g.,  represents the fruit-related category.

Stellar-$\mathcal{H}$ |  Stellar-$\mathcal{T}$ |  Template |  Categ.
---|---|---|---
juggling oranges on a rooftop |  juggling apples in the airport |  juggling [fruit] [building loc.] |
riding a skateboard in outer-space |  riding a skateboard near Saturn |  riding [vehicle] [space loc.] |
wrestling an octopus on a pirate boat |  fighting a shark on a boat |  fighting [animal] on [vehicle] |
as an astronaut walking to a rocket |  as an astronaut near a spaceship |  as [uniformed] near [space-object] |
playing the trombone in a castle |  playing the saxophone in a castle |  playing [musical instrument] [building loc.] |
riding an elephant through the jungles of Thailand |  riding a tiger in the jungle |  riding [animal] [nature loc.] |
eating a croissant at a cafe in Paris |  eating a souvlaki in Athens |  eating [food] [city] |

##  2 Related Works

#### Grounded Visio-Linguistic Datasets.

There is a vast literature and accompanying datasets concerning objective
multimodal captioning of images [13, 29, 62, 48, 64, 40, 31, 33], 3D objects
and scenes [3, 43, 1, 4, 14, 70], or even more subjective bi-modal data
concerning e.g., visually grounded emotions [5, 44, 7], or dynamic actions
predicated on visual stimuli [47]. In stark contrast, datasets containing
textual prompts capable of rendering personalized subject-driven generations
aided by T2I AI, are a scarce commodity. DiffusionDB [72] is a large-scale
dataset that includes user prompts that are not personalized and typically
contain multiple complex modifiers (e.g., famous painters, photographic
techniques, etc.), resulting in obscure, not natural language. Recent
personalized datasets (e.g., [57, 81, 32]) are not focused on human-centric
actions, cover few subjects, and/or lack rich meta-annotations accompanying
the paired prompts/images. These shortcomings are addressed in Stellar (refer
to Table 1 for a short contrastive summary).

#### Metrics for T2I Generations.

Several metrics in the literature aim to evaluate different aspects of de novo
T2I systems. For example, metrics such as CLIPT [73] and CLIP-Score [23]
attempt to evaluate holistically the semantic alignment between a sentence and
a given image. More recent approaches, such as PickScore [30] and HPS [77],
aim at algorithmically evaluating the alignment of textually-grounded
generations and user-preference based on ground-truth preferential user data.
Orthogonally to these metrics, unimodal (vision-based) metrics, such as the
Aesthetic-Score [62], try to quantify the aesthetic quality of a given image.
Besides the poor correlation shown between CLIP-based metrics and human-
preference [49, 30], and the fact that subjective quantities such as an
image’s aesthetic value are intrinsically hard to compute [5]; all the above
metrics are not designed to take into account the additional image portraying
the subject that is driving the generation. I.e., they are effectively
oblivious to the personalized angle we are focusing on.

#### Personalized T2I Systems.

Capitalizing on the remarkable capabilities and modularity shown recently by
T2I models [56, 49, 54, 58], powerful extensions personalizing their output
generations have emerged. Starting with the pioneering works of Gal et. al
[19] and Ruiz et. al [57], personalized networks with admirable capabilities
have started to appear. One important limitation of these two initial works
was their lack of efficiency since they required a separate model optimization
(in the form of finetuning) for each input subject. More recently, works like
ELITE [73], or similar ones( [20, 27]), have lifted this bottleneck by
learning more general and robust mappings between visual and textual
embeddings capable of generalizing simultaneously to many individual inputs.
Similar to these later works, our introduced network (StellarNet) also avoids
a per-subject-based optimization and, per extensive quantitative and
qualitative analysis, a SoTA alternative.

##  3 Personalized Text-to-Image Data

The goal of the Stellar (Systematic Evaluation of Personalized Imagery)
dataset is to provide a large-scale standardized evaluation dataset for human-
based personalized T2I generation tasks. Stellar is comprised of 20,000
carefully curated prompts (selected from a pool of 187k candidate prompts)
describing imaginative situations and actions for rendering human-centric
fictional images. Moreover, Stellar’s prompts are paired to 400 unique human
identities, selected from CelebAMask-HQ [37, 36]; and all paired items come
with rich meta-annotations promoting a rigorous evaluation of personalized T2I
systems, as described in the following paragraphs.

#### Stellar Prompts.

For the textual part of Stellar, our goal is to offer a large set of natural
prompts that describe imaginative renderings of humans involved in many
activities, real-world object interactions, portrayed at distinct locations,
etc. In doing so, we only enforce two crucial constraints. First, we restrict
our prompts from including commands that explicitly ask to alter the physical
characteristics of an individual, such as their age, sex, eye color, or other
facial characteristics that, in large, remain constant over long periods.
Besides the many existing methods and corresponding data (e.g., [28, 35, 71,
65, 34, 37, 36] specializing in editing/describing such characteristics - the
derived invariance to such attributes eases the evaluation between the
input/output subject (by assessing if such characteristics remained
unchanged). Our second restriction concerns requests that could otherwise
intentionally control the fine-grained appearance of a human in terms of
clothes, eyewear, footwear, etc. (e.g., wearing a red dress). We believe that
T2I generators handling apparel in a try-on fashion deserve unique attention
and future work.

With the above setup established, we first curate Stellar-$\mathcal{H}$(uman)
comprised by 10,000 prompts given by English-speaking expert annotators111We
point the reader to Supp. for a detailed explanation of all AMT experiments
and safeguarding protocols promoting high-quality annotations., recruited via
Amazon Mechanical Turk (AMT). Upon reading a detailed set of instructions and
inspecting examples of good vs. bad prompts, the annotators were free to type
natural prompts they would use to create images of themselves assuming access
to an imaginative (and powerful) personalized T2I model. The resulting corpus
is remarkably diverse and imaginative, containing 6.4$k$ distinct words in
8-word-long sentences on average. It contains $\sim$1.5$k$ verbs and 4.5$k$
nouns, and the average length-normalized pairwise cosine distance of its
sentences computed in an ST5-based [45] embedding-space, is 1.58 larger ($p$
$<$$.001$) than the average distance among captions of COCO [13]. Typical
examples are shown in Table 1 and an extensive analysis in our Supplementary
Material [2].

Metric | Human Pref. Kendall-$\tau$
---|---
Obj | Rel | Overall
CLIPT | 0.130 | 0.089 | 0.106
HPSv2 | 0.067 | 0.004 | 0.224
PickScore | 0.139 | 0.110 | 0.246
DreamSim | 0.031 | 0.132 | 0.270
CLIPI | 0.049 | 0.011 | 0.304
HPSv1 | 0.094 | 0.103 | 0.304
ImageReward | 0.040 | 0.060 | 0.320
Aesth. | 0.049 | 0.146 | 0.359
GOA | 0.175 | 0.110 | 0.149
RFS | 0.121 | 0.163 | 0.167
APS | 0.013 | 0.018 | 0.389
SIS | 0.112 | 0.011 | 0.435
IPS | 0.094 | 0.018 | 0.455

Figure 1: Left: User preferences between StellarNet and existing personalized
approaches. StellarNet’s output is preferred by a very large margin (78% of
all trials). Right:  Kendall’s ($\tau$) correlation among existing and our
introduced metrics. Our metrics correlate significantly better with key
aspects of personalized generations and human-preference.

While the diversity mentioned above and the open-endedness of
Stellar-$\mathcal{H}$ make it a realistic dataset for an intrinsically
imagination-based task, they can also place certain limitations in its usage
when analyzing personalized T2I systems. Namely, given the ongoing
performance-differential of close vs. open-vocabulary discriminative methods
[75], Stellar-$\mathcal{H}$’s variance and complexity make its evaluation
harder with classical methods. To compensate for this limitation and further
standardize Stellar’s prompts, we augment Stellar with 10,000 additional
prompts generated in a semi-automatic and controlled way. Specifically, we
bootstrap Stellar-$\mathcal{H}$ by fitting to it abstract templates (e.g.,
[person] eating [food] at [place]) that can be used to derive the majority
($>$75%) of its prompts.

Critically, such templates enable us to flexibly generate new sentences that
follow Stellar-$\mathcal{H}$’s lexical structure and semantics, but where we
can control the sets of their constituent parts. For, we we can restrict nouns
associated with [food] to specific (e.g., detectable [31]) food items. In
practice, we use a mixture of frequently appearing objects and object-to-
object relations by co-analyzing Stellar-$\mathcal{H}$ with well-established
object-detection datasets (COCO [13], Objects365 [63], OpenImages [33]); as
well as using recommendations from ChatGPT [46] regarding typical examples
from broad categories extracted from the aforementioned resources. The
exhaustive generation of all underlying production rules gives rise to $177k$
candidate prompts. We sub-sample those to 10$k$ final prompts, constituting
Stellar-$\mathcal{T}$(emplate), by simultaneously keeping a high-variance
(using farthest-point-sampling [16]) and semantic-proximity to
Stellar-$\mathcal{H}$ per ST5’s embedding space.

Notably, the above process achieves the implicit meta-annotation of our
prompts. For instance, in Stellar-$\mathcal{T}$, $350$ unique detectable
objects exist, with an average of $1.5$ ‘detectable’ objects per prompt.
Moreover, its prompts cover $45$ distinct predicates and $\sim$$1k$ unique
relational triplets of the form <human-predicate-object>. Last but not least,
we manually curate discrete categories (“themes”) characterizing all prompts
(see Table 1). The semantics captured by this meta-data will be indispensable
in evaluating T2I approaches in Section 6.

#### Stellar Subject Images.

Equipped with the above prompt-based part of Stellar, we further couple these
prompts with high-resolution human-portraying images existing in CelebAMask-
HQ’s [36]. In doing so, we sampled 400 human subjects, ensuring that the
resulting image-collection accompanying Stellar satisfies the following
desiderata. First, it is unbiased to the sex and age of its subjects (per
Celeb’s binary labels - see original images of Fig. 3 which span these
categories). Second, it covers subjects for which more than one distinct image
exists. Finally, all its subjects and images are annotated with ground-truth
physical attributes and segmentation masks separating each human from the
background. Those latter masks are helpful for evaluation purposes and for
training modern T2I generators where background-free grounding images
typically result in qualitatively superior generations [73]. Finally, we note
that we combine each subject with 50 prompts chosen uniformly at random,
without repetition, to partition our 20$k$-large text collection fully.

##  4 Metrics

Table 3: Quantitative evaluation of StellarNet against popular SoTA alternatives using existing and introduced metrics. | Models |
---|---|---
Metrics |  DreamBooth [57] |  ELITE∗ [73] |  Text. Inv. [19] |  StellarNet (Ours) | Type
Aesth. ($\uparrow$) [62] | 5.316 | 5.095 | 5.263 | 5.713 | Img-to-Img
CLIPI ($\uparrow$) [51] | 0.301 | 0.387 | 0.477 | 0.531
DreamSim ($\downarrow$) [18] | 0.789 | 0.691 | 0.608 | 0.566
CLIPT ($\uparrow$) [51] | 0.413 | 0.365 | 0.311 | 0.382 | Text-to-Img
HPSv1 ($\uparrow$) [77] | 0.197 | 0.192 | 0.188 | 0.202
HPSv2 ($\uparrow$) [76] | 0.276 | 0.272 | 0.267 | 0.278
ImageReward ($\uparrow$) [79] | -0.049 | -0.510 | -0.998 | 0.320
PickScore ($\uparrow$) [30] | 0.212 | 0.205 | 0.198 | 0.212
GoA ($\uparrow$) | 0.302 | 0.260 | 0.229 | 0.305 |  Personalized (Ours)
RFS ($\uparrow$) | 0.103 | 0.106 | 0.082 | 0.134
APS ($\uparrow$) | 0.317 | 0.490 | 0.510 | 0.693
SIS ($\uparrow$) | 0.232 | 0.355 | 0.262 | 0.577
IPS ($\uparrow$) | 0.252 | 0.383 | 0.287 | 0.637
Human Preference ($\uparrow$) | 0.077 | 0.087 | 0.055 | 0.781 |

To the best of our knowledge, there are not yet adopted metrics specializing
in generative personalized imagery. Instead, existing evaluation methods for
T2I models (personalized or not) use a combination of metrics originally
designed for unconditional image generation, (e.g., FID [24] or IS [59]), and
bimodal metrics such as CLIP [51] (or extensions [23, 79, 77]) which are
optimized for de novo T2I generations that are not subject-driven.

To make matters worse, humans exhibit a heightened critical awareness of
nuanced distortions or inaccuracies in representations of human faces [12,
10]. For instance, while minor imperfections in the rendering of inanimate
objects like bananas might go unnoticed, even a slight deviation in the
portrayal of human facial features can be met with a strong, adverse reaction.
This disparity in evaluative standards underscores the need for
personalization systems focused on human-driven generations to adhere to a
higher level of precision and accuracy, as human faces are not just
scrutinized more closely but also evoke stronger emotional responses [69] upon
perceived inaccuracies [39].

In what follows, we assume a method that accepts as input the image of a human
subject ($I$), along with a prompt ($P$), and is tasked to produce an output
image ($O$).

###  4.1 Evaluating identity-centric qualities

We introduce three novel metrics aimed at quantifying the similarity between
the input subject’s ‘identity’, and the generated output image; (i) a coarse
similarity metric assessing identity preservation (ii) a fine-grained metric
focused on facial attribute preservation, and (iii) a metric orthogonal to the
first two, designed to evaluate the sensitivity and robustness of a T2I method
in maintaining the subject’s identity consistently under novel input
renderings. Prior to introducing these metrics, we address a key challenge
inherent to ‘naive’ image-to-image-based metric evaluations, setting the stage
for a comprehensive discussion of our proposed methodologies.

#### Metric Exploitation.

In evaluating image-to-image correspondences, it is crucial to recognize that
pairwise metrics directly measuring image similarity (e.g., CLIPI), will score
perfectly a method that merely replicates the input image to the output,
disregarding the prompt. To counteract this, we adapt such metrics to soft-
penalize output generations that deviate significantly from the input prompt.

Specifically, we utilize CLIPT [51], to independently measure the similarity
between the prompt and both the input and output images. An output is
penalized if it fails to improve on the input/prompt similarity by at least 2
standard deviations $\sigma$, calculated across the distribution of CLIPT
scores on all Stellar test data. Given that $\sigma$ is calculated across a
large collection, and our focus is on capturing pronounced overfitting modes
rather than nuanced visual differences, the particular choice of text-to-image
metric (e.g., CLIPT, PickScore) is of limited significance. Thus, the binary
penalty $\mathbb{I}_{p}$ is calculated as follows:

$\mathbb{I}_{p}=[\text{CLIP}_{T}(P,I)\times
2\sigma_{\text{CLIP}_{T}}]<\text{CLIP}_{T}(P,O)$ (1)

#### Identity Preservation Score.

IPS is designed to assess the facial resemblance between the input identity
and the generated images in a rather coarse but specialized way. Our metric
uses a face detector [15] to isolate the identity’s face in both input and
generated images. It then employs a specialized face detection model to
extract facial representation embeddings from the detected regions. IPS is
calculated using the metric (cosine)similarity between the embeddings from the
input image and output image whose face confidence is above a selected
threshold $\alpha$.

Two main considerations must be addressed when evaluating a method with IPS.
Firstly, the metric becomes inapplicable if the output lacks a human subject.
Secondly, multiple subjects in the output introduce ambiguity in identifying
the input identity.

To counter these issues, we assign zero scores to outputs without a detectable
face. Additionally, for outputs with multiple faces, we compute their
similarity against the input identity and use the maximum as the IPS. Lastly,
we use the penalty, $\mathbb{I}_{p}$ (Eq. (1)), to calibrate the plausible
exploitation of image-to-image metrics. Thus, if $f$ represents the facial
feature extractor, our metric is formulated as:

$\texttt{IPS}=\max f(I)\cdot\mathbb{I}_{p}[f_{>\alpha}(O)]$ (2)

#### Attribute Preservation Score.

APS focuses on assessing how well the generated images maintain specific fine-
grained attributes of the identity in question, such as age, gender, and other
invariant facial features (e.g., high cheekbones). Leveraging the annotations
in Stellar images, we can evaluate these binary facial characteristics.

Specifically, we train a series of linear classifiers on the face embeddings
extracted from CelebAHQ, utilizing the corresponding ground-truth attribute
annotations. These classifiers are employed to independently predict the
finer-grained attributes in the generated images of each subject.

To compute APS, considering the imbalanced distribution of the attribute
labels we use the ROC-AUC score for each attribute and report their average.
Additionally, we incorporate the penalization from Eq. (1). Let $M$ denote the
total number of attributes, with $p_{m}$ representing the classifiers for the
$m$-th attribute. Further, let $\mathbf{f_{O}}$ be the outputs of the face-
detector and $\mathbf{Y_{m}}$ the annotations of the $m$-th attribute. With
this notation, the APS is formulated as:

$\texttt{APS}=\frac{1}{M}\sum_{m=1}^{M}{\text{AUC}\left[\mathbb{I}_{n}\left[p_{m}(\mathbf{f_{O}}),\mathbf{Y_{m}})\right]\right]}$
(3)

#### Stability of Identity Score.

SIS explores an orthogonal dimension to the rest of our identity-based
metrics. Namely, it serves as a measure for determining the extent of a
model’s sensitivity to different images of the same individual; further
promoting models where the subject’s identity is consistently well-captured
irrespective of the input’s image irrelevant variations (e.g., lighting
conditions, subject’s pose). To achieve this goal, SIS necessitates having
access to multiple images of the human subject (a condition met in Stellar’s
dataset by design); and is our only evaluation metric with such a more
demanding requirement.

If $S$ represents the total number of the subject’s images available (all
coupled with the same prompt), $I_{m}$ is the $m$-th input image, and $O_{m}$
the corresponding $m$-th output generation, SIS is computed as follows:

$\texttt{SIS}=\frac{1}{S}\sum_{m\in S}\min_{k\in
S\setminus\\{m\\}}\,\texttt{IPS}(I_{k},O_{m})$ (4)

Notice that by forcing the similarity measurement between unaligned
input/output pairs ($k\in S\setminus\\{m\\}$), we naturally dampen the
potential of IPS to overfit to irrelevant for one’s identity details, since
such, are less likely to persist across different (input) images. Furthermore,
to reduce the increased computational requirement of generating outputs for
multiple input images per identity, a fast ‘approximation’ of the above metric
can be formulated using only a single output image $O_{m}$, as:

$\texttt{SIS}_{fast}=\min_{k\in S\setminus\\{m\\}}\,\texttt{IPS}(I_{k},O_{m})$
(5)

###  4.2 Object-centric Context Evaluation

Established bimodal metrics assessing text-image similarity (e.g., CLIPT)
often fall short in providing granular insights into their scoring rationale.
For instance, given a low similarity, it is impossible to reason whether this
stems from a failure to depict the prompt’s objects, their requested
interactions, the desired atmosphere or style, or combinations of the above.
To overcome this limitation, we introduce specialized and interpretable
metrics to evaluate two key aspects of the alignment between the output image
and the input prompt: object representation faithfulness, and the fidelity of
depicted relationships.

#### Grounding Objects Accuracy.

GOA aims to evaluate the visual faithfulness of grounding context objects
referenced in the prompt at the generated image. In contrast to CLIP’s
holistic approach (involving the comparison between the entire image and the
entire prompt), GOA is a localized metric that allows us to decouple the
evaluation of the subject’s identity preservation from that of correctly
grounding the referenced objects of the prompt.

To compute GOA, we utilize an object detector [41] to infer localized visual
objects (in the form of bounding boxes) for each object referenced in the
prompt, and average their prediction confidences, $cf$. Relying on each object
appearing at most once in the Stellar-$\mathcal{T}$ prompts, if multiple
bounding boxes are found for a referenced object, we utilize the one with the
highest confidence ignoring the others. If $Y_{m}$ represents the $m$-th
referenced object in a single prompt out of $M$ in total, and $d$ is the
object detector used that generates detection confidences, $cf$, then GOA is
formulated as:

$\texttt{GOA}=\frac{1}{M}\sum_{m=1}^{M}\max_{cf}d(Y_{m},O)$ (6)

#### Relation Fidelity Score.

RFS aims to evaluate the success of representing the desired prompt-referenced
object interactions on the generated image. Considering the difficulty of even
specialized Scene Graph Generation (SGG) models to understand visual
relations, this metric introduces a valuable insight into the ability of the
personalized model to faithfully depict the prompted relations.

Our approach leverages an SGG model [84] to extract relationship triplets in
the form of <subject \- predicate \- object> from the generated images. First,
given the set of ground truth relation labels from Stellar-$\mathcal{T}$, we
map those to the vocabulary of the SGG model, following [9, 85], to get the
metric labels, $Y$. If $G$ is the generated graph from the SGG model, we only
keep triplets with a human subject, and additionally, pick triplets from $G$
if their object matches the objects in $Y$. This way, for each filtered
triplet, $k$, we have an associated annotated relation, $R_{k}$, from
Stellar-$\mathcal{T}$, and a probability distribution vector,
$\mathbf{D_{k}(}O\mathbf{)}$ from $G$. Letting the probability $P_{k}$ of the
$R_{k}$ relation from $\mathbf{D_{k}(}O\mathbf{)}$, be denoted as
$P_{k}(O,R_{k})$, and averaging over all $N$ relations, we define RFS as:

$\texttt{RFS}=\frac{1}{N}\sum_{k=1}^{N}{P_{k}(O,R_{k})}$ (7)

##  5 StellarNet Baseline

Figure 2: StellarNet overview. The Dynamic Textual Inversion (DTI) module
inverts the foreground-masked identity image (found in CelebAMask-HQ [36])
into textual embeddings, $S^{*}$ (left). The $S^{*}$ augments the textual
prompt passed to the pre-trained text-to-image model (SDXL [49]) to guide the
model into generating images with the given identity (middle). Additionally,
we finetune the UNet backbone of SDXL using LoRA weight-offsets [26] for
efficient and stable training (middle bottom). We apply a masked MSE loss
during training over the input image and the output generation (right).

To analyze and evaluate human-centric personalized T2I methodologies more
comprehensively, we introduce a new baseline model, drawing substantial
inspiration from prior approaches. Our primary objective is not to devise a
new personalized T2I paradigm but to meaningfully enrich our comparative
assessment pool. Secondarily, we wish to highlight the potential of leveraging
more powerful open-source, non-personalized T2I systems (e.g., SDXL [49]) in
combination with well-curated data (e.g., CelebAMask-HQ [36]) to assist
personalized generators.

Similar to existing research [19, 57], our architecture relies on large pre-
trained T2I diffusion models [56, 49] tapping on their excellent capacity for
prompt-guided image generation. Following recent works [73, 20], originally
inspired by GAN-Inversion techniques [78], we also aim to derive a latent
representation for the provided subject image, $I$, in the pre-trained word-
embedding space of a T2I model. Assuming learning a successful projection of
$I$ in this space, a personalized generator can then treat the subject as a
word and leverage the compositionality of natural language to synthesize novel
images involving the subject.

#### Dynamic Textual Inversion.

To achieve the above goal efficiently and overcome the need for per-subject
optimization, we incorporate an encoder-based inversion technique (similar to
[73, 20]). To this end, we use image encoders to acquire image embeddings for
$I$ and map those to textual embeddings (denoted as $\mathbf{S^{*}}$) through
shallow Multilayer Perceptrons (MLP). During training, we optimize
$\mathbf{S^{*}}$, comprised of a series of embeddings projected from different
layers of the image encoder. Still, we use only the last layer’s embedding
during inference to de-emphasize lower-abstraction details learned from the
input image, enhancing the model’s focus on the subject’s salient
characteristics. Additionally, to better guide the network to focus on the
input subject, we use binary masks to separate it from the background of the
images passed to the image encoders. We refer to this process and its
underlying neural module as Dynamic Textual Inversion (DTI) (Fig. 2 \- left).

#### SDXL Incorporation.

At the core of our architecture lies the SDXL [49] model (base model,
excluding the refiner), extended with a DTI module that augments the textual
input with embeddings representing the personalization input. The DTI module
utilizes two CLIP image encoders (CLIP ViT-L & OpenCLIP ViT-bigG), aligned
with the original SDXL’s two CLIP text encoders. Additionally, we finetune the
pre-trained UNet backbone of SDXL to more effectively interpret the new
$\mathbf{S^{*}}$ embeddings. Efficient and stable training for the larger SDXL
is achieved using Low-Rank Adaptation (LoRA) [26] weight offsets (Fig. 2 \-
middle bottom). Finally, to avoid unnecessary model penalization due to
irrelevant input details, during training, we also apply a masked MSE loss
over the input and output generation (Fig. 2 \- right). More implementation
details can be found in Sec. 3 \- Supp. [2].

Figure 3: StellarNet generations grounded on different noise-controlling
random seeds. StellarNet produces rich variations given a human subject (top
row [36]) and a fixed prompt (bottom of each column). Best viewed by zooming
in on the digital version.

##  6 Experimental Results

In this section, we detail the experiments conducted to 1 validate the
alignment of our Metrics (Sec. 4) with human judgment, 2 contrast them with
existing metrics, and 3 show the effectiveness of the StellarNet Baseline
(Sec. 5) compared to popular personalized generators. Our experimental
analysis is partly possible due to the introduction of the Stellar Dataset
that enables the usage of our fine-grained metrics over a large-scale and
standardized benchmark.

In all the experiments we evaluate along StellarNet, DreamBooth (DB) [57],
Textual Inversion (TI) [19], and ELITE [73]. Specifically, we train StellarNet
and ELITE on the official training set of CelebAMask-HQ consisting of 24k
images. We denote ELITE trained on this image dataset as ELITE∗. Note that
human identities and their underlying images in Stellar (evaluation) Dataset
do not overlap with the training set of CelebAMask-HQ. For fairness, we also
train separate DB and TI models, each with the single image input used for its
evaluation.

We use these methods to generate output images with the 20k subject-prompt
pairs in Stellar. During inference, an image matting method [50] is used to
extract human subject masks for StellarNet. We present our primary findings
and analyses in the following subsections. Additional experiments and detailed
results are in Supp.

###  6.1 Validating the Proposed Metrics

We quantitatively compare our proposed metrics with metrics typically used in
existing T2I studies (Aesthetic Score [62], CLIPI [51], DreamSim [18], CLIPT
[51], HPSv1 [77], HPSv2 [76], ImageReward [79] and PickScore [30]) over the
entire Stellar Dataset.

#### Alignment with Human Judgment.

To assess the alignment of ours and pre-existing metrics with human judgment,
we perform two studies collecting a total of 2.5k responses using AMT. In the
quantitative analysis involving these responses, we only consider answers
where there is majority-agreement among the evaluators (2 out of 3 votes being
the same), which account for $95\%$ of all responses. We present the results
of these studies in Fig. 1.

The first study, Overall, presents the human evaluators with the output image
of each method conditioned on the same subject and prompt and asks them to
choose the one they prefer the most considering both the alignment of the
output with the input human subject and the prompt. Our second study presents
the evaluators with two images generated from two systems on the same
conditioning (we pair methods at random in each trial), ignoring the
representation accuracy of the input subject, and asks them to 1 select the
one whose objects in the output image best align with the prompt, Obj, and 2
select the one that better aligns the human-objects relations with the prompt,
Rel.

We start by computing the Kendall-$\tau$ rank correlation between the
collected human responses and different metrics. We find our metrics
complementing each other with IPS outperforming existing metrics in human
preference alignment. As expected, GOA and RFS excel in the Obj and Rel
studies, respectively, indicating the efficacy of our proposed object-centric
context metrics in decoupling and evaluating specific localized aspects of
text-to-image alignment.

#### Correlation with Existing Metrics.

Expanding beyond human judgment, we study the intra and inter-correlation for
our metrics and pre-existing ones, revealing a low Pearson correlation
coefficient of $\rho=0.180$ with existing metrics, which is 3x lower than the
intra-correlation observed among established metrics ($\rho=0.471$). This
analysis suggests that our metrics add a new dimension to the evaluation of
T2I personalization models.

Furthermore, our metrics are distinct in that they encompass various aspects
of the input. For instance, object-oriented metrics like GOA and RFS exhibit a
low correlation with image-based metrics (APS, IPS, SIS), evidenced by a
Pearson correlation coefficient of $\rho=0.08$.

#### On the Complementarity of APS and IPS.

APS and IPS are expected to be highly correlated. However, Fig. 4 (bottom row)
highlights the importance of distinguishing coarse facial features (IPS) from
fine-grained facial attributes (APS). Unlike IPS, which struggles to
accurately discern StellarNet’s improved capture of the input identity due to
changes in non-facial characteristics (e.g., removing the input’s eyeglasses);
the focus of APS to solely invariant face attributes manages to effectively
showcase our method’s enhanced ability to represent this identity.

###  6.2 Surveying Personalized Generators

Figure 4: Qualitative comparison of StellarNet vs. SoTA personalized-T2I
methods. The leftmost column depicts the input image from CelebAMask-HQ [36]
portraying the actor’s identity (marked in text as $\mathbf{S^{*}}$). The four
rightmost images are generations based on the system delineated in the
column’s title. All methods input the corresponding prompt shown next to each
row. Additionally, with every generation, we include five colored circles
representing the preference of the five metrics listed at the bottom of the
figure. An opaque circle (e.g.,  ), indicates the image with the highest score
for each metric among the generations of the same row.

We compare our baseline method, Sec. 5, against established personalized T2I
generators. This comparison utilizes both pre-existing metrics from the
literature and our newly introduced metrics from Sec. 4, with results
presented in Tab. 3. Notably, StellarNet emerges as the top performer by a
wide margin in almost all metrics. The exception is CLIPT, where our
performance does not surpass other methods. This aligns with findings from
previous studies [49, 30], which also report a disconnect between CLIPT scores
and human preferences. This discrepancy, as further discussed in Sec. 2,
highlights an inherent limitation of CLIPT.

Using the results from our first study, Overall (Sec. 6.1) we find
StellarNet’s output generations to be preferred by humans when given images
generated by all ablated systems (Fig. 1 \- left). StellarNet outperforms
other methods by a wide margin – it is preferred in $\mathbf{78.1\%}$ of all
trials, compared to $8.7$% for the next best method. Additionally, Fig. 4
offers a qualitative assessment of our method as compared to other methods
showing the effectiveness of Stellar (2nd column). Finally, we show that our
method produces diverse outputs for different random seeds while being
faithful to the input identity, as shown in Fig. 3.

##  7 Conclusion & Ethical Considerations

In this paper, we identified two impediments in the progress of personalized,
human-centric T2I methods. First, there is a lack of large-scale data
appropriate for this task, and second, there is a literature gap regarding
existing evaluation metrics. To aid future developments, we curated Stellar: a
bi-modal dataset with thousands of diverse prompts and semantically rich
annotations. We also introduced an array of intuitive metrics specializing in
personalization systems, which correlate significantly better with human
judgment than previously used alternatives. The combination of annotated data
and our explainable metrics allowed us to rigorously compare recent methods,
including our simple yet powerful method and last contribution: StellarNet,
which established quantitative and qualitatively a new SoTA.

#### Ethical Considerations.

Image generation models can be misused as tools for generating false content
or prompting misinformation from bad actors. Personalized Generative AI could
be misused to create fake images of specific individuals, portraying them
offensively or inappropriately. StellarNet heavily relies on SDXL, which is
trained with millions of pictures from around the web. Therefore, our model
inherits typical SDXL’s shortcomings, such as the potential to generate biased
content, exaggerating social stereotypes, or making otherwise NSFW material.
One can use modern tools, such as NSFW classifiers [61] to attempt to control
and reduce inappropriate content generation (a practice we followed in this
work). Nevertheless, similar to other Generative AI models, StellarNet should
not be directly used in real-world applications without a careful inspection
of the model’s output. We urge future users to utilize our work responsibly
[38]. Moreover, we advise proper content moderation and relevant regulation
[25] to prevent malpractice.

## References

  * Abdelreheem et al. [2022] Ahmed Abdelreheem, Kyle Olszewski, Hsin-Ying Lee, Peter Wonka, and Panos Achlioptas.  ScanEnts3D: Exploiting phrase-to-3d-object correspondences for improved visio-linguistic models in 3d scenes.  _Computing Research Repository (CoRR)_ , abs/2212.06250, 2022.
  * Achlioptas et al. [(accessed December 2023] Panos Achlioptas, Alexandros Benetatos, Iordanis Fostiropoulos, and Dimitris Skourtis.  _Supplemental Materials for Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image_ , (accessed December 2023).  Available at https://stellar-gen-ai.github.io/materials/stellar_supplementary.pdf.
  * Achlioptas et al. [2019] Panos Achlioptas, Judy Fan, Robert XD Hawkins, Noah D Goodman, and Leonidas J. Guibas.  ShapeGlot: Learning language for shape differentiation.  In _International Conference on Computer Vision (ICCV)_ , 2019.
  * Achlioptas et al. [2020] Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas J. Guibas.  ReferIt3D: Neural listeners for fine-grained 3d object identification in real-world scenes.  In _European Conference on Computer Vision (ECCV)_ , 2020.
  * Achlioptas et al. [2021] Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas Guibas.  ArtEmis: Affective language for visual art.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Achlioptas et al. [2023a] Panos Achlioptas, Ian Huang, Minhyuk Sung, Sergey Tulyakov, and Leonidas Guibas.  ShapeTalk: A language dataset and framework for 3d shape edits and deformations.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023a.
  * Achlioptas et al. [2023b] Panos Achlioptas, Maks Ovsjanikov, Leonidas Guibas, and Sergey Tulyakov.  Affection: Learning affective explanations for real-world visual data.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023b.
  * AI [(accessed November 2023] Open AI.  Chatgpt can now see, hear, and speak, (accessed November 2023).  Available at https:https://openai.com/blog/chatgpt-can-now-see-hear-and-speak.
  * Benetatos et al. [2023] Alexandros Benetatos, Markos Diomataris, Vassilis Pitsikalis, and Petros Maragos.  Generating salient scene graphs with weak language supervision.  In _European Signal Processing Conference (EUSIPCO)_ , 2023.
  * Bentin et al. [1996] Shlomo Bentin, Truett Allison, Aina Puce, Erik Perez, and Gregory McCarthy.  Electrophysiological studies of face perception in humans.  _Journal of Cognitive Neuroscience_ , 1996.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A. Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Bruce and Young [1986] Vicki Bruce and Andy Young.  Understanding face recognition.  _British journal of psychology_ , 1986.
  * Chen et al. [2015] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and Lawrence C. Zitnick.  Microsoft COCO Captions: Data collection and evaluation server.  _Computing Research Repository (CoRR)_ , abs/1504.00325, 2015.
  * Chen et al. [2019] Z. Dave Chen, Angel X. Chang, and Matthias Nießner.  ScanRefer: 3D object localization in RGB-D scans using natural language.  _Computing Research Repository (CoRR)_ , abs/1912.08830, 2019.
  * Deng et al. [2020] Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou.  Retinaface: Single-shot multi-level face localisation in the wild.  In _CVPR_ , 2020.
  * Eldar et al. [1997] Y. Eldar, M. Lindenbaum, M. Porat, and Y.Y. Zeevi.  The farthest point strategy for progressive image sampling.  _IEEE Transactions on Image Processing (TIP)_ , 1997.
  * fastcompany.com [(accessed November 2023] fastcompany.com.  Now anyone can have their own disney studio, thanks to generative ai, (accessed November 2023).  Available at https://www.fastcompany.com/90895402/now-anyone-can-have-their-own-disney-studio-thanks-to-generative-ai.
  * Fu et al. [2023] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola.  Dreamsim: Learning new dimensions of human visual similarity using synthetic data.  _Computing Research Repository (CoRR)_ , abs/2306.09344, 2023.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _Computing Research Repository (CoRR)_ , abs/2103.00020, 2022.
  * Gal et al. [2023] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  Encoder-based domain tuning for fast personalization of text-to-image models.  _Computing Research Repository (CoRR)_ , abs/2302.12228, 2023.
  * Google [(accessed November 2023] Google.  Powering talking characters with generative ai, (accessed November 2023).  Available at https://developers.googleblog.com/2023/05/generative-ai-talking-character.html.
  * Han et al. [2022] Ligong Han, Jian Ren, Hsin-Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris Metaxas, and Sergey Tulyakov.  Show me what and tell me how: Video synthesis via multimodal conditioning.  _Computing Research Repository (CoRR)_ , abs/2203.02573, 2022.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  CLIPScore: a reference-free evaluation metric for image captioning.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2021\.
  * Heusel et al. [2018] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Computing Research Repository (CoRR)_ , abs/1706.08500, 2018.
  * House [(accessed November 2023] The White House.  Executive order on the safe, secure, and trustworthy development and use of artificial intelligence, (accessed November 2023).  Available at https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin C. K. Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  _Computing Research Repository (CoRR)_ , abs/2304.02642, 2023.
  * Karras et al. [2018] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  _Computing Research Repository (CoRR)_ , abs/1812.04948, 2018.
  * Kazemzadeh et al. [2014] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.  Referitgame: Referring to objects in photographs of natural scenes.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , 2014\.
  * Kirstain et al. [2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  _Computing Research Repository (CoRR)_ , abs/2305.01569, 2023.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, and et al.  Visual Genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision (IJCV)_ , 2017.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023\.
  * Kuznetsova et al. [2020] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari.  The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale.  _International Journal of Computer Vision (IJCV)_ , 2020.
  * Kwak et al. [2020] Jeong-gi Kwak, David K. Han, and Hanseok Ko.  CAFE-GAN: arbitrary face attribute editing with complementary attention feature.  _European Conference on Computer Vision (ECCV)_ , 2020.
  * Lample et al. [2018] Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic Denoyer, and Marc’Aurelio Ranzato.  Fader Networks: Manipulating images by sliding attributes.  _Computing Research Repository (CoRR)_ , abs/1706.00409, 2018.
  * Lee et al. [2020] Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo.  Maskgan: Towards diverse and interactive facial image manipulation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Liu et al. [2015] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In _International Conference on Computer Vision (ICCV)_ , 2015.
  * Lo Piano [2020] S. Lo Piano.  Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward.  _Nature_ , Humanities and Social Sciences Communications, 2020.
  * M. [1970] Mori M.  Bukimi no tani [the uncanny valley].  _Energy_ , 1970.
  * Mao et al. [2016] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy.  Generation and comprehension of unambiguous object descriptions.  _Computing Research Repository (CoRR)_ , abs/1511.02283, 2016.
  * Matthias Minderer [2023] Neil Houlsby Matthias Minderer, Alexey Gritsenko.  Scaling open-vocabulary object detection.  _NeurIPS_ , 2023.
  * Menapace et al. [2023] Willi Menapace, Aliaksandr Siarohin, Stéphane Lathuilière, Panos Achlioptas, Vladislav Golyanik, Elisa Ricci, and Sergey Tulyakov.  Plotting behind the scenes: Towards learnable game engines.  _Computing Research Repository (CoRR)_ , abs/2303.13472, 2023.
  * Michel et al. [2021] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka.  Text2mesh: Text-driven neural stylization for meshes.  _Computing Research Repository (CoRR)_ , abs/2112.03221, 2021.
  * Mohamed et al. [2022] Youssef Mohamed, Farooq Faizan Khan, Kilichbek Haydarov, and Mohamed Elhoseiny.  It is okay to not be okay: Overcoming emotional bias in affective image captioning by contrastive data collection.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Ni et al. [2022] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_. Association for Computational Linguistics, 2022.
  * OpenAI [2023] OpenAI.  Chatgpt.  https://chat.openai.com/, 2023.  Accessed: 2023.
  * Park et al. [2020] Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi.  Visualcomet: Reasoning about the dynamic context of a still image.  _Computing Research Repository (CoRR)_ , abs/2004.10796, 2020.
  * Plummer et al. [2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k google refexps: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _Computing Research Repository (CoRR)_ , abs/2307.01952, 2023.
  * Qin et al. [2020] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaïane, and Martin Jägersand.  U${}^{\mbox{2}}$-net: Going deeper with nested u-structure for salient object detection.  _Computing Research Repository (CoRR)_ , abs/2005.09007, 2020.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  _Computing Research Repository (CoRR)_ , abs/2103.00020, 2021.
  * Raj et al. [2023] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani.  Dreambooth3d: Subject-driven text-to-3d generation.  _Computing Research Repository (CoRR)_ , abs/2303.13508, 2023.
  * Ramesh et al. [2021a] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  _International Conference on Machine Learning (ICML)_ , 2021a.
  * Ramesh et al. [2021b] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  _Computing Research Repository (CoRR)_ , abs/2102.12092, 2021b.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _Computing Research Repository (CoRR)_ , abs/2204.06125, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _Computing Research Repository (CoRR)_ , abs/2112.10752, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  _Computing Research Repository (CoRR)_ , abs/2205.11487, 2022.
  * Salimans et al. [2016] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2016.
  * Sanghi et al. [2021] Aditya Sanghi, Hang Chu, Joseph G Lambourne, Ye Wang, Chin-Yi Cheng, and Marco Fumero.  CLIP-Forge: Towards zero-shot text-to-shape generation.  _Computing Research Repository (CoRR)_ , abs/2110.02624, 2021.
  * Schuhmann [2023] Christoph Schuhmann.  Clip based nsfw detector.  github.com/LAION-AI/CLIP-based-NSFW-Detector, 2023.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Computing Research Repository (CoRR)_ , abs/2210.08402, 2022.
  * Shao et al. [2019] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun.  Objects365: A large-scale, high-quality dataset for object detection.  In _International Conference on Computer Vision (ICCV)_ , 2019.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In _Annual Meeting of the Association for Computational Linguistics (ACL)_ , 2018.
  * Shen and Liu [2017] Wei Shen and Rujie Liu.  Learning residual images for face attribute manipulation.  _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017\.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _Computing Research Repository (CoRR)_ , abs/2304.03411, 2023.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.  Make-a-video: Text-to-video generation without text-video data.  _Computing Research Repository (CoRR)_ , abs/2209.14792, 2022.
  * Snap [(accessed November 2023] Snap.  Dream on, together and try on new personas, powered by ai, (accessed November 2023).  Available at https://newsroom.snap.com/dreams.
  * Stone A [2019] Potton A. Stone A.  Emotional responses to disfigured faces and disgust sensitivity: An eye-tracking study.  _Journal of Health Psychology (HPQ)_ , 2019.
  * Thomason et al. [2021] Jesse Thomason, Mohit Shridhar, Yonatan Bisk, Chris Paxton, and Luke Zettlemoyer.  Language grounding with 3d objects.  _Computing Research Repository (CoRR)_ , abs/2107.12514, 2021.
  * Tripathy et al. [2020] Soumya Tripathy, Juho Kannala, and Esa Rahtu.  FACEGAN: facial attribute controllable reenactment GAN.  _Computing Research Repository (CoRR)_ , abs/2011.04439, 2020.
  * Wang et al. [2022] Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau.  DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models.  _Computing Research Repository (CoRR)_ , abs/2210.14896, 2022.
  * Wei et al. [2023] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  ELITE: Encoding visual concepts into textual embeddings for customized text-to-image generation.  In _International Conference on Computer Vision (ICCV)_ , 2023.
  * Wu et al. [2021] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan.  GODIVA: generating open-domain videos from natural descriptions.  _Computing Research Repository (CoRR)_ , abs/2104.14806, 2021.
  * Wu et al. [2023a] Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, and Dacheng Tao.  Towards open vocabulary learning: A survey.  _Computing Research Repository (CoRR)_ , abs/2306.15880, 2023a.
  * Wu et al. [2023b] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li.  Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.  _Computing Research Repository (CoRR)_ , abs/2306.09341, 2023b.
  * Wu et al. [2023c] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Better aligning text-to-image models with human preference.  _Computing Research Repository (CoRR)_ , abs/2303.14420, 2023c.
  * Xia et al. [2021] Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang.  GAN inversion: A survey.  _Computing Research Repository (CoRR)_ , abs/2101.05278, 2021.
  * Xu et al. [2023] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  _Computing Research Repository (CoRR)_ , abs/2304.05977, 2023.
  * Yin et al. [2023] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen.  A survey on multimodal large language models.  _Computing Research Repository (CoRR)_ , 2306.13549, 2023.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation.  _Computing Research Repository (CoRR)_ , 2206.10789, 2022.
  * Zhang et al. [2023] Jingbo Zhang, Xiaoyu Li, Ziyu Wan, Can Wang, and Jing Liao.  Text2nerf: Text-driven 3d scene generation with neural radiance fields.  _Computing Research Repository (CoRR)_ , abs/2305.11588, 2023.
  * Zhang et al. [2022] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris Metaxas, and Jian Ren.  Sine: Single image editing with text-to-image diffusion models.  _Computing Research Repository (CoRR)_ , abs/2212.04489, 2022.
  * Zheng et al. [2023] Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song.  Prototype-based embedding network for scene graph generation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Zhong et al. [2021] Yiwu Zhong, Jing Shi, Jianwei Yang, Chenliang Xu, and Yin Li.  Learning to generate scene graph from natural language supervision.  In _International Conference on Computer Vision (ICCV)_ , 2021.


* Kilpatrick et al. (2021) Kilpatrick, C. D., Drout, M. R., Auchettl, K., et al. 2021, MNRAS, 504, 2073
  * Kilpatrick et al. (2018) Kilpatrick, C. D., Takaro, T., Foley, R. J., et al. 2018, MNRAS, 480, 2072
  * Kim et al. (2015) Kim, H.-J., Yoon, S.-C., & Koo, B.-C. 2015, ApJ, 809, 131
  * Klencki et al. (2021) Klencki, J., Istrate, A. G., Nelemans, G., & Pols, O. 2021, arXiv e-prints, arXiv:2111.10271
  * Klencki & Nelemans (2019) Klencki, J. & Nelemans, G. 2019, IAU Symposium, 346, 417
  * Klencki et al. (2020) Klencki, J., Nelemans, G., Istrate, A. G., & Pols, O. 2020, A&A, 638, A55
  * Kruckow et al. (2018) Kruckow, M. U., Tauris, T. M., Langer, N., Kramer, M., & Izzard, R. G. 2018, MNRAS, 481, 1908
  * Langer (1989) Langer, N. 1989, A&A, 210, 93
  * Langer (2012) Langer, N. 2012, ARA&A, 50, 107
  * Langer et al. (2020) Langer, N., Schürmann, C., Stoll, K., et al. 2020, A&A, 638, A39
  * Laplace et al. (2020) Laplace, E., Götberg, Y., de Mink, S. E., Justham, S., & Farmer, R. 2020, A&A, 637, A6
  * Laplace et al. (2021) Laplace, E., Justham, S., Renzo, M., et al. 2021, A&A, 656, A58
  * Law-Smith et al. (2020) Law-Smith, J. A. P., Everson, R. W., Ramirez-Ruiz, E., et al. 2020, arXiv e-prints, arXiv:2011.06630
  * Li et al. (2011) Li, W., Chornock, R., Leaman, J., et al. 2011, MNRAS, 412, 1473
  * Lipunov et al. (1994) Lipunov, V. M., Postnov, K. A., Prokhorov, M. E., & Osminkin, E. Y. 1994, ApJ, 423, L121
  * Lunnan et al. (2014) Lunnan, R., Chornock, R., Berger, E., et al. 2014, ApJ, 787, 138
  * Mandel & Müller (2020) Mandel, I. & Müller, B. 2020, MNRAS, 499, 3214
  * Mandel et al. (2021) Mandel, I., Müller, B., Riley, J., et al. 2021, MNRAS, 500, 1380
  * Marchant et al. (2021) Marchant, P., Pappas, K. M. W., Gallegos-Garcia, M., et al. 2021, A&A, 650, A107
  * McClelland & Eldridge (2016) McClelland, L. A. S. & Eldridge, J. J. 2016, MNRAS, 459, 1505
  * Meynet & Maeder (2003) Meynet, G. & Maeder, A. 2003, A&A, 404, 975
  * Meynet & Maeder (2005) Meynet, G. & Maeder, A. 2005, A&A, 429, 581
  * Miller-Jones et al. (2021) Miller-Jones, J. C. A., Bahramian, A., Orosz, J. A., et al. 2021, Science, 371, 1046
  * Modjaz et al. (2011) Modjaz, M., Kewley, L., Bloom, J. S., et al. 2011, ApJ, 731, L4
  * Moe & Di Stefano (2017) Moe, M. & Di Stefano, R. 2017, ApJS, 230, 15
  * Müller et al. (2016) Müller, B., Heger, A., Liptai, D., & Cameron, J. B. 2016, MNRAS, 460, 742
  * Neijssel et al. (2019) Neijssel, C. J., Vigna-Gómez, A., Stevenson, S., et al. 2019, MNRAS, 490, 3740
  * Neijssel et al. (2021) Neijssel, C. J., Vinciguerra, S., Vigna-Gómez, A., et al. 2021, ApJ, 908, 118
  * O’Connor & Ott (2011) O’Connor, E. & Ott, C. D. 2011, ApJ, 730, 70
  * Owocki (2015) Owocki, S. P. 2015, Instabilities in the Envelopes and Winds of Very Massive Stars, ed. J. S. Vink, Vol. 412, 113
  * Patton & Sukhbold (2020) Patton, R. A. & Sukhbold, T. 2020, MNRAS, 499, 2803
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15
  * Paxton et al. (2018) Paxton, B., Schwab, J., Bauer, E. B., et al. 2018, ApJS, 234, 34
  * Perley et al. (2016) Perley, D. A., Quimby, R. M., Yan, L., et al. 2016, ApJ, 830, 13
  * Podsiadlowski et al. (1992) Podsiadlowski, P., Joss, P. C., & Hsu, J. J. L. 1992, ApJ, 391, 246
  * Powell & Müller (2020) Powell, J. & Müller, B. 2020, MNRAS, 494, 4665
  * Prieto et al. (2008) Prieto, J. L., Stanek, K. Z., & Beacom, J. F. 2008, ApJ, 673, 999
  * Rappaport & van den Heuvel (1982) Rappaport, S. & van den Heuvel, E. P. J. 1982, in Be Stars, ed. M. Jaschek & H. G. Groth, Vol. 98, 327–344
  * Reig (2011) Reig, P. 2011, Ap&SS, 332, 1
  * Repetto et al. (2017) Repetto, S., Igoshev, A. P., & Nelemans, G. 2017, MNRAS, 467, 298
  * Rivinius et al. (2013) Rivinius, T., Carciofi, A. C., & Martayan, C. 2013, A&A Rev., 21, 69
  * Romero-Shaw et al. (2020) Romero-Shaw, I. M., Farrow, N., Stevenson, S., Thrane, E., & Zhu, X.-J. 2020, MNRAS, 496, L64
  * Salpeter (1955) Salpeter, E. E. 1955, ApJ, 121, 161
  * Sana et al. (2012) Sana, H., de Mink, S. E., de Koter, A., et al. 2012, Science, 337, 444
  * Sana & Evans (2011) Sana, H. & Evans, C. J. 2011, in IAU Symposium, Vol. 272, Active OB Stars: Structure, Evolution, Mass Loss, and Critical Limits, ed. C. Neiner, G. Wade, G. Meynet, & G. Peters, 474–485
  * Sana et al. (2014) Sana, H., Le Bouquin, J. B., Lacour, S., et al. 2014, ApJS, 215, 15
  * Sander et al. (2019) Sander, A. A. C., Hamann, W. R., Todt, H., et al. 2019, A&A, 621, A92
  * Sander & Vink (2020) Sander, A. A. C. & Vink, J. S. 2020, MNRAS, 499, 873
  * Sander et al. (2020) Sander, A. A. C., Vink, J. S., & Hamann, W. R. 2020, MNRAS, 491, 4406
  * Sanyal et al. (2015) Sanyal, D., Grassitelli, L., Langer, N., & Bestenlehner, J. M. 2015, A&A, 580, A20
  * Schneider et al. (2021) Schneider, F. R. N., Podsiadlowski, P., & Müller, B. 2021, A&A, 645, A5
  * Schulze et al. (2018) Schulze, S., Krühler, T., Leloudas, G., et al. 2018, MNRAS, 473, 1258
  * Sen et al. (2021) Sen, K., Langer, N., Marchant, P., et al. 2021, arXiv e-prints, arXiv:2111.03329
  * Shenar et al. (2020) Shenar, T., Gilkis, A., Vink, J. S., Sana, H., & Sand er, A. A. C. 2020, A&A, 634, A79
  * Shivvers et al. (2017) Shivvers, I., Modjaz, M., Zheng, W., et al. 2017, PASP, 129, 054201
  * Smartt (2009) Smartt, S. J. 2009, ARA&A, 47, 63
  * Smartt et al. (2009) Smartt, S. J., Eldridge, J. J., Crockett, R. M., & Maund, J. R. 2009, MNRAS, 395, 1409
  * Stevenson et al. (2017) Stevenson, S., Vigna-Gómez, A., Mandel, I., et al. 2017, Nature Communications, 8, 14906
  * Sukhbold & Adams (2020) Sukhbold, T. & Adams, S. 2020, MNRAS, 492, 2578
  * Sukhbold & Woosley (2014) Sukhbold, T. & Woosley, S. E. 2014, ApJ, 783, 10
  * Summa et al. (2018) Summa, A., Janka, H.-T., Melson, T., & Marek, A. 2018, ApJ, 852, 28
  * Taddia et al. (2018) Taddia, F., Stritzinger, M. D., Bersten, M., et al. 2018, A&A, 609, A136
  * Tauris et al. (2017) Tauris, T. M., Kramer, M., Freire, P. C. C., et al. 2017, ApJ, 846, 170
  * Tauris et al. (2015) Tauris, T. M., Langer, N., & Podsiadlowski, P. 2015, MNRAS, 451, 2123
  * Tramper et al. (2016) Tramper, F., Sana, H., & de Koter, A. 2016, ApJ, 833, 133
  * Tutukov & Yungelson (1993) Tutukov, A. V. & Yungelson, L. R. 1993, MNRAS, 260, 675
  * van den Bergh et al. (2005) van den Bergh, S., Li, W., & Filippenko, A. V. 2005, PASP, 117, 773
  * van den Heuvel et al. (2017) van den Heuvel, E. P. J., Portegies Zwart, S. F., & de Mink, S. E. 2017, MNRAS, 471, 4256
  * Vigna-Gómez et al. (2018) Vigna-Gómez, A., Neijssel, C. J., Stevenson, S., et al. 2018, MNRAS, 481, 4009
  * Vigna-Gómez et al. (2021) Vigna-Gómez, A., Schrøder, S. L., Ramirez-Ruiz, E., et al. 2021, ApJ, 920, L17
  * Vigna-Gómez et al. (2022) Vigna-Gómez, A., Wassink, M., Klencki, J., et al. 2022, MNRAS, 511, 2326
  * Vinciguerra et al. (2020) Vinciguerra, S., Neijssel, C. J., Vigna-Gómez, A., et al. 2020, MNRAS, 498, 4705
  * Vink (2017) Vink, J. S. 2017, A&A, 607, L8
  * Webster & Murdin (1972) Webster, B. L. & Murdin, P. 1972, Nature, 235, 37
  * Wellstein & Langer (1999) Wellstein, S. & Langer, N. 1999, A&A, 350, 148
  * Wellstein et al. (2001) Wellstein, S., Langer, N., & Braun, H. 2001, A&A, 369, 939
  * Woosley (2019) Woosley, S. E. 2019, ApJ, 878, 49
  * Woosley & Heger (2021) Woosley, S. E. & Heger, A. 2021, ApJ, 912, L31
  * Woosley et al. (1993) Woosley, S. E., Langer, N., & Weaver, T. A. 1993, ApJ, 411, 823
  * Woosley et al. (1995) Woosley, S. E., Langer, N., & Weaver, T. A. 1995, ApJ, 448, 315
  * Woosley et al. (2020) Woosley, S. E., Sukhbold, T., & Janka, H. T. 2020, ApJ, 896, 56
  * Yoon (2015) Yoon, S.-C. 2015, PASA, 32, e015
  * Yoon (2017) Yoon, S.-C. 2017, MNRAS, 470, 3970
  * Yoon et al. (2017) Yoon, S.-C., Dessart, L., & Clocchiatti, A. 2017, ApJ, 840, 10
  * Yoon et al. (2012) Yoon, S. C., Gräfener, G., Vink, J. S., Kozyreva, A., & Izzard, R. G. 2012, A&A, 544, L11
  * Yoon et al. (2010) Yoon, S.-C., Woosley, S. E., & Langer, N. 2010, ApJ, 725, 940
  * Zapartas et al. (2017) Zapartas, E., de Mink, S. E., Van Dyk, S. D., et al. 2017, ApJ, 842, 125
  * Zevin et al. (2020) Zevin, M., Spera, M., Berry, C. P. L., & Kalogera, V. 2020, ApJ, 899, L1

##  Appendix A Summary of explodability tests

Figure 15: Compactness parameter $\xi_{2.5}$of the core-collapse models in
this study as a function of their initial helium star mass. The dotted line at
$\xi_{2.5}=0.35$ separates most models that might explode (below the line) or
implode (above the line). Orange points indicate models that are predicted to
explode according to the explodability of the Ertl et al. (2016) and Müller et
al. (2016) tests (using their “standard” parameters), blue points are models
that are predicted to explode according the Müller et al. (2016) test but not
the Ertl et al. (2016) test, green points are models that is predicted to
explode according to the Ertl et al. (2016) test but not the Müller et al.
(2016) test, and gray points are models that would not successfully explode
according to both tests.

Additional information is provided about the analysis performed on our models
using the O’Connor & Ott (2011), Ertl et al. (2016), Müller et al. (2016) and
Mandel & Müller (2020) explodability tests in Figs. 15, 16 and 17.

Figure 16: Compactness parameter $\xi_{2.5}$of the core-collapse models in
this study as a function of their initial helium star mass. The dotted line at
$\xi_{2.5}=0.35$ separates most models that might explode (below the line) or
implode (above the line). Orange points indicate models that are predicted to
explode according to the explodability of the Ertl et al. (2016) and Müller et
al. (2016) tests (using their “standard” parameters), blue points are models
that are predicted to explode according the Müller et al. (2016) test but not
the Ertl et al. (2016) test, green points are models that is predicted to
explode according to the Ertl et al. (2016) test but not the Müller et al.
(2016) test, and gray points are models that would not successfully explode
according to both tests.

Figure 17: Summary of the distribution and relationships between a selection
of stellar parameters from our simulations, and parameters resulting from the
analysis of core collapse models with the Mandel & Müller (2020) model.

##  Appendix B Effect of input parameter variation in the explosion model

In this Section, we describe the effect of varying the binding energy and the
parameters that go into the Müller et al. (2016) and Mandel & Müller (2020)
explosion models.

In the main text, we calculate the outcome of the explosion models using a set
of parameters given by $\alpha_{\mathrm{out}}=0.4$,
$\alpha_{\mathrm{turb}}=1.18$, $\beta_{\mathrm{expl}}=4$, $\zeta=0.75$ and
$\tau_{1.5}=1.2$. The binding energy that is input into the model is
calculated as the sum of the (negative) gravitational energy, and the
(positive) thermal energy at core collapse. The effect of kinetic energy is
neglected (but kinetic energy is small compared with the other two
components), and rotation is not included.

In Sect. B.1, we show the effect of artificially increasing the binding energy
of the core collapse models, and on Sect. B.2, we briefly describe the effect
of varying the model parameters.

###  B.1 Effects of binding energy variation on supernova explosions

To measure the effect of the binding energy of a stellar model at core
collapse in the outcome of the Müller et al. (2016) and Mandel & Müller (2020)
explosion models, we compare the outcome of our fiducial set of parameters
(Fiducial Set) with the outcome that results from calculating the explosion
properties using the same set of parameters, but using the approximation of
$E_{\mathrm{bind}}=3GM^{2}/5R$ as a proxy for the binding energy (Set B). This
approximation effectively decreases the absolute value of the binding energy
outside of the iron core.

As shown in Fig. 18, the explosion energies are reduced to lower values in Set
B, with values that are more in line with the results of Müller et al. (2016).
These models have a lower $\mathrm{E}/\mathrm{M}$ ratio, ranging from about
0.2 and up to $4\times 10^{50}\ \mathrm{erg}/\,\mathrm{M}_{\odot}$, with most
models having a value of around 2.

Figure 18: Explosion energy as a function of final mass from our Fiducial Set
(top), and from Set B, calculated with the Müller et al. (2016) model.

The distribution of NS masses is not heavily impacted, although a few models
with initial masses in the range 4–10 $\,\mathrm{M}_{\odot}$ are predicted to
produce NSs of more than 1.5 $\,\mathrm{M}_{\odot}$, whereas they produce NSs
with masses lower than 1.4 $\,\mathrm{M}_{\odot}$ in the Fiducial Set.
However, fewer models result in successful explosions. In most cases, the
models that result in successful explosions in the Fiducial Set, but do not
produce explosions in Set B, are expected to produce explosions according to
the Ertl et al. (2016) criterion. This implies that the resulting distribution
of BH masses, and the number of BHs per SN is impacted.

When the effect of fallback is included in the calculations, as shown in Fig.
19 many models that produce NSs in the Fiducial Set produce low energy
explosions that leave behind mass gap BHs instead. In general, explosion
energies are lower in Set B than in the Fiducial Set. In the latter case,
fallback SNe from models with initial masses above about 15
$\,\mathrm{M}_{\odot}$ produce explosions with energies that range from
$6\times 10^{51}$ erg, but that can reach energies of up to a few times
$10^{52}$ erg (not shown in Fig 19). However, as shown in Sect. 4, they are
statistically rare and could be representative of rare, very luminous
transients. On the other hand, fallback SNe in this regime in Set B produce
explosions that are only a few times more energetic than typical SNe, and in
many cases in the lower mass regime, such explosions are considerably less
energetic than typical SNe.

The initial masses of the majority of progenitors of fallback SNe in the
Fiducial Set range from about 35 and up to 70 $\,\mathrm{M}_{\odot}$, whereas
in Set B, models of all masses result in fallback SNe. This has a very large
effect on the expected BH and NS mass distributions. In contrast to the
Fiducual Set (see Figs. 9 and 10) in Set B we expect most of the NSs resulting
from a population of stripped-envelope stars to be around the more massive
peak of the NS mass distribution, and most BHs produced are expected to be
mass gap objects with masses of about 3 $\,\mathrm{M}_{\odot}$. Notably, many
fallback SNe from models with initial mass lower than 15
$\,\mathrm{M}_{\odot}$ produce low energy explosions, have low nickel masses,
and no kicks.

Figure 19: Explosion energy as a function of final mass from our Fiducial Set
(top), and from Set B (bottom), calculated with the Mandel & Müller (2020)
model. Circles in the figures correspond to normal SNe, whereas triangles
correspond to fallback SNe.

Nickel masses in Set B are lower than in the Fiducial Set, regardless of
whether we include the effect of fallback or not. Kick velocities in Set B
tend to be lower in Set B, and in particular some models with low explosion
energy are also predicted to have 0 kick velocity.

Regardless of the choice of model, however, the resulting remnant mass
distributions, shown in Fig. 20, are similar to each other, and very different
from typically assumed relations, such as the often employed relationships of
Fryer et al. (2012). This demonstrates the importance of including detailed,
physically motivated explodability prescriptions in population synthesis
studies.

Figure 20: Remnant masses as a function of final mass from our Fiducial Set
(top), and from Set B, calculated with the Müller et al. (2016) model.

###  B.2 Effects of parameter variation on supernova explosions

The outcomes of the Müller et al. (2016) and Mandel & Müller (2020) explosion
models depends somewhat sensitively on the choice of input parameters.
However, many trends and correlations persist in the explosion outcome
regardless of the parameter choices.

In this Section, we describe general trends observed when varying the
explosion parameters in both models. Besides output from the Fiducial Set and
Set B, we calculated several sets of calculations varying each parameter
independently, and keeping all others fixed, with the same base values as
those employed by Müller et al. (2016).

Variations in $\alpha_{\mathrm{out}}$ result in a significant variation in the
number of models that produce successful explosions. We changed the value of
$\alpha_{\mathrm{out}}$ from 0.3 to 0.7 in steps of 0.1, and found that lower
values result in fewer explosions, removing explosions that come from models
with low $\xi_{2.5}$ (below the 0.35 threshold shown in Fig. 5), ant that have
low energy in comparison to the rest of the population. Similarly, explosions
with low nickel mass and low kicks are similarly removed when increasing the
value of $\alpha_{\mathrm{out}}$. The resulting explosion energies have lower
energies than models of similar mass in the Fiducial Set, and a few NSs are
formed that are more massive than expected from the correlation between
$\xi_{2.5}$ and NS mass.

We varied the value of $\zeta$ from 0.5 to 0.9 in steps of 0.1. We found that
as $\zeta$ is increased, the initial mass threshold between models that
produce successful explosions and those that produce BHs is increased. These
explosions have a similar distribution of $\mathrm{E}/\mathrm{M}$ than the
rest of the explosions with the same given set of parameters, and therefore
are increasingly more energetic. Similarly, such explosions have a higher
nickel mass and kick velocity.

We varied the value of $\alpha_{\mathrm{turb}}$ from 1.14 to 1.22 in steps of
0.02, and of $\tau_{1.5}$ from 1.2 to 1.5 in steps of 0.1, and found no
significant difference in the outcome of our calculations.

The set of parameters selected as our Fiducial Set was therefore calibrated
varying $\alpha_{\mathrm{out}}$ and $\zeta$ to yield a better agreement with
the explodability landscape predicted by the Ertl et al. (2016) criterion,
resulting in the landscape shown in Fig. 15.