# Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for
Text-to-Image Generation

Katherine M. Collins1 University of CambridgeUnited Kingdom<EMAIL_ADDRESS>,
Najoung Kim  Google DeepMindUnited States ,  Yonatan Bitton  Google
ResearchIsrael ,  Verena Rieser  Google DeepMindUnited Kingdom ,  Shayegan
Omidshafiei3 Field AIUnited States ,  Yushi Hu1 University of WashingtonUnited
States ,  Sherol Chen  Google DeepMindUnited States ,  Senjuti Dutta2
University of TennesseeUnited States ,  Minsuk Chang  Google ResearchUnited
States ,  Kimin Lee3 Korea Advanced Institute of Science and TechnologySouth
Korea ,  Youwei Liang2 University of California
San DiegoUnited States ,  Georgina Evans  Google DeepMindUnited States ,
Sahil Singla  Google DeepMindUnited States ,  Gang Li  Google ResearchUnited
States ,  Adrian Weller  University of Cambridge
The Alan Turing InstituteUnited Kingdom ,  Junfeng He  Google ResearchUnited
States ,  Deepak Ramachandran  Google DeepMindUnited States and  Krishnamurthy
Dj Dvijotham4 ServiceNowUnited States

(2024)

###### Abstract.

Human feedback plays a critical role in learning and refining reward models
for text-to-image generation, but the optimal form the feedback should take
for learning an accurate reward function has not been conclusively
established. This paper investigates the effectiveness of fine-grained
feedback which captures nuanced distinctions in image quality and prompt-
alignment, compared to traditional coarse-grained feedback (for example,
thumbs up/down or ranking between a set of options). While fine-grained
feedback holds promise, particularly for systems catering to diverse societal
preferences, we show that demonstrating its superiority to coarse-grained
feedback is not automatic. Through experiments on real and synthetic
preference data, we surface the complexities of building effective models due
to the interplay of model choice, feedback type, and the alignment between
human judgment and computational interpretation. We identify key challenges in
eliciting and utilizing fine-grained feedback, prompting a reassessment of its
assumed benefits and practicality. Our findings – e.g., that fine-grained
feedback can lead to worse models for a fixed budget, in some settings;
however, in controlled settings with known attributes, fine grained rewards
can indeed be more helpful – call for careful consideration of feedback
attributes and potentially beckon novel modeling approaches to appropriately
unlock the potential value of fine-grained feedback in-the-wild.

text-to-image models, human feedback, efficient elicitation, reward modeling

††journalyear: 2024††copyright: none††conference: ; ††booktitle:
;11footnotetext: Work conducted while a Student Researcher at Google
DeepMind22footnotetext: Work conducted while a Student Researcher at Google
Research33footnotetext: Work conducted while at Google Research11footnotetext:
Work conducted while at Google DeepMind Figure 1. Example text-image pair
where granular feedback matters. Figure 2. Top: a typical coarse-grained
feedback reward pipeline; bottom: proposed method for modeling fine-grained
feedback.

##  1\. Introduction

Human feedback serves as a critical element in adapting large-scale generative
models, particularly within the Reinforcement Learning with Human Feedback
(RLHF) paradigm (Casper et al., 2023; Christiano et al., 2017; Ouyang et al.,
2022). However, conventional methods often rely on coarse-grained feedback,
such as a single binary preference or Likert-scale rating, which may not
adequately capture the nuances of quality and prompt-alignment in complex
domains like text-to-image generation. A generated image may be highly
visually appealing while deviating from the prompt, or conversely, align with
the prompt and target visuals in some but not all desired ways, as illustrated
in Figure 1. Recent research in text-to-text models (i.e. Large Language
Models) suggests the potential of fine-grained feedback to address this
challenge by enabling users to express their preferences with greater
granularity, targeting specific features or interactions within the generated
output (Wu et al., 2023a; Lee et al., 2023; Liang et al., 2023). This granular
feedback mechanism promises more precise control over model adaptation and
behavior, which could lead to text-to-image models that exhibit enhanced
responsiveness to diverse user needs and preferences. For example, a user may
want more photorealistic owls for a presentation but less for a fun t-shirt
(Dutta et al., 2024).

This paper explores the complexities and trade-offs associated with utilizing
fine-grained feedback for text-to-image model adaptation. We investigate its
impact on model performance and explore strategies for integrating this
nuanced information, taking into account the difficulty of elicitation. Our
experiments reveal a complex interplay between model architecture, feedback
type, and the alignment between human judgment and simulated AI preference
judgments, ultimately influencing the effectiveness of fine-grained feedback.
In particular, we critically examine the hypothesis that reward models trained
on fine-grained feedback exhibit superior performance in capturing human
preferences within the text-to-image setting. Our empirical investigations,
encompassing simulated and real human judgments alongside controlled
scenarios, reveal surprisingly that while fine-grained feedback can provide an
advantage under specific conditions, it does not consistently outperform
coarse-grained feedback in the construction of effective reward models. In
fact, coarse-grained feedback occasionally led to superior performance,
highlighting the complexities of human preference modeling and the potential
influence of architectural choices. However, when we do have complete
knowledge of the attributes that may “matter” in preference judgements, as we
explore through a de novo controlled experimental set-up, we do indeed
illuminate the potential value of fine-grained feedback.

This work underscores the need for further exploration of alternative modeling
paradigms capable of effectively harnessing the richness of fine-grained
feedback while addressing the limitations of current approaches. Additionally,
careful consideration of feedback attributes and task characteristics is
crucial for maximizing the value and efficiency of incorporating human
feedback into text-to-image model development.

Our key contributions include:

  * •

Assessment framework: We propose a framework utilizing rejection sampling as a
proxy for large-scale adaptation, enabling efficient evaluation of fine-
grained feedback utility.

  * •

Empirical case studies: Our experiments demonstrate that the additional value
of fine-grained feedback for training text-to-image reward models is highly
conditional.

  * •

Open challenges: We identify and discuss open challenges surrounding the
construction and evaluation of reward models for text-to-image systems based
on fine-grained feedback.

##  2\. Related Work

The rise of generative AI systems and power of RLHF – fueled by human feedback
– has ignited further interest in gleaning insights from human data at scale
to guide improved generation, e.g., to better align images to text prompts
(Yarom et al., 2023; Hu et al., 2023; Kirstain et al., 2023; Xu et al., 2023)
or to personalize image-generation models (von Rutte et al., 2023; Fan et al.,
2023). Improved generative models offer immense potential to transform human
productivity and creativity – provided they adequately meet the diverse needs
and preferences of users. It is not yet clear what kind of feedback is best to
elicit to improve their output – nor how best to incorporate such feedback
when elicited. Works have taken steps to explore human feedback in text-to-
image contexts (Liang et al., 2023; Lee et al., 2023) as well as at
evaluation-time (Lee et al., 2024). Yet, the sheer computational scale of such
generative models renders them challenging to systematically explore design
choices around feedback elicitation and incorporation. Here, we take steps to
address this gap by offering further empirical glimpses into the nuanced value
of fine-grained feedback in generative AI applications.

Several prior works have investigated the interplay between richer forms of
human feedback and model performance. Notably, recent research in the text-to-
text domain has explored the potential of fine-grained feedback (Wu et al.,
2023a; Lee et al., 2023; Liang et al., 2023; Ouyang et al., 2022). In these
studies, “fine-grained” refers to feedback that goes beyond simple binary
judgments or single-score ratings, allowing users to target specific aspects
of the output, such as factual accuracy, logical coherence, or stylistic
elements. This granular feedback enables more precise control over model
adaptation and behavior, leading to outputs that better align with diverse
user preferences. In addition to granularity, richer feedback can also
encompass representations of disagreement and uncertainty in human labels. For
instance, researchers have explored the value of eliciting and learning with
traces of human uncertainty in the form of soft labels (Peterson et al., 2019;
Uma et al., 2020; Collins et al., 2022, 2023b; Sucholutsky et al., 2023).

These studies have primarily focused on training classifiers for traditional
machine learning tasks like image recognition, where human feedback is
typically collected before training. However, practical applications often
necessitate the ability to provide feedback on already trained models,
potentially due to legislative requirements (e.g., Article 13 in the EU AI Act
(European-Parliament, 2024)) or evolving user needs. Concept Bottleneck Models
(CBMs), which map inputs to higher-level attributes before regressing a final
target, offer a potential solution by allowing humans to provide feedback on a
model’s intermediate outputs (Koh et al., 2020), which we explore in the
remainder of our work for modeling fine-grained feedback.

##  3\. Problem Setting

We first provide a primer on the task of learning a reward model for adapting
a generative model; we then introduce our model.

#### Reward Modeling

Our goal is to learn a reward function
$R_{\theta}:\mathcal{X}\rightarrow\mathcal{S}$ that takes in a set of features
$x\in\mathcal{X}$ and produces a scalar score $s\in\mathcal{R}$ indicating the
quality of $x$222The reward models we consider in this work produce pointwise,
rather than pairwise, quality scores for each input.. Many popular reward
learning-based frameworks exist that adapt pre-trained language (Ouyang et
al., 2022) or diffusion-based models (Fan et al., 2024; Black et al., 2023;
Dvijotham et al., 2023) to generate outputs with high reward scores.

How do we build a good $R_{\theta}$? A popular approach is to learn the
parameters of the reward model ($\theta$) from human feedback. That is, we
curate a bank of examples D = {$(x_{1},s_{1})$, $(x_{2},s_{2})$, …
$(x_{N},s_{N})$} where $s$ are the result of human annotations. We can then
update $\theta$ on any standard loss function $\mathcal{L}$ to improve our
mapping from features to the score. Ideally, this produces a reward model
$R_{\theta}$ that matches human preferences over “good” $x$.

#### Fine-Grained Feedback

But what makes an $x$ “good”? Casper et al. raise several crucial open
questions for learning reward models from human feedback, e.g., heterogeneity
across humans and the challenge of going beyond single aggregate preferences.
This task becomes all the more challenging when we consider generative models
producing complex and highly structured outputs such as text-to-image models.
For example, an image might be visually compelling and highly creative, but
not necessarily aligned to either the main intention or minor attributes of
the input prompt (or vice versa). Moreover, there may be multiple factors that
together determine the quality or aesthetic value of the image (e.g., it does
not have artifacts, uses an appealing color palette etc).

We consider the setting of eliciting _fine-grained feedback_ where human
annotators are asked to provide a set of $M$ scores
$s_{i}=(s^{1}_{i},s^{2}_{i},...,s^{M}_{i})$ for each example $x_{i}$ with
$i\in{1,\ldots,N}$. The individual scores are scalar values representing the
degree to which the prompt-image pair satisfies some particular aspect of
quality (e.g. photorealism). Our goal then is to learn a “good” (see Section
4) $R_{\theta}$ from such feedback. We refer to reward models learned from
more than one feedback as “fine-grained reward models” compared to those
trained on a single aggregate score, $\tilde{s}$ (“coarse-reward models”) and
discuss different ways to concretely operationalize “good” in our experiments.

#### Costs

While there seems to be intuitive value to collecting finer-grained feedback
over more attributes, the elicitation of such feedback necessarily incurs some
additional time (or other resource) cost, over the cost of collecting coarse-
grained feedback. We consider the setting where each $j=1\ldots M$ dimension
of feedback is associated with some elicitation cost $c_{j}>0$. For our
computational experiments, we assume each form of feedback has equivalent
procurement costs. We discuss deviations from this assumption in Section
6.2.1.

##  4\. Reward Models from Fine-Grained Feedback

What kind of model structure empowers us to effectively learn from such rich
feedback? We consider a two-stage structure which first predicts each fine-
grained attributes and then aggregates the scores, similar to (Wu et al.,
2023a). This structure parallels a Concept Bottleneck Model (Koh et al., 2020)
(see above), wherein our concepts our fine-grained attributes (is the image
malformed? is the image blurry? is the image aligned to the text for verbs?).
Rather than learn a mapping directly from $R_{\theta}:x\rightarrow\tilde{s}$,
we learn two sets of parameters: $f_{\phi}:x\rightarrow s^{1},...s^{M}$ and
$g_{\psi}:s^{1},...s^{M}\rightarrow\tilde{s}$. The functions then compose to
produce a single aggregate score for a given input
$g_{\psi}(f_{\phi}(x))=\tilde{s}$ with the added benefit that we can inspect
the fine-grained attributes predicted rendering our model more interpretable.

#### Our Model

There are many functional forms that our two-stage modeling pipeline can take
on. Here, we let $f$ be a multi-headed multi-layer perception (MLP). Following
popular practice in the CBM literature, $g$ is a simple linear aggregator (Koh
et al., 2020; Espinosa Zarlenga et al., 2022; Margeloiu et al., 2021; Collins
et al., 2023a). We consider the sequential CBM setting (Koh et al., 2020),
learning $\phi$ then $\psi$ separately.

#### Embeddings

Additionally, in the text-to-image setting, we need some way of providing the
multimodal stimulus (text and image) as input to our model. Following standard
practice, we use learned embeddings of the image pixels and text tokens
respectively. We assume that the feature extractors which produce the
embeddings are fixed (see Appendix A.1). This approach has a benefit in common
real-world scenarios where practitioners only have access to features in
black-box fashion.

##  5\. Experiment Outline

It is natural to expect that more informative supervision (finer-grained
feedback) will be better for RLHF scenarios. In particular, we hypothesize
that fine-grained feedback will be valuable when training reward models in
regimes with few examples, following the literature around informativeness of
label supervision (Sucholutsky et al., 2023). We posit based on previous
results in text-to-text generation e.g. (Wu et al., 2023a) that reward models
trained from fine-grained feedback will be able to better capture preference
judgments than a model trained on coarse preference judgments alone.

To address our hypothesis, we design a series of computational experiments
with feedback of varying levels of granularity and dataset sizes. We consider
two classes of fine-grained feedback important for measuring the quality of a
text-to-image generation following (Lee et al., 2024):

  1. (1)

Image quality: whether the image itself meets a desired criteria (e.g.,
photorealistic, not malformed).

  2. (2)

Text-image alignment: whether the image is aligned to the text according to a
particular semantic category (e.g., attributes from the prompt are captured in
the image).

Recall in our reward model that we employ a two-stage pipeline: first
predicting fine-grained feedback, then predicting aggregate targets. As such,
like in CBMs, we need two sets of labels (over fine-grained attributes and
aggregate targets). In this work, the additional fine-grained feedback signals
are inferred by querying the state-of-the-art PaLI model (Chen et al., 2022)
(see Section 6) — we leave the expansion of collecting and incorporating
granular feedback from human annotators for future work. In all experiments,
the coarse- and fine-grained models are trained on the same final targets —
real human preference judgments in Experiment 1 and a synthetic target
permitting more controlled exploration in Experiment 2.

We consider two experimental settings. In the first, we explore the impact of
fine-grained feedback for capturing real human preference judgments. In light
of our negative result on the utility of learning CBM-based reward models from
fine-grained feedback, we design a second, controlled domain to further
disentangle the source of the poor performance.

##  6\. Experiment 1: Predicting Human Preference Judgments from Fine-Grained
Feedback

Figure 3. Comparing reward models trained on coarse feedback (i.e., direct
human preference judgments; black) against CBM-based models learned from fine-
grained feedback. Reward models are differentiated by whether they were
trained on granular feedback only about image quality (blue), image-text
alignment (red), or both (purple). Left: Each point represents a reward model
trained on $N$ image-prompt examples (x axis); ROC-AUC of the binary reward
against held-out human preference judgments is presented on the y axis. Higher
is better. Right: The same reward models, where the x axis (presented on a log
scale) depicts estimated annotation cost, if each attribute is assumed to be
equally costly to procure.

In this section, we consider the task of predicting real human preference
judgements. We first overview our experimental set-up before presenting our
results. We close with a discussion of what may underlie our observations that
fine-grained feedback may not be preferable in this setting.

###  6.1. Experimental Details

#### Data

We use the approximately 5k images from (Dutta et al., 2024). Images were
generated from DALL-E (Ramesh et al., 2021) and Stable Diffusion (Rombach et
al., 2022) for over 1.3k text prompts from PartiPrompts (Yu et al., 2022).
Each image has been annotated by nine humans, where images are scored rated on
a scale of 1-4 for how “satisfied” the viewer is with the image for the
intended prompt, with a particular motivational context in mind (i.e.,
participants were asked to rate how good an image-prompt pair is for a
particular motivational context: for use as a phone background, graphic
t-shirt, or presentation slide-deck). Here, we average all context-conditioned
scores to form a single preference (“goodness”) score per image-prompt pair.
This score forms our “coarse”-grained preference of interest. We conduct a
50/25/25 train/val/test split at the level of the prompts (as there are four
images per prompt). Additional details on data processing are included in the
Appendix.

#### Fine-Grained Attributes

The generative output of text-to-image models can be scored along both
fidelity of the image to the prompt (text-image alignment (Yarom et al.,
2023)) and image quality. We consider fine-grained attributes along each type:
image quality and prompt alignment. For each image, we elicit granular quality
attribute scores by querying PaLI (Chen et al., 2022) as discussed above. PaLI
is a multimodal model which takes as input image and text and produces a text
response; we can therefore use the model to simulate crowdsourced responses to
text-image pairs by asking for a numerical score along some dimension. In
particular, we train our model on eight image quality attributes (whether the
image is distorted, photorealistic, bright, captivating, chaotic, visually
compelling, disturbing, or funny). We select these attributes to span a range
of valence and relevance. Additional details on granular queries are included
in Appendix A.3.1.

To gather granular feedback for text-image alignment, we employ the $VQ^{2}$
framework from Yarom et al.. $VQ^{2}$ evaluates image-text alignment by
generating question-answer pairs from the input text. For example, the text “A
black apple and a green backpack” could yield the question-answer pair “What
is the fruit in the picture? - Apple” or “What color is the backpack? -
Green”. These pairs are then assessed against the image using a Visual
Question Answering model, which produces a “Yes” probability indicating the
validity of each pair. The average of these “Yes” probabilities across all
question-answer pairs constitutes the final $VQ^{2}$ score, reflecting the
overall image-text alignment. We group $VQ^{2}$ questions using a customized
modular semantic parser into four attribute classes, whether the image matches
the text in regards to: any actions/verbs mentioned in the prompt, any
attributes/adjectives, any objects/nouns, or any relations mentioned in the
prompt. See Appendix Section A.3.2 for further details on our feedback
extraction.

#### Models

We compare a reward model trained directly on coarse feedback (the aggregate
human scores from Dutta et al.) against a suite of CBMs trained on varying
amounts ($d$) and types (e.g., image-based or prompt-based) fine-grained
attributes. Modeling details are included in Appendix LABEL:model-details.
Again, each attribute of fine-grained feedback acts like a “concept” predicted
in the first stage of our two-stage model (Figure 2.

#### Evaluation

We evaluate models in two ways:

  * •

Accuracy of the reward models, in terms of predicting the aggregated human
“goodness” score for held-out examples from (Dutta et al., 2024) (scored with
ROC-AUC), and

  * •

Simulated adaptation of a downstream text-to-image generative model through
rejection sampling, wherein we use our reward models to score generated
examples, and check whether people agree with the relative rankings (i.e.,
that the stimulus rated higher by a fine-grained model indeed is preferred to
a human over a stimulus rated highly by a coarse-grained model). We design and
conduct a series of human evaluations here along aggregate and fine-grained
dimensions.

###  6.2. Results

We compare a reward model trained directly on the coarse human preferences
judgments against our CBM-based fine-grained models built from varying classes
of fine-grained feedback. We train the suite of models over varying number of
training examples and test on held-out image-prompt pairs.

Interestingly, in Figure 3, we find that training reward models simply on
coarse-grained feedback is not only more cost-efficient for a given budget
(i.e., if we assume equal costs for collecting labels for each additional
fine-grained attribute), but yield better fits than the putatively more
“information-rich” fine-grained reward models. We discuss the impact of
deviations from the equal-cost assumption in Section 6.2.1.

We do see that combining information about image quality with text-alignment
boosts performance (see Figure 3) over image quality information alone, but it
is clear that the extra information in such attributes is not inducing higher
match to held-out (in-distribution) preference judgments versus simply using
the coarse-grained feedback. This raises the question that if such feedback
has “more information” why are we seeing a performance drop compared to the
coarse-grained model? We posit several hypotheses for why this may be the
case. There may be:

  1. (1)

Challenges stemming from the human data, e.g. our targets here are averages of
human judgments originally produced for separate use cases (it may be better
to model the full distribution), and by the motivation of our work, we may not
want to try to match;

  2. (2)

Misalignment between the synthetic feedback and real human judgment, in the
values that we produce from PaLI and $VQ^{2}$ on the one hand, and the
attributes that we collect from raters;

  3. (3)

Model expressivity: CBMs may not be adequately expressive to capture the
nuances in the fine-grained feedback;

  4. (4)

Or, it could be that fine-grained feedback is fundamentally not useful here
and provides no added value to the aggregate attributes.

####  6.2.1. Evaluation Proxy for Adapting a Generative Model with Reward
Models

We take a step to address the first point by running a fresh human evaluation.
Recall, one of our goals for learning reward models from fine-grained feedback
is to better tailor adaptation of downstream generative models and improve the
quality of their output rather than just matching aggregate human preference
judgments – we made a case in our motivation that aggregate preference
judgments can obfuscate important information. We simulate adapting a
downstream generative model, and getting a sense of preference along the fine-
grained attributes (focusing on the image quality attributes for simplicity).
Due to computational costs, it is not always sensible to test out the quality
of a gamut of reward models by adapting text-to-image models (see Section 8).
Instead, we follow Lee et al. in running rejection sampling with our reward
models as a proxy for adapting a generative model directly. That is, we draw
samples from a generative model and use our reward models to score the
outputs; we then run head-to-head preference judgments over the generations
favored by the respective reward models. Here, we consider two reward models:
the model trained only on coarse judgments (i.e., the aggregate human
preference judgments from the train set) and our model trained on fine-grained
simulated attribute annotations333We consider our most “fine-grained”-trained
model with $d=12$ attributes. Details on the sampled images and annotation
procedures are included in the Appendix Section A.5.

We find in Table 1 (first row) that there is not a clear preference for the
images that the reward model trained on coarse-grained feedback preferentially
sample compared to the model trained on fine-grained feedback (i.e.,
annotators indicate that they prefer the text-image prompt rated more highly
by the coarse-grained model than the fine-grained model for about 25% of the
examples we survey, and vice versa for fine- over coarse-). This finding
suggests that the difference between the coarse- and fine-grained feedback
trained models are not as strong as our in-distribution prediction-based
evaluation (from Sec. 6.2) make them appear as we move out-of-distribution to
a new task: scoring generated images instead of judging (in-distribution)
reward model accuracy. However, we observe high rates of uncertainty in the
human judgements of which image is better along each attribute (annotators
express that they are unsure which image they prefer for approximately 50% of
the samples).

Such high levels of annotator uncertainty are exacerbated when we elicit
judgments over individual dimensions (see Table 1). We observe strong signals
only along the distorted and brightness dimensions. These findings suggest: 1)
the preference for the results of the coarse-grained model are not consistent,
and 2) eliciting preferences over fine-grained attributes may not be
particularly meaningful nor informative. We might observe more interesting
preference judgments along granular attributes with a different stimuli pool.
Nonetheless, our results urge caution on the blind elicitation and
incorporation of more granular feedback from annotators – more is not always
better (or at least not always informative). Further, we observe that,
contrary to our simplifying assumptions, annotation times per dimension are
not uniform (see Table 2), underscoring the importance of judiciously
recognizing when to collect fine-grained attributes, and which to collect.

Table 1. SxS preference scores, from rejection sampling. From these studies, we’ll get out a proportion of prompt-image pairs for which our raters prefer the example scored highly by the fine-grained model vs. the coarse-grained model (and vice versa), as well as the proportion where the annotators are unsure which is best. We compare pairwise general preferences (top row) as well as preference along particular granular attributes (rows below the line). Scores depict % of images where coarse- vs. fine-grained were preferred (or people were uncertain), where % depict the votes for each preferred image over the total number of votes. Feedback Type | Coarse Pref | Fine-Grained Pref | Unsure
---|---|---|---
Aggregate | 25.6 | 24.9 | 49.5
Distorted | 36.9 | 31.8 | 31.3
Bright | 30.2 | 26.1 | 43.6
Captivating | 18.4 | 19.1 | 62.5
Photorealistic | 31.1 | 31.4 | 37.5
Chaotic | 13.7 | 12.4 | 73.9
Visually compelling | 20.6 | 15.8 | 63.6
Disturbing | 8.2 | 8.6 | 83.2
Funny | 0.5 | 0.9 | 98.6
Table 2. Average annotator answer time (in seconds) for each annotation task. Feedback Type | Time (s)
---|---
Aggregate | 52.7
Distorted | 56.1
Bright | 18.4
Captivating | 20.2
Photorealistic | 19.4
Chaotic | 24.1
Visually compelling | 16.2
Disturbing | 19.2
Funny | 12.8

##  7\. Experiment 2: Controlled Granular Image Quality Assessments

As mentioned, there could be a variety of reasons that we observe a null
result for the utility of fine-grained feedback in Experiment 1. Crucially, we
do not know what attributes people are considering when they are making their
preference judgments. Perhaps if we elicited the correct attributes, we would
be able to learn better reward models? As such, we are motivated to create a
more controlled experimental setup where we do know the attributes that are
being considered in the final preference judgment.

To address this gap, we design a second domain wherein we have complete
knowledge over the attributes that inform the target preference. Unlike in
Experiment 1, the target here is completely synthetic – we build a decision
tree over the fine-grained attributes (obtained from simulated AI feedback)
that exactly determines the quality of an attribute444As in Experiment 1, we
predict a single quality score per point, not a preference rating.. This
enables us direct control, but again, necessitates cautious interpretation as
it side-steps the question (which may drive the null result in Experiment 1)
of whether our AI-based feedback is even aligned with the judgments humans –
or particular humans – make.

For simplicity, we focus on the case of only image-dependent evaluation (i.e.,
just considering the image attributes along and not those of the prompt).
Since the ground truth is exactly captured by a custom decision tree with the
simulated AI feedback attributes as leaves; adequately capturing and modeling
each dimension of granular feedback (i.e., the leaves), should be sufficient
to learn reward models that accurately predict the target quality score. We
emphasize that we construct this experiment to explore the impact of feedback
granularity where we have direct access to the target (and know that it is
constructed from multiple attributes); this set of experiments are not
indicative of which attributes matter for human aesthetic judgements.

###  7.1. Experimental Setup

#### Data and Evaluation

We consider the same images from Dutta et al. as in Experiment 1. For
simplicity, however, we consider only the images; our experiments in this
Experiment do not depend on the prompt.

#### Controlled Target

We design a controlled and fully intepretable target preference score formed
from simulated attributes that enables us to more precisely understand the
impact of granular feedback than trying to capture potentially nebulous real
human preference scores. Specifically, we design a decision tree which takes
in an image and at each node assesses a particular attribute; specifically, it
checks whether it is photorealistic, then visually compelling, then chaotic.
The output is a binary score which we take as representing whether an image is
“good” or “bad”.

We evaluate reward models with ROC-AUC on held-out decision tree scored
examples. We reiterate that the decision tree is intended to serve as a
controlled target where we know which attributes underlie the final preference
score; we do not claim this decision tree models human preference judgments
nor is generalizable in all contexts.

Figure 4. Comparing reward models trained on varying levels of granularity. As
in Figure 3, each point represents a reward model trained on $N$ images.
Models are scored according to the contrived decision tree on held-out
examples. We compare a model trained directly on the single scalar decision
tree scores (black) against a suite CBM-based fine-grained models trained on:
1) the same three attributes which make up the decision tree (red), 2) the
same three attributes as the decision tree along with the remainder of the
full set of image attributes under consideration (blue), and 3) only
attributes not included in the decision tree (orange).

#### Models

We employ the same model architectures and training procedures as in our
Experiment 1 experiments, with the exception that we only feed the image
embeddings as input (further explorations of joint text-image modeling are
important next steps). We train a coarse-grained model on the final output of
the controlled decision tree, and compare this model against fine-grained CBMs
which have access to varying numbers of image attributes (which may or may not
include the attributes used to form the decision tree).

###  7.2. Results

We start by considering the setting where our fine-grained CBM has access to
the same attributes as in the target decision tree; i.e., we compare a model
trained directly on only the coarse score from the decision tree versus a CBM
trained over the same attributes that make up the decision tree. Here, by
design of our controlled decision tree, we ought to achieve the same or better
performance to the coarse-grained setting; indeed, we do find in Figure 4 that
we can achieve better performance by training on fine-grained feedback. Of
note, adding more attributes beyond those in the “true” decision tree have
little positive impact on reward quality, and may not be economical to elicit.
While these data indicate that fine-grained feedback can be used to learn
better reward models than those from aggregate preference judgements alone,
one may ask why there is any gap between our learned reward models and the
maximal achievable fit (which should be 100% as the target is formed from in-
distribution feedback). The gap suggests that our reward model is not as
strong as it could be, possibly stemming from our embeddings (as discussed in
the Open Questions). However, qualitative inspection of the reward models in
the Appendix reveals that the fine-grained attribute models do reflect
meaningful differences along the attributes, for example, distinguishing
between images that are photorealistic or not.

Barring model architecture selection – we make a crucial assumption – that we
know the true attributes. What if we do not have access to the attributes that
form the decision tree? To begin to explore this question, we consider the
setting where there is an attribute mismatch (yellow points in Fig 4). Here,
we see a dramatic drop in performance. These data point to the importance of
incorporating the right attributes if building a fine-grained reward model;
here, we simulated and precisely controlled what attributes matter (by design
of our decision tree). In practice, we may not know which attributes are
“right” to elicit: the precise situation we found ourselves in for Experiment
1, potentially underlying our null result. One could envision selecting
attributes that are most correlated with the target (see Figure 5), but this
requires having access to target annotations. Determining what attributes to
elicit is a key open challenge, as we discuss next.

Figure 5. Estimated similarity between PaLI scores for different attributes.
We depict the proportion of images in the train set for which PaLI marks an
image as having the same attribute (e.g., the cell blurry and malformed
highlights that PaLI marks an image as blurry and malformed, or not blurry and
not malformed for 71% of the examples). Darker red means higher level of
similarity in scores; yellow represents lower similarity.

##  8\. Open Challenges

Nicely, we indeed find in Experiment 2 that fine-grained feedback can be
useful to model if we know the attributes. But crucially, how do we actually
find these attributes? Our work urges further study of what attributes humans
consider when making preference judgments and what is economical to elicit.
Our work exposes key challenges that arise in the study of the impact of
choice of feedback on reward models for adapting generative models.

###  8.1. Which Attributes to Elicit?

Our work points to a key challenge for practitioners interested in collecting
fine-grained feedback: what attributes should you elicit? In Experiment 2, we
demonstrate that reward model performance may suffer if the elicited
attributes do not match those that form the target preferences. How can we
know what attributes we should elicit? Such a question grows more challenging
when we consider individual differences. Different attributes may matter to
different people and depend on context (Gordon et al., 2022; Kirk et al.,
2024b; Dutta et al., 2024). We do not want reward models to collapse to a
monoculture (Kleinberg and Raghavan, 2021; Bommasani et al., 2022), but also
ought to be mindful of the risks of personalization (Kirk et al., 2024a).

Additionally, it is not enough just to have the “right” attributes. In
practice, elicitation needs to balance informativity and cost. We already see
that attributes may take different amounts of time to annotate in Experiment 1
(see see Table 2) and in Experiment 2, we demonstrate that, in simulation,
adding attributes is not always valuable. Interdisciplinary works that
straddle AI, cognitive science, and human-computer interaction are already
exploring the impact of requiring humans to provide feedback on many
attributes, noting that such a practice can overwhelm cognitive load and risk
bringing more error into downstream modeling (Sucholutsky et al., 2023;
Ramaswamy et al., 2022; Barker et al., 2023). Indeed, we do not want to waste
annotations on attributes where users are highly unsure (though future work
can explore the benefits from learning with uncertainty at feedback time
(Collins et al., 2023a)). And further, we already see in our human studies
that annotators spend substantially different time annotating some attributes
over others. Nicely, the CBM model class naturally supports the implementation
and study of cost-aware acquisition strategies for human feedback (Chauhan et
al., 2022; Sheth et al., 2022; Shin et al., 2023; Espinosa Zarlenga et al.,
2024). We see promise in the hybridization of elicitation development for such
models and the determination of which attributes to elicit.

###  8.2. Reward Model Structure

Yet, perhaps CBMs are not the best model structure for fine-grained feedback.
Indeed, recent work has raised questions about the ability of CBM-based
systems to effectively handle rich, soft-labeled feedback if not explicitly
trained to do so Collins et al.. This highlights the importance of considering
the interplay between feedback type and model architecture when designing
systems for human-in-the-loop adaptation. We reiterate that our work is a
preliminary exploration of ways to learn reward models from fine-grained
feedback. It is likely that alternate modeling choices induce different cost-
benefit analyses on the value of learning from fine-grained feedback; we look
forward to future works that explore such possibilities. For instance, several
other approaches have been proposed e.g. (Rame et al., 2024; Liang et al.,
2023). We see the design of model architectures which incorporate information
efficiently from granular feedback, and can flexibly grow to handle new
dimensions (e.g, if we learn that a new attribute actually matters more to
annotators that we had not previously modeled), as ripe for future work.
Moreover, the image and text embeddings we considered in our work were always
fixed. It is possible that different choices of embedding, or even jointly
learning embeddings, may improve performance and perhaps salvage the utility
of a CBM-based architecture.

###  8.3. Accessible, Efficient Evaluation

However, rapidly evaluating such modeling choices in the context of assessing
reward models is not easy. The massive computational overhead of actually
training and adapting large-scale generative models poses a crucial practical
challenge for researchers attempting to study what kind of feedback yields
powerful reward models. In our work, we attempted to deal with these
challenges in two ways: 1) computational experiments wherein we have direct
access to the target, and 2) simulating the impact of adapting a generative
model downstream through our rejection sampling paradigm. While we hope our
experimental approach illuminates one potential workflow that other
researchers can take, more work is needed to characterize how much of a gap
there is between such proxy settings and at-scale generative model adaptation.

###  8.4. Human vs. Model Feedback

Computational overhead is not the only challenge: we are also limited by the
elicitation of feedback itself. Eliciting information from humans can be
expensive. Here, our granular feedback was derived from an AI system, not
humans. A natural question is how well our simulated feedback here actually
correlates to human judgments. It is possible that our null results in
Experiment 1 stem from a mismatch between human and model judgments over the
granular attributes, either or both along the image quality and text-image
alignment dimensions. While there is a push to employ AI-generated feedback
rather than humans for scalable generative evaluation (Wu et al., 2023b;
Gilardi et al., 2023), it is essential to understand where such feedback may
diverges from human expectations.

##  9\. Conclusion

In this work, we uncover at least one setting where fine-grained feedback may
not help immediately, under particular caveats (model choice, embedding
efficacy, fidelity of fine-grained feedback, choice of attributes, minimal
fine-tuning). Our work urges practitioners to consider carefully, particularly
under a fixed annotation budget, what kind of feedback is useful and efficient
to collect. It may not always make sense to collect fine-grained feedback –
and even if it does, some attributes may be more valuable than others. We need
more interdisciplinary studies to identify what attributes people are
considering and how well they align with model-derived feedback, and which
attributes are worth encouraging people to consider to inform preference
judgments for adapting text-to-image models. We hope our work inspires further
study of efficient and robust ways of interleaving human and machine
computation to study and improve generative models in a way that reflects the
nuance replete in the world in which such systems are being deployed.

#### Acknowledgments

We thank Ravi Rajakumar, Ellie Pavlick, and Sunny Mak for helpful feedback on
the data used in the work. We additionally thank Jordi Pont-Tuset, Isabelle
Guyon, and Andreas Terzis for helpful conversations and Alena Butryna for
tremendous help coordinating our participant studies. AW acknowledges support
from a Turing AI Fellowship under grant EP/V025279/1, and the Leverhulme Trust
via CFI.

## References

  * (1)
  * Barker et al. (2023) Matthew Barker, Katherine M Collins, Krishnamurthy Dvijotham, Adrian Weller, and Umang Bhatt. 2023.  Selective concept models: Permitting stakeholder customisation at test-time. In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , Vol. 11. 2–13.
  * Black et al. (2023) Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. 2023.  Training Diffusion Models with Reinforcement Learning.  _arXiv e-prints_ (2023), arXiv–2305.
  * Bommasani et al. (2022) Rishi Bommasani, Kathleen A Creel, Ananya Kumar, Dan Jurafsky, and Percy S Liang. 2022.  Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?  _Advances in Neural Information Processing Systems_ 35 (2022), 3663–3678.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023\.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _arXiv preprint arXiv:2307.15217_ (2023).
  * Chauhan et al. (2022) Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. 2022.  Interactive Concept Bottleneck Models.  _arXiv preprint arXiv:2212.07430_ (2022).
  * Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. 2022\.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_ (2022).
  * Chen et al. (2023) Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Alexander Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. 2023\.  PaLI: A Jointly-Scaled Multilingual Language-Image Model.  https://arxiv.org/abs/2209.06794
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ 30 (2017).
  * Collins et al. (2023a) Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. 2023a.  Human uncertainty in concept-based ai systems. In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_. 869–889.
  * Collins et al. (2023b) Katherine M Collins, Umang Bhatt, Weiyang Liu, Vihari Piratla, Ilia Sucholutsky, Bradley Love, and Adrian Weller. 2023b.  Human-in-the-loop mixup. In _Uncertainty in Artificial Intelligence_. PMLR, 454–464.
  * Collins et al. (2022) Katherine M Collins, Umang Bhatt, and Adrian Weller. 2022.  Eliciting and Learning with Soft Labels from Every Annotator. In _HCOMP_.
  * Dutta et al. (2024) Senjuti Dutta, Sherol Chen, Sunny Mak, Amnah Ahmad, Katherine Collins, Alena Butryna, Deepak Ramachandran, Krishnamurthy Dvijotham, Ellie Pavlick, and Ravi Rajakumar. 2024.  Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction.  _arXiv preprint arXiv:2403.05576_ (2024).
  * Dvijotham et al. (2023) Krishnamurthy Dj Dvijotham, Shayegan Omidshafiei, Kimin Lee, Katherine M Collins, Deepak Ramachandran, Adrian Weller, Mohammad Ghavamzadeh, Milad Nasr, Ying Fan, and Jeremiah Zhe Liu. 2023.  Algorithms for Optimal Adaptation ofDiffusion Models to Reward Functions. In _ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems_.
  * Espinosa Zarlenga et al. (2022) Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. 2022.  Concept Embedding Models.  https://doi.org/10.48550/ARXIV.2209.09056
  * Espinosa Zarlenga et al. (2024) Mateo Espinosa Zarlenga, Katie Collins, Krishnamurthy Dvijotham, Adrian Weller, Zohreh Shams, and Mateja Jamnik. 2024.  Learning to Receive Help: Intervention-Aware Concept Embedding Models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * European-Parliament (2024) European-Parliament. 2024.  https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138-FNL-COR01_EN.pdf
  * Fan et al. (2023) Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. 2023.  DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models.  _arXiv preprint arXiv:2305.16381_ (2023).
  * Fan et al. (2024) Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. 2024.  Reinforcement learning for fine-tuning text-to-image diffusion models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences_ 120, 30 (2023), e2305016120.
  * Gordon et al. (2022) Mitchell L Gordon, Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022.  Jury learning: Integrating dissenting voices into machine learning models. In _CHI Conference on Human Factors in Computing Systems_. 1–19.
  * Honnibal et al. (2020) Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020.  spaCy: Industrial-strength Natural Language Processing in Python.  (2020).  https://doi.org/10.5281/zenodo.1212303
  * Hu et al. (2023) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023.  TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering.  _arXiv e-prints_ (2023), arXiv–2303.
  * Kirk et al. (2024a) Hannah Kirk, Bertram Vidgen, Paul Röttger, and Scott Hale. 2024a.  The benefits, risks and bounds of personalizing the alignment of large language models to individuals.  _Nature Machine Intelligence_ 6 (04 2024), 383–392.  https://doi.org/10.1038/s42256-024-00820-y
  * Kirk et al. (2024b) Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024b.  The PRISM Alignment Project: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models.  _arXiv preprint arXiv:2404.16019_ (2024).
  * Kirstain et al. (2023) Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  _arXiv preprint arXiv:2305.01569_ (2023).
  * Kleinberg and Raghavan (2021) Jon Kleinberg and Manish Raghavan. 2021.  Algorithmic monoculture and social welfare.  _Proceedings of the National Academy of Sciences_ 118, 22 (2021), e2018340118.
  * Koh et al. (2020) Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020.  Concept bottleneck models. In _ICML_. PMLR, 5338–5348.
  * Lee et al. (2023) Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. 2023.  Aligning text-to-image models using human feedback.  _arXiv preprint arXiv:2302.12192_ (2023).
  * Lee et al. (2024) Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Teufel, Marco Bellagente, et al. 2024\.  Holistic evaluation of text-to-image models.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Liang et al. (2023) Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Junjie Ke, Krishnamurthy Dj Dvijotham, Katie Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, and Vidhya Navalpakkam. 2023.  Rich Human Feedback for Text-to-Image Generation.  arXiv:2312.10240 [cs.CV]
  * Margeloiu et al. (2021) Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. 2021.  Do concept bottleneck models learn as intended?  _arXiv preprint arXiv:2105.04289_ (2021).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ 35 (2022), 27730–27744.  arXiv:2203.02155 [cs.CL]
  * Peterson et al. (2019) Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. 2019.  Human uncertainty makes classification more robust. In _ICCV_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Ramaswamy et al. (2022) Vikram V Ramaswamy, Sunnie SY Kim, Ruth Fong, and Olga Russakovsky. 2022.  Overlooked factors in concept-based explanations: Dataset choice, concept salience, and human capability.  _arXiv e-prints_ (2022), arXiv–2207.
  * Rame et al. (2024) Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2024.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation. In _International Conference on Machine Learning_. PMLR, 8821–8831.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10684–10695.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. 2022\.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ 35 (2022), 25278–25294.
  * Sheth et al. (2022) Ivaxi Sheth, Aamer Abdul Rahman, Laya Rafiee Sevyeri, Mohammad Havaei, and Samira Ebrahimi Kahou. 2022.  Learning from uncertain concepts via test time interventions. In _Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022_.
  * Shin et al. (2023) Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. 2023.  A Closer Look at the Intervention Procedure of Concept Bottleneck Models.  _arXiv preprint arXiv:2302.14260_ (2023).
  * Sucholutsky et al. (2023) Ilia Sucholutsky, Ruairidh M Battleday, Katherine M Collins, Raja Marjieh, Joshua Peterson, Pulkit Singh, Umang Bhatt, Nori Jacoby, Adrian Weller, and Thomas L Griffiths. 2023.  On the informativeness of supervision signals. In _Uncertainty in Artificial Intelligence_. PMLR, 2036–2046.
  * Uma et al. (2020) Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020.  A Case for Soft Loss Functions.  _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ 8, 1 (Oct. 2020), 173–177.  https://ojs.aaai.org/index.php/HCOMP/article/view/7478
  * von Rutte et al. (2023) Dimitri von Rutte, Elisabetta Fedele, Jonathan Thomm, and Lukas Wolf. 2023.  FABRIC: Personalizing Diffusion Models with Iterative Feedback.  (2023).  arXiv:2307.10159 [cs.CV]
  * Wu et al. (2023b) Tongshuang Wu, Haiyi Zhu, Maya Albayrak, Alexis Axon, Amanda Bertsch, Wenxing Deng, Ziqi Ding, Bill Guo, Sireesh Gururaja, Tzu-Sheng Kuo, et al. 2023b.  Llms as workers in human-computational algorithms? replicating crowdsourcing pipelines with llms.  _arXiv preprint arXiv:2307.10168_ (2023).
  * Wu et al. (2023a) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023a.  Fine-Grained Human Feedback Gives Better Rewards for Language Model Training.  _NeurIPS_ (2023).
  * Xu et al. (2023) Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2023.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  _arXiv preprint arXiv:2304.05977_ (2023).
  * Yarom et al. (2023) Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. 2023.  What You See is What You Read? Improving Text-Image Alignment Evaluation.  _arXiv preprint arXiv:2305.10400_ (2023).
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. 2022\.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ 2, 3 (2022), 5.

##  Appendix A Appendix

###  A.1. Image and Text Embeddings

Image and text inputs are represented as dense embedding vectors. We use CLIP
to extract embeddings for text captions (Radford et al., 2021). Through
preliminary experimentation, we found the frozen CLIP embeddings has been
shown to poorly capture aesthetic properties of images; as such, we opted to
extract the output of the first layer of the LAOIN Aeshetics model (Schuhmann
et al., 2022) as our image embeddings. Embeddings are concatenated across
modalities for Experiment 1; only the image embeddings are fed as used as
input for training and inference in Experiment 2. Image and text embeddings
are frozen for all models. Future work could explore the impact of jointly
fine-tuning the embeddings and predicting granular feedback.

###  A.2. Reward Models

As introduced in Section 3, we run a two-phased training procedure for fine-
grained reward models. We first train a mapping from the input embeddings
(described above) to individual attributes; this takes the form of a multi-
headed MLP. We then learn a simple linear aggregator over the outputs of the
multi-headed MLP. All stages leverage binary classifiers for a form of
feedback (specifically, multi-class binary classifiers when we have multiple
attributes); the input to the Stage 2 linear aggregator for all settings is
the sigmoided logit from Stage 1. The coarse-grained baseline only involves
stage one (we directly map from the input embeddings to the coarse score);
i.e., the coarse-grained model is not a CBM.

We emphasize that alternate ways of training on coarse- and fine-grained
feedback are feasible; for instance, here, we only consider point-wise scores,
rather than pairwise-based training.

We use the same model architecture for Stage 1 of all reward models. Models
take the form of an MLP with two 256-dimension hidden layers and are trained
for 100 epochs, with a learning rate of 1e-4. We use a batch size of 128. MLP
training is implemented in jax. Linear aggregators are trained with class-
balancing using the Logistic Regression scikit-learn model; all other sckit-
learn defaults were used.

###  A.3. Additional Details on Forms of Fine-Grained Feedback

####  A.3.1. Image Quality

We query PaLI, a large-scale language-and-text model(Chen et al., 2023), as to
whether a given image meets a particular attribute. Specifically, we ask
yes/no questions of the form: “is the image [attribute]” where attribute is
$\in\\{$blurry, distorted, visually compelling, captivating, funny,
photorealistic, bright, disturbing, chaotic$\\}$. We select these subset of
attributes to span a range of axes along which one may consider eliciting
feedback: positive / negative framing; relevant / irrelevant. We normalize the
resulting scores as a softmax over the “yes” and “no” returned scores.

####  A.3.2. Prompt Alignment

We build on the $VQ^{2}$ method developed in (Yarom et al., 2023) to measure
the alignment between a prompt and the image. As discussed, $VQ^{2}$ takes as
input an image and a prompt (e.g., “green dog to the left of the river”) and
generates a series of binary questions that the image ought to address if fit
for the prompt (e.g., “is there a dog?”, “is the dog green?”, “is the dog
running along the left of the river?”). Each question is then assessed against
the image, wherein the probability that the question can be answered as “Yes”
is computed. The mean of the “Yes” probabilities forms the final $VQ^{2}$
score.

Here, we construct four scores to reflect different ways in which an image may
be aligned to a prompt. An image may be aligned in its: 1) representations of
objects / nouns (e.g., “is there a dog?”), 2) attributes / adjectives (“is the
dog green?”), 3) actions / verbs (“is the dog running?”), and 4) relations
(“is the dog to the left of the river?”). We categorize each of the questions
generated by $VQ^{2}$ into one of these categories using a custom semantic
parser built from spaCy (Honnibal et al., 2020) with hand-crafted rules to
catch exceptions. Two authors from our author team manually inspected hundreds
of the categorizations to affirm their quality – while the parsing was
generally sensible – we note that it is not perfect and likely could be
improved in future work. We then average the $VQ^{2}$ scores for all questions
grouped in a category, which are then thresholded into a binary aligned/not
aligned which we use as feedback. Image-prompt pairs for which $VQ^{2}$ does
not generate a question for a particular class are binarized into the positive
(i.e., aligned) class as we care more about cases which are mis-aligned along
an attribute. We encourage future work to improve both the question
generation, classification, and answer categorization.

###  A.4. Additional Details on Data Processing

We form preference judgements by aggregating over the contextually-annotated
images from (Dutta et al., 2024). We apply simple averaging, where each
annotation is weighted equally – alternate weighting schemes could be worth
exploring in the future, as well as a breakdown by the context. We split the
data along the prompts, as there are four different images per prompt, each
annotated with human scores.

The models we consider in this work involve binary classification; as such, we
binarize all scores – for the aggregate and fine-grained preference judgments.
Thresholds are selected manually via a mix of attempting to class-balancing
and manual inspection. Future work can explore more expansive threshold
selection.

###  A.5. Additional Details on Rejection Sampling

###  A.6. Stimuli Generation

We sample images from a generative text-to-image model similar to Rombach et
al., trained on web-scale image data, using the prompts from the test set of
Dutta et al.. To that end, our stimuli are slightly out-of-distribution (in-
distribution prompts, out-of-distribution generated images).

###  A.7. Reward Model Scoring and Selection

We run two reward models (one coarse-, one fine-) over all generated prompt-
image pairs. We apply our same embedding extraction pipeline and concatenate
the text and image embeddings. We extract an aggregate reward score from each
reward model. We select a subset of 194 text-image pairs where the reward
models substantially differ in their preference judgements.

###  A.8. Human Study: SxS Evals

We conducted a side-by-side evaluation of 194 pairs of images selected through
the reward model scoring, where one image in the pair is scored highly by the
fine-grained model and the other, scored highly by the coarse-grained model.
The participants were asked to select an image that they preferred (general
preference experiment) or asked to select and image that was “more X”, where X
is one of the features used for the fine-grained model (e.g., bright, funny).
Judgments for each feature were collected in separate tasks, leading to a
total of nine tasks (eight fine-grained features and one general preference
judgment task). The participants had the option to answer “unsure”. We
recruited three participants per image pair through an internal crowdsourcing
platform. All of the questions and the sides for each question (left/right)
were randomly shuffled.

###  A.9. Visualizing the Learned Aggregators

One of the advantages of the CBM structure of our reward model is that humans
can inspect, and therefore audit, the attributes that are learned and most
contribute to the final reward preference by inspecting the linear aggregator.
We depict the linear aggregator weights for a sampling of the models in
Experiment 1 (Figure 6) and Experiment 2 (when the decision tree attributes
were included in Figure 7 and when missing 8).

Figure 6. Attribute weights learned for the prompt-aware setting; most weight
is placed on attributes scoring prompt-image alignment. Figure 7. Attribute
weights learned for the decision tree setting. The aggregator appropriately
learns the importance and direction of the attributes which make up the leaves
of the tree. Figure 8. Attribute weights learned for the decision tree
setting. The aggregator learns to focus on attributes semantically related to
those which form the decision tree (e.g., captivating versus visual
compelling).

###  A.10. Samples Scored by Fine-Grained Models

We include some images from our data pool which were scored as good or bad
along a sampling of attributes in Figure 9.

Figure 9. Images scored by the trained fine-grained reward model. Images in
the top row are those which are rated high on an attribute (noted on the y
axis), and in the bottom row, rated low by the reward model in terms of that
attribute. Note, “compelling” here is “visually compelling”.


# Inpaint Biases: A Pathway to Accurate and Unbiased Image Generation

Jiyoon Myung
MODULABS
<EMAIL_ADDRESS>Jihyeon Park
MODULABS
<EMAIL_ADDRESS>
###### Abstract

This paper examines the limitations of advanced text-to-image models in
accurately rendering unconventional concepts which are scarcely represented or
absent in their training datasets. We identify how these limitations not only
confine the creative potential of these models but also pose risks of
reinforcing stereotypes. To address these challenges, we introduce the Inpaint
Biases framework, which employs user-defined masks and inpainting techniques
to enhance the accuracy of image generation, particularly for novel or
inaccurately rendered objects. Through experimental validation, we demonstrate
how this framework significantly improves the fidelity of generated images to
the user’s intent, thereby expanding the models’ creative capabilities and
mitigating the risk of perpetuating biases. Our study contributes to the
advancement of text-to-image models as unbiased, versatile tools for creative
expression.

##  1 Introduction

The advent of sophisticated text-to-image models [4, 3, 5] has substantially
transformed the landscape of creative content generation, becoming a crucial
component in this domain. As these models become increasingly influential, it
is crucial to evaluate the content they produce because any inherent biases or
defects could have widespread implications.

This paper delves into a critical examination of text-to-image models’
difficulties in rendering unconventional concepts, like blue bananas or square
watermelons, which are barely represented or absent in their training data.
This limitation not only constricts the creative potential of the models but
also poses the risk of leading to stereotypical representations of various
demographics or societal roles. For example, consistently misrepresenting or
failing to acknowledge specific cultural or societal elements could reinforce
or exacerbate existing social and cultural inequalities. Moreover, using
images generated by these models as training data could further solidify these
biases, creating a feedback loop of biased content generation.

Addressing this, our study aims to develop methods that enable text-to-image
models to more accurately depict unusual and innovative concepts. Our
exploration involved experimenting with a range of prompts to assess how these
models handle uncommon or novel concepts. Although the models efficiently
generated these unique concepts in isolation, they struggled to maintain
accuracy when these concepts were integrated with standard objects or within
complex style prompts. This observation prompted us to devise a method that
specifically enhances the portrayal of incorrectly rendered, novel objects
through focused inpainting techniques.

To deal with this concern, we introduce the Inpaint Biases framework that
leverages user-created masks and inpainting techniques for precise corrections
within images. This process initiates with generating an image from a user’s
prompt. If the image inaccurately reflects the intended concept, such as
rendering a blue banana as yellow, the user can apply a mask to the incorrect
element, directing the model’s correction efforts specifically to this area.
Enhanced prompts, refined using large language models (LLM), are then focused
on the masked region to improve accuracy and detail, followed by an inpainting
process that melds the corrections with the original image seamlessly.

By refining and inpainting these areas, the Inpaint Biases framework
significantly narrows the gap between the generated image and the user’s
original intention, maintaining the overall aesthetic while meticulously
correcting the inaccuracies. This approach not only expands the creative
capabilities of text-to-image models but also plays a pivotal role in
mitigating the risk of perpetuating biases, steering the technology towards a
future where it can serve as an unbiased, versatile tool for artistic and
creative expression. Through this framework, we are taking a significant step
toward harnessing the full potential of text-to-image models in a manner that
is both creatively enriching and socially responsible.

##  2 Exploring Bias Towards Unusual Concepts

In our experimentation with text-to-image models, we systematically tested
various prompts to assess the models’ ability to generate images based on
uncommon concepts, such as ’blue bananas’. The results indicated that when
tasked with rendering these unique concepts in isolation, the tool generally
produced accurate and satisfactory images (Figure. 1). However, when we
introduced a combination of concepts, such as ’blue bananas and red apples on
the table,’ we observed a noticeable decline in the models’ ability to
accurately depict the less common concept (the blue banana) alongside more
conventional objects (Figure. 2).

This phenomenon suggests that when multiple concepts are included in a prompt,
the tool may prioritize accurately rendering more common or familiar objects,
consequently compromising the depiction of rarer concepts. Based on these
findings, we deduced that a targeted approach focusing on the specific
refinement of inaccurately rendered elements could enhance the overall image
quality. Consequently, we propose an inpainting method that specifically
addresses and corrects the portions of the image that are not accurately
rendered.

By applying this inpainting technique, we could selectively refine the
depiction of the ’blue bananas’ without needing to regenerate the entire
image. This approach not only improved the accuracy of the uncommon concept’s
representation but also preserved the integrity of the overall image.
Therefore, we conclude that an inpainting strategy, which allows for the
focused correction of inaccurately depicted elements, offers an effective
solution for enhancing the text-to-image models’ performance, particularly
when dealing with a mix of common and uncommon concepts within a single
prompt.

Figure 1: Illustrations of ’blue bananas’.

Figure 2: Illustrations of ’blue bananas and red apples on the table’.

##  3 Inpaint Biases Framework

Figure 3: Inpaint Biases Framework

The Inpaint Biases framework is strategically developed to improve image
generation by inpainting targeted areas. The framework marks an important
first step toward addressing biases in image generation, aiming to minimize
the discrepancy between the user’s initial intent and the resultant image.

###  3.1 Image Generation from Initial Prompt

The initial step in the Inpaint Biases framework involves generating an image
based on a user-defined initial prompt using a text-to-image tool. This prompt
is expected to encompass an atypical concept, such as ‘yellow clouds over the
mountains’, which standard image generation processes might not accurately
capture.

###  3.2 Mask Generation Based on User Input

In instances where the generated image inadequately represents the unusual
concept (yellow clouds), the user intervenes to delineate the target area for
modification. This is achieved by selecting a specific point or defining a
bounding box on the image. Subsequently, utilizing the zero-shot segmentation
capabilities of the Segment Anything Model (SAM) [1], a mask of the targeted
area is created, setting the stage for focused inpainting. To enhance this
process, users have the additional capability of selecting the target area by
painting it themselves. This intuitive interaction allows for more precise and
personalized input, ensuring that the modification aligns closely with the
user’s vision.

###  3.3 Prompt Refinement for Targeted Inpainting

The third step diverges from the broader image generation to concentrate on
the targeted concept. Here, the Large Language Model (LLM) plays a crucial
role by suggesting a refined prompt that encapsulates the essence of the
concept needing precise representation. For instance, if the user’s objective
is to depict ’yellow clouds’ accurately, the LLM might suggest a detailed
prompt like ’An imaginative illustration of vibrant yellow clouds,’
facilitating a more targeted and conceptually aligned inpainting process.

###  3.4 Inpainting Using the Refined Prompt

With the newly refined prompt provided by the LLM, the final step is to
inpaint the previously masked area, thereby aligning the unusual concept with
the user’s vision. This inpainting is executed with the intent to accurately
fill in the masked section, reflecting the detailed attributes outlined in the
refined prompt. This precision is achieved using a mask-generation strategy
presented by LAMA [6], where the latent VAE (Variational Autoencoder)
representations of the masked image are utilized for additional conditioning.

##  4 Methodology Implementation

In this section, we illustrate the application of the Inpaint Biases framework
through a series of implementation examples, showcasing how this methodology
can effectively correct inaccuracies in generated images. We selected a range
of concepts that typically challenge text-to-image models to test the
framework’s capability to refine and correct these complex visual concepts.

As an example, we provide a compelling example with Figure. 4, where the
initial image (Figure. 4(a)) was generated from the prompt ’A fantasy world
where a river is made of dark chocolate.’ Despite the creative intent, the
text-to-image model defaulted to a blue river, influenced by its training on
prevalent images of rivers in standard blue. To address this discrepancy, we
utilized a mask (Figure. 4(b)) to isolate the river area and then applied our
inpainting process with a refined prompt: ’a smooth, flowing dark chocolate
texture, rich and velvety, blending seamlessly into a whimsical, candy-themed
fantasy landscape,’ as suggested by the LLM. The result was a striking
transformation, with the river convincingly altered to resemble flowing dark
chocolate, aligning with the original creative vision (Figure. 4(c)).

Figures 5 and 6 further demonstrate the effectiveness of our framework in
rectifying biases. Figures 5 present another instance of model bias, where the
text-to-image model rendered diamonds as large and intact, contrary to the
user’s intent to depict broken diamonds. Similarly, Figure 6 presents an
initial image of a conventional cat, which did not match the user’s unique
request for a cat with a polka-dotted fur pattern. The framework effectively
adjusted both images to align more closely with the user’s specifications,
introducing broken features to the diamonds and the distinct polka-dot pattern
to the cat’s fur.

(a) Initial Image

(b) Mask Image

(c) Inpainted Image

Figure 4: Example 1: chocolate river

We quantitatively validated our framework’s effectiveness by comparing the
CLIP scores [2] of original and inpainted images, as shown in Table 1. The
increased CLIP scores for the inpainted images quantitatively confirm the
framework’s ability to enhance the correspondence between images and textual
prompts, showcasing its effectiveness in refining image outputs to better
reflect user intentions.

(a) Initial Image

(b) Mask Image

(c) Inpainted Image

Figure 5: Example 2: broken diamonds

(a) Initial Image

(b) Mask Image

(c) Inpainted Image

Figure 6: Example 3: a cat with a polka-dotted fur pattern Figure No. | Prompt | Initial Image | Inpainted Image
---|---|---|---
Figure 4 | A fantasy world where a river is made of dark chocolate | 20.441 | 20.518
Figure 5 | Diamonds broke into pieces | 22.653 | 22.824
Figure 6 | A cat with a polka-dotted fur pattern | 24.501 | 24.680
Table 1: CLIP Score Comparison

##  5 Conclusion

This research aims to address the biases present in generative AI, which
frequently produces images reflective of the predominant features in its
training data, thereby struggling with less common or unconventional concepts.
Our initial experiments showed that the accuracy with which text-to-image AI
models depict uncommon concepts decreases when these concepts are combined
with more familiar ones in a single prompt.

In response, we introduced the Inpaint Biases Framework, which employs
inpainting techinique to selectively refine and correct the parts of an image
that are not accurately represented. The process is directed by enhanced
prompts from a Large Language Model (LLM), focusing on those parts of the
image that are inaccurately represented.

Consequently, the framework not only improves the visual accuracy of AI-
generated images but also addresses a critical ethical dimension by reducing
the bias towards dominant data features. This study offers a technological
pathway to diminish the biases in generative AI, demonstrating a commitment to
ethical principles in AI development and application.

##  6 Future Research

In this paper, we examined a specific bias inherent in text-to-image models
and proposed the Inpaint Biases Framework as a preliminary solution to this
issue. While the framework marks an important first step toward addressing
biases in image generation, further research is necessary to refine and
enhance its effectiveness. Future studies could explore methods whereby the
model autonomously identifies and corrects misalignments between prompts and
generated images without user intervention. For instance, the model could
automatically detect areas with low alignment to the prompt or select the
specific aspects of a prompt to refine. Such advancements would not only
streamline the process but also potentially enhance the accuracy and relevance
of the corrections. We hope that future research will continue to advance
text-to-image models in a direction that systematically addresses and
mitigates biases, further empowering these models as unbiased tools for
creative expression.

## Acknowledgements

This research was supported by Brian Impact Foundation, a non-profit
organization dedicated to the advancement of science and technology for all.

## References

  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding, 2022.
  * Suvorov et al. [2021] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions, 2021.