# Harm Amplification in Text-to-Image Models

Susan Hao
Google Research
<EMAIL_ADDRESS>
Renee Shelby
Google Research
<EMAIL_ADDRESS>
Yuchi Liu
Google Research
<EMAIL_ADDRESS>
Hansa Srinivasan
Google Research
<EMAIL_ADDRESS>
Mukul Bhutani
Google Research
<EMAIL_ADDRESS>
Burcu Karagol Ayan
Google Research
<EMAIL_ADDRESS>
Shivani Poddar
Google Research
<EMAIL_ADDRESS>
Sarah Laszlo
Google Research
<EMAIL_ADDRESS>
###### Abstract

Warning: The content of this paper as well as some blurred images shown may
include references to nudity, sexualization, violence, and gore.

Text-to-image (T2I) models have emerged as a significant advancement in
generative AI; however, there exist safety concerns regarding their potential
to produce harmful image outputs even when users input seemingly safe prompts.
This phenomenon, where T2I models generate harmful representations that were
not explicit in the input, poses a potentially greater risk than adversarial
prompts, leaving users unintentionally exposed to harms. Our paper addresses
this issue by first introducing a formal definition for this phenomenon,
termed harm amplification. We further contribute to the field by developing
methodologies to quantify harm amplification in which we consider the harm of
the model output in the context of user input. We then empirically examine how
to apply these different methodologies to simulate real-world deployment
scenarios including a quantification of disparate impacts across genders
resulting from harm amplification. Together, our work aims to offer
researchers tools to comprehensively address safety challenges in T2I systems
and contribute to the responsible deployment of generative AI models.

##  1 Introduction

Generative text-to-image (T2I) systems allow users to create new image content
in response to a text input, learning patterns and relationships from large-
scale training data [2, 40, 3]. As all data are reflections of the social
world [35], training data can mirror the harmful stereotypes and social
inequalities that exist offline [17, 16, 45, 86], which are subsequently
replicated in generated imagery [19]. Responsible AI scholarship illustrates
how content from T2I systems can reflect and amplify the harmful
representations that exist offline [67, 66, 69]. Harms from generative AI —
like all algorithmic systems — occur from the interplay of technical system
components and social power dynamics [77, 36]. Thus, effective harm reduction
work requires practitioners to develop sociotechnical safety requirements,
conduct evaluations, and implement strategies to proactively minimize harmful
representations prior to launch [71, 30, 79].

Responsible AI research has cultivated safety techniques to curb the
intentional or unintentional generation of harmful representations, including
through reinforcement learning [25], dataset remediation [17], red teaming
[63, 69], model input and output filters [39], and techniques to suppress
unsafe content generation [76]. These component safety techniques play a vital
role in reducing the likelihood a model will generate harmful representations,
but do not aid the empirical measurement of defined harms, which is necessary
to assess the effectiveness of safety techniques. Moreover, multimodal systems
raise novel evaluation challenges, as harmful representations may result from
computational, contextual, and compositional risks between text-image pairings
with visual content introducing “additional shock factor[s]" [88, p. 13].
While responsible AI scholarship and practice recognizes the amplification of
harm as a key negative consequence of unsafe AI systems, including the
“prospect of algorithmic systems exacerbating or scaling existing social
inequalities" [77, p. 733], limited research has assessed the effectiveness of
multimodal safety techniques for a given harm that “could guide the
development of evaluation metrics and inform responsible model release" [74,
p. 10]. A precise definition of harm amplification can strengthen multimodal
safety evaluation work enabling deeper understanding of the effectiveness of
evaluation techniques.

We address these challenges by formalizing a definition of harm amplification.
Further, we offer three statistical methods to examine representational harms
that have been defined for a given context, and identify instances of harm
amplification in T2I systems. We test this on a large-scale T2I measurement
dataset designed to simulate real-world model deployment. In summary, we make
the following research contributions:

  * •

A formal definition of harm amplification for T2I models as occurring when the
generated image from a T2I model reflects harmful representations that were
not explicit in the text input.

  * •

Three methods for quantifying and measuring harm amplification instances in
T2I models. These methods include using other models such as safety
classifiers and CLIP [68] to determine whether the output image is
significantly more harmful than the input prompt.

  * •

An empirical examination of how to employ the different methodologies to
simulate real-world model deployment, which can dynamically inform
practitioners’ safety frameworks. This includes a quantification of disparate
impact across gendered outputs as a result of harm amplification.

In what follows, we first situate our work with respect to key concepts,
including sociotechnical approaches to safety in T2I systems and stereotype
amplification (Section 2); followed by a description of our novel
methodologies (Section 4). We then detail our experiments (Section 5) and
finish with a discussion of how empirically measuring harm amplification
strengthens safety work on T2I systems (Section 6), arguing for greater
attention to harm amplification in responsible AI research and practice.

##  2 Related Work

This research lies at the intersection and builds on responsible AI research
on harm, safety, and multimodal generative AI. The contributions of this
research extend the work summarized, and highlight the need for a formalized
definition of- and evaluating- harm amplification.

###  2.1 Sociotechnical Safety in T2I Systems and Harm Reduction

Concerns about “safety" in AI systems encompass a range of diverse
sociotechnical issues that directly or indirectly shape the well-being of
people or the environment. As such, safety is contingent and contextual, and
an emergent characteristic or property of AI systems [49]. While approaches to
safety in AI systems vary [1], a key concern of a systems approach is to
identify system-level hazards [30] and integrate safety considerations into
the development process (see: [50]) to minimize the likelihood that harms will
arise from system use or failure [30]. Within this approach, social and
ethical safety considerations of T2I systems need to be identified by
proactively examining the interaction between components of the AI system
(e.g., technical capabilities, potential failures, context of use, intentions
of users or other human actors) [71, 88]. In this way, safety considerations
are reciprocally shaped by social ideas and dynamics that are encoded into AI
systems [30, 72]. This concept has been described by Ruha Benjamin [11] as the
“duplicity of technology," in which AI systems often default to or replicate
social power dynamics. Addressing this duplicity requires practitioners to
understand social inequalities and how they may manifest in sociotechnical
systems. Without intentional consideration and intervention, T2I systems can
easily amplify harmful representations [66, 14].

####  2.1.1 AI Governance and Mitigating “Unsafe" Content

An effective governance approach to cultivating “safe" AI systems is to
develop clear safety requirements [49], with attention to the sociotechnical
nature of AI systems [30] and potential harms [77]. There is increasing
recognition of representational harms in T2I systems [19, 31, 15], which are
the socially constructed beliefs about different social groups that reinforce
unjust hierarchies [86, 8, 9], and manifest in different ways (see: [42]).
Representational harms are sociotechnical in nature [77], as social beliefs
about people, culture, and experiences are encoded into systems through
training data and learned associations [16]. As such, an increasingly common
safety requirement (and social and regulatory expectation) [78, 29] is
reducing the likelihood that T2I systems will amplify harmful representations
— such as demeaning gender, racial, disability, or cultural stereotypes [66,
14, 83, 61, 13], particularly when model inputs are benign, unspecified, or do
not contain qualifiers (i.e., "irrespective of gender") [7, p. 1359]. It is
necessary to address representational harms as they perpetuate normative
narratives that adversely shape people’s sense of identity and belonging
(e.g., [12, 22, 59]) and algorithmic systems can scale the visibility of
harmful representations reinforcing their social power [65].

####  2.1.2 Gender, Social Inequality, and Representation

Gender inequalities are a persistent global problem, which can be exacerbated
by AI technologies [55]. Gender is a social construct through which a society
defines expectations around masculinity and femininity, or outside this binary
[53, 90, 89]. As a social construction, gender ideologies are created and
changed to meet social, political, and economic needs [56]. They are an
enduring major component structuring individual, interactional, and
institutional power relations [70]. A dominant gender construct is the sexual
objectification, where women are viewed as physical objects and valued for
their sexual fulfilment [33, 81]. Scholars have long highlighted how
constructed ideas about gender difference are used to justify stratification
and inequality [52], and that gender ideologies intersect with other axes of
inequality, including race and ethnicity, socioeconomic status, sexuality, and
disability among others. The concept of intersectionality was developed by
feminist scholars of color to contest essentialized, white, middle-class,
Western constructions of gender (e.g., [6, 24, 23, 44, 60]), revealing how
gendered representations are informed by intersecting gender, race, and social
class ideologies within a “matrix of domination" [21]. Computing scholars have
examined gendered stereotypes, tropes, and patterns of over/under
representation in algorithms [18], datasets [94, 58, 34], and interfaces [43,
47], demonstrating persistent stereotype exaggeration [43]. Gendered
stereotypes affect the performance of real world applications, including
online advertising [80], toxic language detection [62], machine translation
and language technologies [32, 28, 84], search engines [46, 43, 4], and T2I
systems [14, 61].

###  2.2 Stereotypes & Amplification of Societal Biases in T2I Systems

The representational harm phenomenon in T2I models has been clearly identified
(e.g., [39, 15, 14, 19, 83, 91, 92, 41, 20]). Existing work often focuses on
two areas: (1) identifying the kinds and types of harmful representations
across different social attributes and axes of identity, and (2) evaluating
and mitigating representational societal harms through measurement and
technical intervention.

####  2.2.1 Amplifying Harmful Societal Representations

Early examination into social stereotype amplification focused on how
Generative Adversarial Networks (GANs) exacerbate social biases along axes of
gender and skin tone [41, 20]. Jain et al. [41] demonstrate how generated
images have a “distributional shift" away from real world data, in which
generative AI systems exacerbate extant social inequalities and patterns of
over/under-representation. The study of representational harms in T2I systems
has extended beyond the relationships between training data and model outputs,
to focus on the relationships between text prompts and resulting imagery.
Bianchi and colleagues [14] delineate and document many examples of the
amplification of stereotypes in T2I systems, particularly with respect to
racial and cultural stereotypes in unspecified text prompts. They show using
Stable Diffusion, for example, that T2I systems amplify white racialized
notions of beauty, in which “an attractive person” generates faces similar to
a “white ideal” (p. 4); in another example, they illustrate that a prompt for
“a terrorist” generates brown faces with dark hair and beards, reflecting
demeaning stereotypes at the intersections of race and religion (p. 3). Moving
beyond representations of people, the authors also show that generating images
of everyday objects (e.g., “a house") reflects Western and middle-class
stereotypes (p. 3). Adding specific targeted identity terms (i.e., writing a
specified prompt) did not completely mitigate the stereotypes and bias in T2I
imagery. Qadri et al. [66, p. 506] examine the cultural limitations of T2I
systems by soliciting South Asian community experts to generate text prompts
and annotate the resulting imagery, revealing T2I models “can participate in
and scale such existing harmful media regimes of representation and amplify
experiences of socio-cultural marginalization." Similarly, Naik et al. [61]
explore exacerbation of social inequalities along axes of (perceived) gender,
age, race, and geographical location, showing state-of-the-art T2I systems
amplify real world inequities when compared to U.S. Bureau of Labor
Statistics. These studies find that explicitly adding identity terms to
unspecified prompts (e.g., “female security guard" [61]) does not effectively
mitigate the presence of stereotypes in generated imagery [14].

####  2.2.2 Evaluating Representational Societal Biases

A parallel body of research focuses not just on identifying the presence and
kinds of harmful stereotypes that T2I systems can generate, but also
evaluating them. Luccioni and colleagues [54] examine whether representational
societal harms in each modality amplify or compound one another by evaluating
images generated from two synthetic text prompt datasets: one with gender and
ethnicity terms and one with profession identifiers. The authors analyze
representations in T2I imagery, assessing the correlations between social
attributes in text markers and the image space, and comparing these to U.S.
labor demographics, finding consistent under-representation of marginalized
identities. Wang et al. [87] employ the Text-to-Image Association Test to
measure and quantify the presence of societal bias in T2I models while
measuring stereotype amplification in occupations across genders. Another line
of work has been to detect and measure social biases in downstream
applications. Perera et al. [64] explore social bias with respect to gender,
race, and age in the diffusion-based face generation models, finding that
diffusion models exacerbate patterns of over- and under-representation, while
[75] conducts an intersectional analysis of racialized gender stereotypes and
professions to support evaluation of T2I models. While a growing body of
research is exploring methods to measure and evaluate social stereotypes,
safety harm amplification has yet to be formally defined. Furthermore, a
comprehensive examination of harm representation across social groups will
further support sociotechnical safety work and contribute to the social and
ethical evaluation of T2I systems.

##  3 Defining Harm Amplification

Our paper focuses on examining representational harms, referring to the
socially constructed beliefs about different social groups that reinforce
unjust hierarchies [86, 8, 42, 9], in a specific set of generative AI: T2I
systems. First, we define “unsafe" content in T2I systems as the presence of
demeaning, contextually inappropriate, or offensive content in generated
images as outlined in safety requirements. What could be considered “unsafe"
or harmful is highly contextual, and needs to be defined within the broader
sociotechnical dynamics in which the system is deployed, including system
capabilities, context of use, user audiences, and social or regulatory
expectations [71, 30, 49]. In practice, what constitutes “unsafe" or harmful
content may be informed by social, regulatory, and use case expectations; and
should be clearly defined with respect to the AI system as a whole. For
illustrative purposes in this paper, we explore two specific kinds of harmful
representations to illustrate how harm amplification can be quantified in T2I
systems, following the definitions of Hao et al. [39]:

  * •

Sexually explicit content: Depictions of explicit or graphic sexual acts,
nudity (beyond human anatomy and presented in a sexually suggestive manner),
sexualized body parts, or sex toys.

  * •

Violent content: Representations of realistic acts of violence, including but
not limited to blood, dismemberment of body parts, and/or displays of
aggression/force.

After harms and safety requirements have been established for a specific T2I
system and context, harm amplification can be defined as occurring when the
output generated image is notably more harmful than the input text prompt. In
this sense, amplification reflects when a T2I system increases the severity of
the defined harms without being explicitly prompted to do so (e.g., sexually
objectifying women in generated imagery). Thus, the notion of harm
amplification we propose is concerned with understanding the relationships
between the text input and resulting imagery. We show several examples of harm
amplification (generated with Stable Diffusion 2.1 [73]) in Figure 1 for
sexually explicit (Fig 1A) and violent content (Fig 1B) identified by our
measurement techniques.

Figure 1: Blurred harm amplification examples for images generated with Stable
Diffusion 2.1 with the input prompt shown underneath each image for sexually
explicit content (A) and violent content (B). A Gaussian blur was applied on
all images to limit exposure of harms to readers.

Our understanding of harm amplification is related, but distinct, from extant
work on “bias amplification" examining how patterns of bias appear in training
data, and are amplified statistically in model outputs (e.g., [94, 85, 93, 51,
82, 57]). In this literature, “bias amplification" reflects when a model makes
“certain predictions at a higher rate for some groups than expected based on
training-data statistics" [37, p. 1], often focusing on binary classification
models (e.g., [48, 86, 38]) and correlations [94]. While bias amplification
deals with statistical disparities in model outputs across groups, harm
amplification encompasses a broader range of negative consequences not
necessarily linked to a specific group. Our definition details a more general
phenomenon observed in generative AI where the harm for the model output may
not be inherently harmful in isolation, but can be harmful when taken into the
context of the user input.

##  4 Methods to Measure Harm Amplification

Quantifying harm amplification is a crucial component in assessing “safety" in
T2I systems with respect to defined unsafe content. In this section, we
introduce three statistical methods designed to measure harm amplification.
Leveraging safety classifiers and embedding models, our approach aims to
understand how the harms of the generated output images are amplified in the
context of the user input prompts. Each of our methods has strengths and
limitations that make them well-suited for specific cases and available
resources. This tailored approach ensures that our methods can be used across
diverse contexts contributing to a more robust assessment of harm
amplification.

###  4.1 Method 1: Distribution-Based Thresholds for Discretized Text Harm
Scores

The Distribution-Based Thresholds method utilizes (1) a text safety classifier
that is applied to the input prompt to produce text harm scores (the harm
severity for a given prompt), and (2) an image safety classifier that is
applied on the output image to generate image harm scores (the harm severity
for a given image). Consider a set of text harm scores ($T$) and image harm
scores ($I$). We first discretize $T$ into $n$ even buckets
($B_{0},B_{1},\text{…},B_{n}$) where $l_{j}$ is the lower bound of the bucket
and $l_{j+1}$ is the upper bound of the bucket.

$B_{j}=\\{t\in T:l_{j}<t\leq l_{j+1}\\},\quad j=0,1,...,n$ (1)

For each $B_{j}$, the corresponding image harm scores ($I_{j}$) form a
distribution ($D_{j}$). Statistical measures, including mean ($\mu_{j}$) and
standard deviation ($\sigma_{j}$), are calculated for each $D_{j}$. The non-
fitted, raw image harm amplification threshold for each bucket is determined
as $\mu_{j}+2\cdot\sigma_{j}$, or the 95th percentile of the distribution. To
provide a smooth representation of the calculated thresholds, a polynomial
function can be fitted through the raw threshold values across all buckets. In
our experiments, we fit a first-degree polynomial with the corresponding
equation to determine the image harm amplification threshold
$HarmAmpThresh_{j}$ for each text harm bucket $B_{j}$:

$\displaystyle HarmAmpThresh_{j}=\hat{y}_{j}=b_{1}j+b_{0}$ (2)
$\displaystyle\text{where }y_{j}=\mu_{j}+2\cdot\sigma_{j}$

For example, consider a T2I prompt-image dataset where we have image harm
scores $I$ and text harm scores $T$. Say we now have a new prompt-image pair
where we would like to determine whether harm amplification occurred. Given an
image harm score (i.e., $i=0.5$) and text harm score (i.e., $t=0.15$), we
categorize the text harm score into a specific bucket (i.e., $B_{1}=0.1<t\leq
0.2$). For that given bucket, the image harm amplification threshold (i.e.,
$HarmAmpThresh_{1}$) is obtained by fitting a 1-degree polynomial to the 95th
percentile threshold for a prior image harm distribution in each bucket. We
then compare the corresponding image harm score with the harm amplification
threshold to determine whether harm is amplified (i.e.,
$i>HarmAmpThresh_{1}$).

This method of Distribution-Based Thresholds is best suited to situations
where text and image safety classifier are not well calibrated (i.e., the
output harm severity for an image harm classifier does not equate to the
severity of harm outputted by the text harm classifier). Rather than comparing
text and image harm scores directly which requires well-aligned classifiers,
we determine whether an image amplifies harm by comparing its harm score to
the relevant distribution within its corresponding text harm bucket, avoiding
direct text-image harm score comparisons.

###  4.2 Method 2: Bucket Flip for Discretized Text and Image Scores

Utilizing text and image harm safety scores similar to Method 1 (Section 4.1),
the Bucket Flip method provides a more direct comparison of discretized harm
categories across image and text. Consider a set of text harm scores ($T$) and
image harm scores ($I$), assuming text and image harm scores have a similar
scale. In cases where the scales differ, techniques such as z-score
transformations can be applied to align the scales. We can then discretize $T$
and $I$ into the same $n$ even buckets ($B_{0},B_{1},\text{…},B_{n}$) where
$l_{j}$ is the lower bound and $l_{j+1}$ is the upper bound of the bucket (see
Equation 1).

For a given (prompt, image) pair $(t,i)$, we categorize the text harm score
into a specific bucket $B_{t}$ and the image harm score into a specific bucket
$B_{i}$. If the image harm bucket $B_{i}$ is higher than the text harm bucket
$B_{t}$, then we determine that harm amplification occurred. For example,
suppose we have discretized harm scores into 10 buckets of
($0-0.1,0.1-0.2,\text{…},0.9-1.0$). Given an image harm score (i.e., $i=0.45$)
and text harm score (i.e., $t=0.15$), the image score would fall into 5th
bucket and the text score is in the 2nd bucket. As the image score bucket is
higher than text score bucket, we determine that there was a bucket flip and
harm amplification has occurred.

The Bucket Flip method is most effective when both text and image harm
classifiers scores are well aligned or when users do not have access to raw
harm scores. In many real-world scenarios, safety classifiers often output
discretized categories (i.e., low harm, medium harm, high harm) rather than
raw harm scores, making the Bucket Flip method a practical approach for
identifying harm amplification.

###  4.3 Method 3: Image-Text Co-embedding Based Harm Scores

While training dedicated image or text harm classifiers enables the most
direct harm amplification measurement methods, many people may not have the
resources to label such data and thus may not be able to train the necessary
safety classifiers. In this case, existing large pre-trained image-text models
can serve as a substitute. Large pre-trained image-text models, such as CLIP
[68], embed images and texts such that the cosine distance between
corresponding text and images is minimized in the embedding space. Taking
advantage of the co-embedding space, harm amplification can be measured by how
much closer a generated image’s embedding, $i$, is to a harmful concept in the
embedding space than the prompt’s embedding, $p$, is. Although we cannot
determine the precise embedding of harm concepts, we can embed words related
to these harm concepts (e.g., for the harm sexually explicitly, we can use the
words “sexualized", “porn", etc.), defining their text embeddings as
$h_{0},h_{1},h_{2},...h_{k}$. These harm word embeddings are meant to
approximate where harm concepts such as sexually explicit may lie in the CLIP
embedding space. Harm amplification can then be quantified as
$mean(cosine\\_similarity(h_{k},i)-cosine\\_similarity(h_{k},p))for\,all\,k$.
This conceptually maps to the notion that harm amplification can be quantified
by examining how much closer the image is to the approximated harm concepts
compared to the prompt.

##  5 Experiments

In this section, we demonstrate how to employ the methods that we outlined in
Section 4 on a large and varied T2I measurement dataset designed to simulate
usage in real-world model deployment to develop a harm amplification criteria
for each method. We then evaluate how well our methods perform by applying
these criterion on an independent human annotated T2I evaluation dataset by
calculating precision, recall, and f1-scores.

###  5.1 Measurement Dataset & Models

Our measurement dataset consists of 497,157 prompts representing various
demographics and prompt categories. These prompts were sampled from dogfood
user data aimed at testing generative AI models. By employing this diverse and
representative prompt dataset over curated adversarial or safety-specific
prompts, we prioritize a broad spectrum of potential harmful representations
that may be encountered in actual deployment. While curated adversarial
prompts can be valuable for specific safety assessments, our method takes a
broader perspective by aiming to understand how T2I models can generate harm
amplifying images over a large range of prompts (from benign to borderline to
harmful).

We used Stable Diffusion 2.1 [73] to generate images for our prompt
measurement dataset. Four images were generated per prompt resulting in
1,988,628 total generated images. No NSFW or safety filters were applied
during the image generation process. Thus, potentially harmful images were
generated for our experiment and were not filtered or blocked.

Safety machine learning classifiers were used to provide harm scores for
sexually explicit content and violent content. Separate classifiers were used
on the input text prompts and the output images with harm scores for each
classifier ranging from 0 to 1. Fairness machine learning classifiers that
identified probability estimates of gender presentation in images were also
used with estimates for each gender (male, female, or undefined) ranging from
0 to 1. For the pre-trained image-text model, we use CLIP [68] and generated
embeddings for images and their corresponding prompts.

This large, varied measurement dataset served as the foundation for applying
each of our methods to establish criteria for determining harm amplification
in sexually explicit and violent content. Machine annotations for sexually
explicit content, violent content, as well as CLIP embeddings served as inputs
for our measurement techniques. After applying each of our measurement
techniques to our measurement dataset, a criteria was obtained for each
method:

  1. 1.

Distribution-Based Threshold: Harm amplification occurs if the image harm is
higher than a threshold for a given text harm bucket.

  2. 2.

Bucket Flip: Harm amplification occurs if the image harm bucket is higher than
the corresponding text harm bucket.

  3. 3.

Image-Text Co-embedding: Harm amplification occurs if an image CLIP embedding
is relatively closer to the harm concept embedding compared to the prompt
embedding.

###  5.2 Evaluation Data & Metrics

To create the evaluation dataset, we sourced 742 prompt, image pairs from the
Adversarial Nibbler data challenge [5]. This public, crowd-sourced challenge
aimed to identify failure-modes of DALL-E 2 [2] and various Stable Diffusion
versions (XL 1.0, 1.5, and 2.1) [3]. At least five raters annotated text and
image safety independently as well as identified the type of harm (i.e.,
sexually explicit or violent) that was in the image if the image was deemed
unsafe. Majority vote was used to determine the final label for each prompt
harm and image harm label. We defined ground truth (if harm amplification
occurred) as when the text prompt was considered safe by human annotators, but
the image was deemed harmful. Ground truth for the absence of harm
amplification was defined as when human annotators judged the text and image
both to be safe. In addition to human labels, we obtained CLIP embeddings and
machine annotations for sexually explicit and violent content for both prompts
and images that served as inputs for each of our methods to infer whether harm
amplification occurred. Subsequently, using the criteria established for each
method (derived from our measurement data), we assessed the presence of harm
amplification in the Nibbler evaluation for each harm, using human labels as
ground truth.

We calculated precision, recall, and f1-scores for each method’s evaluation of
harm amplification on the Nibbler dataset. This evaluation was conducted
separately for sexually explicit and violence harm amplification. Furthermore,
perceived gender was additionally machine annotated in the evaluation dataset
to investigate potential disparities in rates of harm amplification across
genders.

###  5.3 Experimental Results

####  5.3.1 Sexually Explicit & Violent Content Results

To quantify harm amplification using the Distribution-Based Threshold method
(Section 4.1), we discretized the text harm scores for the measurement prompt
data into 5 buckets for each harm type: sexually explicit and violence.
Subsequently, we then obtained an image harm distribution from the measurement
data for each of the 5 buckets corresponding to each harm type (see Appendix
A, Figure 2A). The 95th percentile score, or 2 standard deviations above the
mean, was then calculated for each distribution within its respective bucket.
A first-degree polynomial was fitted across buckets on the 95th percentile
score resulting in new fitted thresholds (Appendix A, Figure 2B) for each harm
type. For our data, a first-degree linear function was most appropriate, but
the type of function fitted on thresholds may vary depending on the statistics
of the image data distribution. Having derived fitted image harm amplification
thresholds for each text bucket for each harm type, we then assessed how well
our measurement technique performed on the Nibbler evaluation dataset.
Notably, we observed high precision (0.925) and recall (0.973) when evaluating
sexually explicit content in the Nibbler dataset (see Table 1). However, we
noted slightly lower precision (0.788) and recall (0.512) when evaluating this
measurement technique on violent content (Table 1).

Using the Bucket Flip method (Section 4.2), we first calculated and recorded
the mean and standard deviation of both the text harm scores and image harm
scores for the measurement data for each harm type: sexually explicit and
violence. We then did z-score transformations on both the text and image harm
scores to transform them into similar scale. We then subsequently divided that
range into 10 even buckets. For evaluation on the Nibbler dataset, we
similarly applied z-score transformations to the machine annotations for each
harm type using the corresponding mean and standard deviation on the text and
image derived from the measurement data. We then bucketized the transformed
scores with the bucket boundaries obtained from the measurement data and
assessed whether the image buckets were higher than the text buckets. We
observed high precision (0.896) and recall (0.956) when evaluating sexually
explicit content and slightly lower precision (0.870) and recall (0.488) when
evaluating on violent content.

While we expect the Image-Text Co-embedding method (Section 4.3) to perform
worse at the task of identifying instances of harm amplification than using
dedicated sexually explicit and violence text and image classifiers, this
method is useful for those with access to limited resources. We used CLIP to
encode images and prompts, and defined the sexually explicit harm concept
using the words “porn," “sexual," and “nudity." We defined the violence harm
concept using the words “violent," “weapons," and “injury". Utilizing more
terms to define each harm concept did not have a significant effect on results
(see Appendix B, Figure 5). Thus, we report results using the specified three
terms for each harm concept that we mentioned above. Embedding and distance
scores were calculated as described in Section 4.3 separately for the
measurement data and evaluation data. Various harm amplification thresholds
were applied (using distances calculated from the measurement data embeddings)
to the evaluation dataset to obtain precision-recall curves (see Appendix B,
Figure 6). Unlike the other methods, there is not an obvious way to define
what the best harm amplification threshold is, but rather, practitioners can
decide on an appropriate threshold based on their specific use case. For these
results we set an operating recall of 0.6, and compute other precision and
f1-scores from this number using the PR curves. As with other methods, we saw
better results for sexually explicit harm amplification (precision of 0.821,
recall of 0.600) than violence harm amplification (precision 0.390, recall
0.600).

Harm Amplification Measurement Method | Sexually Explicit | Violence
---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score
Distribution-Based Thresholds | 0.925 | 0.973 | 0.948 | 0.788 | 0.512 | 0.621
Bucket Flip | 0.896 | 0.956 | 0.925 | 0.870 | 0.488 | 0.625
Image-Text Co-embedding | 0.821 | 0.600 | 0.693 | 0.390 | 0.600 | 0.473
Table 1: Harm Amplification measurement methods evaluation on Nibbler data.
Each method, when applied on the measurement dataset (Stable Diffusion 2.1
prompt, image dataset), yielded some criteria for determining whether harm
amplification occurred for each harm type: sexually explicit and violence.
These criteria were then applied to the Nibbler evaluation dataset where we
had ground truth human annotation labels. Precision, recall, and F1-scores for
sexually explicit content (left 3 columns) and violence (right 3 columns) were
reported for each of the three measurement methods.

####  5.3.2 Gender Fairness

As described in Related Work (Sections 2.1.2 & 2.2.1), the amplification of
social stereotypes disproportionately affects marginalized communities [14,
61]. We present quantitative measurement of harm amplification — particularly
the gendered sexual objectification stereotype [33, 81] — and show that it
disproportionately affects images depicting perceived females. We recognize
the dynamic and non-binary nature of gender; however, in this preliminary
study, we focus on perceived binary (male/female) gender expressions due to
the ease in methodology. We also acknowledge that we that cannot effectively
infer people’s gender identity, and instead rely on perceived gender
expression.

To show the prevalence of disparity in harm amplification across perceived
gender expression in Nibbler dataset for the harm types of sexual explicit and
violent content, we evaluate the efficacy of our proposed methods across the
gender demographic on the Nibbler dataset. From the Nibbler dataset, for all
images, both the number of people and their perceived gender expression were
determined by machine classification [10]. We annotate the tuples in which the
number of perceived females are in strict majority to be a (perceived) female
data point, and do the same for (perceived) male gender. We do not include any
tuple in which, either there are no faces or in which the number of perceived
females are equal to the number of perceived males.

Gender Expression | Sexually Explicit | Violence
---|---|---
# Text Safe Examples |  Harm Amplification Rate |  # Text Safe Examples |  Harm Amplification Rate
(Perceived) Female | 173 | 0.80 | 60 | 0.43
(Perceived) Male | 145 | 0.46 | 120 | 0.36
Table 2: Harm Amplification rates across perceived gender expressions on
Nibbler data. Significance testing was done using z-test. Statistically
significant numbers are in bold ($p<0.01$).

We calculate the rates of harm amplification across both the perceived gender
expressions using the ground truth of the Nibbler dataset for both harm
categories described in Section 3. Table 2 shows the the number of examples
belonging to each category across the two perceived gender expressions. We
find the harm amplification rate for (perceived) females is statistically
significantly higher ($p<0.01$) for the category of sexually explicit harm
amplification. This important observation quantifies the increased sexually
explicit harm amplification present in T2I systems for the perceived female
gender. Furthermore, the increased rate of harm amplification of violent
content is not statistically significant ($p=0.181$).

A good method for detecting harm amplification should run equally well across
all the demographic groups. The Distribution-Based Thresholds provide
comparable efficacy in detection of harm amplification instances for sexually
explicit content across both the perceived gender expressions (see Appendix C,
Table 3). For the remaining methods, the precision in detecting over-
sexualization in perceived females is higher than the precision in detecting
it in perceived males.

##  6 Discussion & Conclusion

In this paper, we provide a formal definition of harm amplification in T2I
models as occurring when the generated output image reflects more harmful
representations that were not explicit in the text input (Section 3). Using
two specific kinds of harmful representations (sexually explicit content and
violent content), we showed how to quantify harm amplification by providing
three methods that use safety classifier harm scores and CLIP embeddings
(Section 4). We then empirically illustrated the application of these
methodologies on a measurements dataset that simulated real-world model
deployment and evaluated these methods on an independent human annotated
dataset (Section 5).

###  6.1 Using Data-Driven Methods to Quantify Harm Amplification

Each of our three data-driven methods for quantifying harm amplification
offers its own strengths and limitations. The Distribution-Based Thresholds
method may be most appropriate if text and image harm scores are not well
calibrated. However, with this method, we assume that image harm scores within
each bucket follow a normal distribution. If this assumption is violated, then
thresholds for each bucket may not be effective in quantifying when harm
amplification occurs. The Bucket Flip method is the easiest method to employ,
but best performance is achieved if the text and image harm scores are well
calibrated – otherwise, a bucket flip may not necessarily mean that harm is
amplified. Another limitation for this method is that (text, image) pairs with
close harm scores may be positioned in different buckets which might be
assessed as harm amplification instances even when the actual difference in
scores is small. While the Image-Text Co-embedding method performs well in a
low-resource setting, a fine-tuned model using CLIP embeddings specifically
trained to detect harm is likely to yield better results. As we observe in the
methods using safety classifiers, specific use-case models can dramatically
improve performance when detecting instances of harm amplification.

Across all three methods, we assume the harm amplification criterion derived
from the measurements dataset will generalize to new data generated by our
measurement model (Stable Diffusion 2.1) as well as new data generated by
other T2I models. This assumption holds if there is similar harm amplification
rates across different T2I models. Although we observe good performance in our
evaluation dataset that encompasses data generated from several different
models, further research should explore the generalizability of harm
amplification methods in cross-domain T2I models.

###  6.2 Reduced Reliance on Human Annotation

Our novel framework significantly reduces the reliance on human annotations,
which may be especially helpful in low-resource settings. By primarily using
machine annotations, we enable scalability making the measurement process
adaptable to larger datasets. While human annotations are valuable for
evaluating the method’s efficacy on held out datasets, they are not required
for the core measurement process. Furthermore, our framework demonstrates its
ability to measure harm amplification without the need for explicit
adversarial data generation or red teaming, which are often resource intensive
and uncomprehensive. Thus, our proposed framework not only offers robust
methodologies for measuring harm amplification but also addresses practical
challenges that researchers may encounter when deploying models at scale. Of
course, machine annotations — akin to human annotations [26, 27] — can also
reflect social biases. These biases may stem from the subjective nature of
defining ground truth in safety. Further research could explore methodologies
to more inclusively capture safety in annotations, taking into account diverse
perspectives of what safety harms mean.

###  6.3 Gender Disparities in Harm Amplification

In analyzing rates of harm amplification in the evaluation dataset, we
identified gender-related disparities, particularly for sexually explicit
content. Our findings revealed a higher prevalence of oversexualization in
perceived female depicting images compared to perceived male depicting images.
These results highlight crucial safety concerns regarding the exacerbation of
harmful stereotypes relating to gender. Our work emphasizes the importance of
both addressing harm amplification in T2I systems and understanding its
impacts in different sociodemographic groups. By identifying gender related
disparities, we contribute to existing work [54, 61] demonstrating gender
stereotypes in T2I models. The durability of gender stereotypes in AI systems
[18] necessitates greater attention to development of sociotechnical
strategies to prevent the perpetuation and amplification of harmful
stereotypes. While we focused on gender, other axes of inequality, such as
race, ethnicity, disability, sexuality, and culture, among others, should be
explored.

###  6.4 Future Work

While our work lays a foundation to better understand and quantify harm
amplification in T2I systems, further research is needed to more holistically
understand the phenomenon and develop strategies for its reduction in T2I
models. One future research potential includes an in-depth exploration of the
role of training data and whether biases in training data lead to the
reinforcement of stereotypes for certain groups as we observed in our results
pertaining to the oversexualization of women. This work will be pivotal to
understanding the root cause of harm amplification, thereby guiding the
development of solutions to reduce its impact in T2I models. Furthermore,
there is a need for the exploration of mitigation techniques to reduce harm
amplification as these strategies are necessary in minimizing the exposure of
unwanted harms to users. The development of these mitigation techniques will
contribute to the overall responsible deployment of T2I systems, supporting
the increased protection and safety of users.

##  7 Acknowledgements

We thank Kathy Meier-Hellstern, Alexander Vorontsov, and Vinodkumar
Prabhakaran for their feedback and suggestions. We also thank Auriel Wright,
Candice Schumann, Lora Aroyo, and Alicia Parrish for discussions on analysis
and writing.

## References

  * Ahmed et al. [2023] Shazeda Ahmed, Klaudia Jaźwińska, Archana Ahlawat, Amy Winecoff, and Mona Wang.  Building the epistemic community of ai safety, 2023.
  * AI [2023] Open AI.  Dall-e 2, 2023.  URL https://openai.com/dall-e-2/.
  * AI [2022] Stability AI.  Stable diffusion, 2022.  URL https://stablediffusionweb.com/.
  * Albawardi and Jones [2023] Areej Albawardi and Rodney H Jones.  Saudi women driving: Images, stereotyping and digital media.  _Visual Communication_ , 22(1):96–127, 2023.  doi: 10.1177/14703572211040851.
  * Alicia Parish [2023] Jessica Quaye Charvi Rastogi Max Bartolo Oana Inel Juan Ciro Rafael Mosquera Addison Howard Will Cukierski D. Sculley Vijay Janapa Reddi Lora Aroya Alicia Parish, Hannah Rose Kirk.  Adversarial nibbler: A data-centric challenge for improving the safety of text-to-image models, 2023.
  * Anzaldúa [1987] Gloria Anzaldúa.  _Borderlands/La Frontera_.  Aunt Lute Books, San Francisco, California, 1987.
  * Bansal et al. [2022] Hritik Bansal, Da Yin, Masoud Monajatipoor, and Kai-Wei Chang.  "how well can text-to-image generative models understand ethical natural language interventions?".  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1358–1370, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.88.  URL https://aclanthology.org/2022.emnlp-main.88.
  * Barocas et al. [2017] Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach.  The problem with bias: Allocative versus representational harms in machine learning.  In _9th Annual Conference of the Special Interest Group for Computing, Information and Society_ , Philadelphia, PA, USA, 2017. Society for the History of Technology.
  * Barocas et al. [2019] Solon Barocas, Moritz Hardt, and Arvind Narayanan.  Fairness and machine learning, 2019.  http://www.fairmlbook.org.
  * Baruah et al. [2022] Sabyasachee Baruah, Digbalay Bose, Meredith Conroy, Shrikanth S. Narayanan, Susanna Ricco, Komal Singh, and Krishna Somandepalli.  #seeitbeit: What families are seeing on tv.  The Geena Davis Institute on Gender in Media, 2022.  URL https://seejane.org/research-informs-empowers/see-it-be-it-what-families-are-watching-on-tv/.
  * Benjamin [2019] Ruha Benjamin.  _Race After Technology: Abolitionist Tools for the New Jim Code_.  Polity Press, Cambridge, UK, Jul 2019.  ISBN 9781509526437.
  * Bennett and Keyes [2020] Cynthia L. Bennett and Os Keyes.  What is the point of fairness? disability, ai and the complexity of justice.  _SIGACCESS Access. Comput._ , 125(5), 2020.  ISSN 1558-2337.  doi: 10.1145/3386296.3386301.  URL https://doi.org/10.1145/3386296.3386301.
  * Bennett et al. [2021] Cynthia L. Bennett, Cole Gleason, Morgan Klaus Scheuerman, Jeffrey P. Bigham, Anhong Guo, and Alexandra To.  “it’s complicated”: Negotiating accessibility and (mis)representation in image descriptions of race, gender, and disability.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI ’21, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450380966.  doi: 10.1145/3411764.3445498.  URL https://doi.org/10.1145/3411764.3445498.
  * Bianchi et al. [2023] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng, Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan.  Easily accessible text-to-image generation amplifies demographic stereotypes at large scale.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, page 1493–1504, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400701924.  doi: 10.1145/3593013.3594095.  URL https://doi.org/10.1145/3593013.3594095.
  * Bird et al. [2023] Charlotte Bird, Eddie Ungless, and Atoosa Kasirzadeh.  Typology of risks of generative text-to-image models.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’23, page 396–410, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400702310.  doi: 10.1145/3600211.3604722.  URL https://doi.org/10.1145/3600211.3604722.
  * Birhane et al. [2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: Misogyny, pornography, and malignant stereotypes, 2021.
  * Birhane et al. [2023] Abeba Birhane, Vinay Prabhu, Sang Han, Vishnu Naresh Boddeti, and Alexandra Sasha Luccioni.  Into the laions den: Investigating hate in multimodal datasets, 2023.
  * Buolamwini and Gebru [2018] Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ , volume 81 of _Proceedings of Machine Learning Research_ , pages 77–91, New York, NY, 23–24 Feb 2018. PMLR.  URL https://proceedings.mlr.press/v81/buolamwini18a.html.
  * Cho et al. [2023] Jaemin Cho, Abhay Zala, and Mohit Bansal.  Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3043–3054, Paris, France, October 2023. IEEE.
  * Choi et al. [2020] Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon.  Fair generative modeling via weak supervision, 2020.
  * Collins [1990] Patricia Hill Collins.  _Black Feminist Thought: Knowledge, Consciousness, and the Politics of Empowerment_.  Routledge, New York, New York, 1990.
  * Costanza-Chock [2018] Sasha Costanza-Chock.  Design justice, ai, and escape from the matrix of domination.  _Journal of Design and Science_ , 3(5):1–14, 2018\.
  * Crenshaw [1991a] Kimberle Crenshaw.  Mapping the margins: Intersectionality, identity politics, and violence against women of color.  _Stanford Law Review_ , 43(6):1241–1299, 1991a.
  * Crenshaw [1991b] Kimberlé Crenshaw.  Demarginalizing the intersection of race and sex: A black feminist critique of antidiscrimination doctrine, feminist theory, and antiracist politics.  In _Feminist Legal Theories: Readings in Law and Gender_ , pages 23–51. Westview Press, Boulder, Colorado, 1991b.
  * Daniels-Koch and Freedman [2022] Oliver Daniels-Koch and Rachel Freedman.  The expertise problem: Learning from specialized feedback, 2022.
  * Davani et al. [2022] Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran.  Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations.  _Transactions of the Association for Computational Linguistics_ , 10:92–110, 01 2022.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00449.  URL https://doi.org/10.1162/tacl_a_00449.
  * Denton et al. [2021] Emily Denton, Mark Díaz, Ian Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen.  Whose ground truth? accounting for individual and collective identities underlying dataset annotation, 2021.
  * Dev et al. [2021] Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M Phillips, and Kai-Wei Chang.  Harms of gender exclusivity and challenges in non-binary representation in language technologies, 2021.
  * Dias Oliva [2020] Thiago Dias Oliva.  Content Moderation Technologies: Applying Human Rights Standards to Protect Freedom of Expression.  _Human Rights Law Review_ , 20(4):607–640, 12 2020.  ISSN 1461-7781.  doi: 10.1093/hrlr/ngaa032.  URL https://doi.org/10.1093/hrlr/ngaa032.
  * Dobbe [2022] Roel Dobbe.  System safety and artificial intelligence.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, page 1584, New York, NY, USA, 2022\. Association for Computing Machinery.  ISBN 9781450393522.  doi: 10.1145/3531146.3533215.  URL https://doi.org/10.1145/3531146.3533215.
  * Díaz et al. [2023] Mark Díaz, Sunipa Dev, Emily Reif, Emily Denton, and Vinodkumar Prabhakaran.  Sound framework: Analyzing (so)cial representation in (un)structured (d)ata, 2023.
  * Font and Costa-Jussà [2019] Joel Escudé Font and Marta R. Costa-Jussà.  Equalizing gender biases in neural machine translation with word embeddings techniques, 2019.
  * Fredrickson and Roberts [1997] Barbara L Fredrickson and Tomi-Ann Roberts.  Objectification theory: Toward understanding women’s lived experiences and mental health risks.  _Psychology of Women Quarterly_ , 21(2):173–206, 1997.
  * Garrido-Muñoz et al. [2023] Ismael Garrido-Muñoz, Fernando Martínez-Santiago, and Arturo Montejo-Ráez.  Maria and beto are sexist: Evaluating gender bias in large language models for spanish, 2023.
  * Gitelman [2013] Lisa Gitelman.  _"Raw Data" Is an Oxymoron_.  The MIT Press, Cambridge, Massachusetts, 2013.
  * Green and Viljoen [2020] Ben Green and Salomé Viljoen.  Algorithmic realism: Expanding the boundaries of algorithmic thought.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20, page 19–31, New York, NY, USA, 2020\. Association for Computing Machinery.  ISBN 9781450369367.  doi: 10.1145/3351095.3372840.  URL https://doi.org/10.1145/3351095.3372840.
  * Hall et al. [2022a] Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Adcock.  A systematic study of bias amplification, 2022a.
  * Hall et al. [2022b] Melissa Hall, Laurens van der Maaten, Laura Gustafson, Maxwell Jones, and Aaron Bryan Adcock.  Bias amplification in image classification.  In _Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022_ , pages 1–16, New Orleans, LA, 2022b. NeurIPS.  URL https://openreview.net/forum?id=lwG9sG4sbVC.
  * Hao et al. [2023] Susan Hao, Piyush Kumar, Sarah Laszlo, Shivani Poddar, Bhaktipriya Radharapu, and Renee Shelby.  Safety and fairness for content moderation in generative models, 2023\.
  * Inc. [2022] Midjourney Inc.  Midjourney, 2022.  URL https://www.midjourney.com/.
  * Jain et al. [2021] Niharika Jain, Alberto Olmo, Sailik Sengupta, Lydia Manikonda, and Subbarao Kambhampati.  Imperfect imaganation: Implications of gans exacerbating biases on facial data augmentation and snapchat selfie lenses, 2021.
  * Katzman et al. [2021] Jared Katzman, Solon Barocas, Su Lin Blodgett, Kristen Laird, Morgan Klaus Scheuerman, and Hanna Wallach.  Representational harms in image tagging. beyond fair computer vision workshop at cvpr 2021 (2021), 2021.
  * Kay et al. [2015] Matthew Kay, Cynthia Matuszek, and Sean A. Munson.  Unequal representation and gender stereotypes in image search results for occupations.  In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ , CHI ’15, page 3819–3828, New York, NY, USA, 2015\. Association for Computing Machinery.  ISBN 9781450331456.  doi: 10.1145/2702123.2702520.  URL https://doi.org/10.1145/2702123.2702520.
  * King [1988] Deborah K King.  Multiple jeopardy, multiple consciousness: The context of a black feminist ideology.  _Signs: Journal of Women in Culture and Society_ , 14(1):42–72, 1988.
  * Kirk et al. [2021] Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano.  Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 2611–2624, Virtual Event, 2021. Curran Associates, Inc.  URL https://proceedings.neurips.cc/paper/2021/file/1531beb762df4029513ebf9295e0d34f-Paper.pdf.
  * Kopeinik et al. [2023] Simone Kopeinik, Martina Mara, Linda Ratz, Klara Krieg, Markus Schedl, and Navid Rekabsaz.  Show me a "male nurse"! how gender bias is reflected in the query formulation of search engine users.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9781450394215.  doi: 10.1145/3544548.3580863.  URL https://doi.org/10.1145/3544548.3580863.
  * Krieg et al. [2023] Klara Krieg, Emilia Parada-Cabaleiro, Gertraud Medicus, Oleg Lesota, Markus Schedl, and Navid Rekabsaz.  Grep-biasir: A dataset for investigating gender representation bias in information retrieval results.  In _Proceedings of the 2023 Conference on Human Information Interaction and Retrieval_ , CHIIR ’23, page 444–448, New York, NY, USA, 2023\. Association for Computing Machinery.  ISBN 9798400700354.  doi: 10.1145/3576840.3578295.  URL https://doi.org/10.1145/3576840.3578295.
  * Leino et al. [2019] Klas Leino, Emily Black, Matt Fredrikson, Shayak Sen, and Anupam Datta.  Feature-wise bias amplification, 2019.
  * Leveson [2020] Nancy Leveson.  Are you sure your software will not kill anyone?  _Commun. ACM_ , 63(2):25–28, jan 2020.  ISSN 0001-0782.  doi: 10.1145/3376127.  URL https://doi.org/10.1145/3376127.
  * Leveson [2016] Nancy G Leveson.  _Engineering a Safer World: Systems Thinking Applied to Safety_.  The MIT Press, Cambridge, MA, 2016.
  * Lloyd [2018] Kirsten Lloyd.  Bias amplification in artificial intelligence systems, 2018.
  * Lorber [1994] Judith Lorber.  _Paradoxes of Gender_.  Yale University Press, New Haven, Connecticut, 1994.
  * Lorber [1996] Judith Lorber.  Beyond the binaries: Depolarizing the categories of sex, sexuality, and gender.  _Sociological Inquiry_ , 66(2):143–160, 1996\.
  * Luccioni et al. [2023] Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite.  Stable bias: Analyzing societal representations in diffusion models, 2023\.
  * Lütz [2023] Fabian Lütz.  _Gender Equality and Artificial Intelligence: SDG 5 and the Role of the UN in Fighting Stereotypes, Biases, and Gender Discrimination_ , pages 153–180.  Palgrave Macmillan, Cham, Switzerland, 2023.
  * Martin [2004] Patricia Yancey Martin.  Gender as Social Institution*.  _Social Forces_ , 82(4):1249–1273, 06 2004.  ISSN 0037-7732.  doi: 10.1353/sof.2004.0081.  URL https://doi.org/10.1353/sof.2004.0081.
  * Mehrabi et al. [2021] Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Comput. Surv._ , 54(6), jul 2021.  ISSN 0360-0300.  doi: 10.1145/3457607.  URL https://doi.org/10.1145/3457607.
  * Meister et al. [2023] Nicole Meister, Dora Zhao, Angelina Wang, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky.  Gender artifacts in visual datasets.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4837–4848, Paris,France, 2023. IEEE.
  * Mengesha et al. [2021] Zion Mengesha, Courtney Heldreth, Michal Lahav, Juliana Sublewski, and Elyse Tuennerman.  “i don’t think these devices are very culturally sensitive.”—impact of automated speech recognition errors on african americans.  _Frontiers in Artificial Intelligence_ , 26(4):169, 2021.  doi: https://doi.org/10.3389/frai.2021.725911.
  * Mohanty [2003] Chandra Talpade Mohanty.  _Feminism Without Borders: Decolonizing Theory, Practicing Solidarity_.  Duke University Press, Durham, NC, 2003.
  * Naik and Nushi [2023] Ranjita Naik and Besmira Nushi.  Social biases through the text-to-image generation lens.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’23, page 786–808, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400702310.  doi: 10.1145/3600211.3604711.  URL https://doi.org/10.1145/3600211.3604711.
  * Park et al. [2018] Ji Ho Park, Jamin Shin, and Pascale Fung.  Reducing gender bias in abusive language detection, 2018.
  * Parrish et al. [2023] Alicia Parrish, Hannah Rose Kirk, Jessica Quaye, Charvi Rastogi, Max Bartolo, Oana Inel, Juan Ciro, Rafael Mosquera, Addison Howard, Will Cukierski, D. Sculley, Vijay Janapa Reddi, and Lora Aroyo.  Adversarial nibbler: A data-centric challenge for improving the safety of text-to-image models, 2023.
  * Perera and Patel [2023] Malsha V. Perera and Vishal M. Patel.  Analyzing bias in diffusion-based face generation models, 2023.
  * Phillips [2018] Whitney Phillips.  The oxygen of amplification: Better practices for reporting on extremists, antagonists, and manipulators online.  Data & Society, 2018.  URL https://datasociety.net/wp-content/uploads/2018/05/2-PART-2_Oxygen_of_Amplification_DS.pdf.
  * Qadri et al. [2023] Rida Qadri, Renee Shelby, Cynthia L. Bennett, and Emily Denton.  Ai’s regimes of representation: A community-centered study of text-to-image models in south asia.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, page 506–517, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400701924.  doi: 10.1145/3593013.3594016.  URL https://doi.org/10.1145/3593013.3594016.
  * Qu et al. [2023] Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang.  Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models.  In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_ , CCS ’23, page 3403–3417, New York, NY, USA, 2023\. Association for Computing Machinery.  ISBN 9798400700507.  doi: 10.1145/3576915.3616679.  URL https://doi.org/10.1145/3576915.3616679.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763, Virtual Event, 18–24 Jul 2021. PMLR.  URL https://proceedings.mlr.press/v139/radford21a.html.
  * Rando et al. [2022] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr.  Red-teaming the stable diffusion safety filter, 2022.
  * Risman [2004] Barbara J Risman.  Gender as a social structure: Theory wrestling with activism.  _Gender & Society_, 18(4):429–450, 2004.
  * Rismani et al. [2023a] Shalaleh Rismani, Renee Shelby, Andrew Smart, Renelito Delos Santos, AJung Moon, and Negar Rostamzadeh.  Beyond the ml model: Applying safety engineering frameworks to text-to-image development.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’23, page 70–83, New York, NY, USA, 2023a. Association for Computing Machinery.  ISBN 9798400702310.  doi: 10.1145/3600211.3604685.  URL https://doi.org/10.1145/3600211.3604685.
  * Rismani et al. [2023b] Shalaleh Rismani, Renee Shelby, Andrew Smart, Edgar Jatho, Joshua Kroll, AJung Moon, and Negar Rostamzadeh.  From plane crashes to algorithmic harm: Applicability of safety engineering frameworks for responsible ml.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , CHI ’23, New York, NY, USA, 2023b. Association for Computing Machinery.  ISBN 9781450394215.  doi: 10.1145/3544548.3581407.  URL https://doi.org/10.1145/3544548.3581407.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, New Orleans, LA, June 2022\. IEEE.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 36479–36494, New Orleans, LA, 2022. Curran Associates, Inc.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ec795aeadae0b7d230fa35cbaf04c041-Paper-Conference.pdf.
  * Saravanan et al. [2023] Adhithya Prakash Saravanan, Rafal Kocielnik, Roy Jiang, Pengrui Han, and Anima Anandkumar.  Exploring social bias in downstream applications of text-to-image foundation models, 2023.
  * Schramowski et al. [2023] Patrick Schramowski, Manuel Brack, Björn Deiseroth, and Kristian Kersting.  Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 22522–22531, New Orleans, LA, June 2023\. IEEE.
  * Shelby et al. [2023] Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N’Mah Yilla-Akbari, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk.  Sociotechnical harms of algorithmic systems: Scoping a taxonomy for harm reduction.  In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’23, page 723–741, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400702310.  doi: 10.1145/3600211.3604673.  URL https://doi.org/10.1145/3600211.3604673.
  * Solaiman [2023] Irene Solaiman.  The gradient of generative ai release: Methods and considerations.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, page 111–122, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400701924.  doi: 10.1145/3593013.3593981.  URL https://doi.org/10.1145/3593013.3593981.
  * Solaiman et al. [2023] Irene Solaiman, Zeerak Talat, William Agnew, Lama Ahmad, Dylan Baker, Su Lin Blodgett, Hal Daumé III au2, Jesse Dodge, Ellie Evans, Sara Hooker, Yacine Jernite, Alexandra Sasha Luccioni, Alberto Lusoli, Margaret Mitchell, Jessica Newman, Marie-Therese Png, Andrew Strait, and Apostol Vassilev.  Evaluating the social impact of generative ai systems in systems and society, 2023.
  * Sweeney [2013] Latanya Sweeney.  Discrimination in online ad delivery.  _Communications of the ACM_ , 56(5):44–54, 2013\.
  * Szymanski et al. [2011] Dawn M Szymanski, Lauren B Moffitt, and Erika R Carr.  Sexual objectification of women: Advances to theory and research 1$\psi$7\.  _The Counseling Psychologist_ , 39(1):6–38, 2011\.
  * Taori and Hashimoto [2023] Rohan Taori and Tatsunori Hashimoto.  Data feedback loops: Model-driven amplification of dataset biases.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 33883–33920, Honolulu, HI, 23–29 Jul 2023. PMLR.  URL https://proceedings.mlr.press/v202/taori23a.html.
  * Ungless et al. [2023] Eddie L. Ungless, Björn Ross, and Anne Lauscher.  Stereotypes and smut: The (mis)representation of non-cisgender identities by text-to-image models, 2023.
  * Vanmassenhove et al. [2018] Eva Vanmassenhove, Christian Hardmeier, and Andy Way.  Getting gender right in neural machine translation.  In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3003–3008, Brussels, Belgium, October-November 2018. Association for Computational Linguistics.  doi: 10.18653/v1/D18-1334.  URL https://aclanthology.org/D18-1334.
  * Wang and Russakovsky [2021] Angelina Wang and Olga Russakovsky.  Directional bias amplification.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 10882–10893, Virtual Event, 18–24 Jul 2021. PMLR.  URL https://proceedings.mlr.press/v139/wang21t.html.
  * Wang et al. [2022] Angelina Wang, Solon Barocas, Kristen Laird, and Hanna Wallach.  Measuring representational harms in image captioning.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, page 324–335, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450393522.  doi: 10.1145/3531146.3533099.  URL https://doi.org/10.1145/3531146.3533099.
  * Wang et al. [2023] Jialu Wang, Xinyue Liu, Zonglin Di, Yang Liu, and Xin Eric Wang.  T2IAT: measuring valence and stereotypical biases in text-to-image generation.  In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 2560–2574, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/V1/2023.FINDINGS-ACL.160.  URL https://doi.org/10.18653/v1/2023.findings-acl.160.
  * Weidinger et al. [2023] Laura Weidinger, Maribeth Rauh, Nahema Marchal, Arianna Manzini, Lisa Anne Hendricks, Juan Mateos-Garcia, Stevie Bergman, Jackie Kay, Conor Griffin, Ben Bariach, Iason Gabriel, Verena Rieser, and William Isaac.  Sociotechnical safety evaluation of generative ai systems, 2023.
  * West and Fenstermaker [1995] Candace West and Sarah Fenstermaker.  Doing difference.  _Gender & Society_, 9(1):8–37, 1995.
  * West and Zimmerman [1987] Candace West and Don H Zimmerman.  Doing gender.  _Gender & Society_, 1(2):125–151, 1987.
  * Wolfe et al. [2023] Robert Wolfe, Yiwei Yang, Bill Howe, and Aylin Caliskan.  Contrastive language-vision ai models pretrained on web-scraped multimodal data exhibit sexual objectification bias.  In _Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’23, page 1174–1185, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400701924.  doi: 10.1145/3593013.3594072.  URL https://doi.org/10.1145/3593013.3594072.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation, 2022.
  * Zhao et al. [2023] Dora Zhao, Jerone Andrews, and Alice Xiang.  Men also do laundry: Multi-attribute bias amplification.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 42000–42017, Honolulu, HI, 23–29 Jul 2023. PMLR.  URL https://proceedings.mlr.press/v202/zhao23a.html.
  * Zhao et al. [2017] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints, 2017.

##  Appendix A Experimental Methods Applied on the Measurement Dataset

Figure 2: Method 1: Distribution-based thresholds applied on the measurements
dataset. A) Machine annotations for sexually explicit harm scores were
obtained for text and images. Text sexually explicit scores were bucketed into
5 buckets. Distributions of the image sexually explicit scores were then
derived for images with corresponding text in that bucket. 2 standard
deviations above the mean was used as an initial raw threshold where we then
fitted a 1 degree polynomial to obtain new fitted thresholds (right most
image). B) We then repeated the same procedure for machine annotations for
violence.

##  Appendix B Evaluation Results on Nibbler Data

Figure 3: Confusion Matrix for Method 1: Distribution-based thresholds on the
Nibbler evaluation dataset. Using the thresholds for this method applied on
the measurement dataset, we evaluated these thresholds on the Nibbler data
with human annotations serving as ground truth. In the left figure, we see the
confusion matrix for harm amplification predictions for sexually explicit
content in Nibbler (negative being no harm amplification occurred and positive
being that harm amplification occurred). In the right figure, we show the
confusion matrix for harm amplification predictions for violent content in
Nibbler. Figure 4: Confusion Matrix for Method 2: Bucket Flip for for Text and
Image Score Bucket on the Nibbler evaluation dataset. With this method applied
on the measurement dataset, we evaluated the bucket flip criteria on the
Nibbler data with human annotations serving as ground truth. The left figure
shows the confusion matrix for sexually explicit content and the right figure
shows the confusion matrix for violent content Figure 5: Comparing the PR
curves with different amount of terms in Method 3. For sexually explicit
content, we show no notable difference between the PR curves for 3 terms
("porn", "sexual", "nudity") and 10 terms ("porn", "sexual", "nudity",
"erotic", "sex", "breasts", "penis", "vagina", "erotica", and "indecency").
AUC difference was also minimal (3 terms: 0.846, 10 terms: 0.848). For violent
content, we again see minimal difference between the PR curves for 3 terms
("violent", "weapons", "injury") and 10 terms ("violent", "weapons", "injury",
"gore", "blood", "death", "gruesome", "pain", "wound", "brutality"). Again,
AUC difference between 3 terms (0.412) and 10 terms (0.397) was small. Figure
6: Precision-recall curves for the Image-Text Co-embedding method for sexually
explicit content (left) and violent content (right) for the final 3 terms
used.

##  Appendix C Gender Fairness Analysis

Table 3 shows the efficacy of our proposed methods across both the perceived
gender expressions for the harm amplification of sexually explicit content.
Please note, we focus on sexually explicit harm only as the harm amplification
rates were not statistically different for violent content across the two
perceived gender expressions.

Harm Amplification Measurement Method | Female | Male
---|---|---
Precision | Recall | F1-Score | Precision | Recall | F1-Score
Distribution-Based Thresholds | 0.978 | 0.971 | 0.974 | 0.970 | 0.970 | 0.970
Bucket Flip | 0.985 | 0.964 | 0.974 | 0.753 | 0.865 | 0.805
Image-Text Co-embedding | 0.978 | 0.619 | 0.758 | 0.760 | 0.567 | 0.650
Table 3: Efficacy of the proposed methods across the two perceived gender
expressions for sexually explicit harm amplification.