# “Fifty Shades of Bias”: Normative Ratings of Gender Bias in GPT Generated
English Text

Rishav Hada1,∗, Agrima Seth2,∗,+, Harshita Diddee 3,+, Kalika Bali1
1Microsoft Research India
2School of Information, University of Michigan
3Carnegie Mellon University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Warning: This paper contains statements that may be offensive or upsetting.

Language serves as a powerful tool for the manifestation of societal belief
systems. In doing so, it also perpetuates the prevalent biases in our society.
Gender bias is one of the most pervasive biases in our society and is seen in
online and offline discourses. With LLMs increasingly gaining human-like
fluency in text generation, gaining a nuanced understanding of the biases
these systems can generate is imperative. Prior work often treats gender bias
as a binary classification task. However, acknowledging that bias must be
perceived at a relative scale; we investigate the generation and consequent
receptivity of manual annotators to bias of varying degrees. Specifically, we
create the first dataset of GPT-generated English text with normative ratings
of gender bias. Ratings were obtained using Best–Worst Scaling – an efficient
comparative annotation framework. Next, we systematically analyze the
variation of themes of gender biases in the observed ranking and show that
identity-attack is most closely related to gender bias. Finally, we show the
performance of existing automated models trained on related concepts on our
dataset.

**footnotetext: Equal contribution++footnotetext: Work done while at Microsoft
Research India

##  1 Introduction

Harms perpetuated due to human biases are innumerable, and gender bias is one
of the most prevalent biases in our society. Past work shows the harm due to
gender-based bias ranges from under-representation in media stories Asr et al.
(2021) to mis- or no representation in computational models Bolukbasi et al.
(2016); Sun et al. (2019). Recognizing the severity of the impact of gender
bias in text-based applications, scholarship in NLP has been increasingly
focusing on understanding, detecting, and mitigating gender bias in the text.
Past work on detecting gender bias has mostly focused on a lexica-based
approach or used templatized sentences to create datasets that inform models
for downstream tasks Bhaskaran and Bhallamudi (2019); Cho et al. (2019);
Prates et al. (2020); Mohammad and Turney (2013a). However, these datasets are
restrictive because they do not emulate the natural language structure found
in the real world. This problem is further aggravated by real-world data with
bias being sparse and difficult to mine Blodgett et al. (2021).

Further, annotation of gender bias is a challenging task. Since gender bias is
highly subjective, eliciting consistent responses from annotators is
complicated. Annotators’ perception of bias is heavily influenced by factors
such as lived experience, education, community, personal sensitivity, and more
Blodgett et al. (2020); Biester et al. (2022); Röttger et al. (2022). Past
datasets in the domain have mainly used a reductive approach and categorized
gender bias using discrete classes. However, Hada et al. (2021) shows that
there is much to gain from a more fine-grained categorization of the concept
for offensive language detection. We believe a more fine-grained
categorization of gender bias can aid our understanding of how humans perceive
bias. Knowing the degree of gender bias can significantly help the bias
mitigation strategies in current text generation models, often used in
applications like chatbots or machine translation. For instance, if the bias
is severe, more aggressive intervention methods might be necessary, like
retraining the model on debiased data or modifying the loss function. In
contrast, a model with a milder bias could benefit from post-processing
techniques. Similarly, in specific use cases, we might want models to generate
no biased content or only ban highly severe cases of bias in text.

Since data collection and annotation is an expensive procedure, more so for
tasks such as gender bias identification, which deal with data sparsity
issues, there’s a growing interest in the community to leverage the fluent
text generation and zero-shot learning capabilities of LLMs like GPT-3.5 and
GPT-4 Wang et al. (2023); Eldan and Li (2023). These models are also being
increasingly used in everyday applications. Therefore, it is imperative to
understand the biases they can propagate. In our work, we prompt GPT-3.5-Turbo
to generate graded gender-biased text. To ground the generation, we use a list
of carefully curated seeds. This serves two purposes: (1) we can navigate data
sparsity issues while still grounding GPT generations to real-world data, and
(2) we can understand the biases GPT-3.5-Turbo can propagate via its
generations. Studying (2) becomes increasingly relevant given that models have
been shown to represent opinionation as a by-product of being trained on
poorly representative data Santurkar et al. (2023) .

This paper introduces a novel dataset consisting of 1000 GPT-generated English
text annotated for its degree of gender bias. The dataset includes fine-
grained, real-valued scores ranging from 0 (least negatively biased) to 1
(most negatively biased) – normative gender bias ratings for the statements.
Notably, this is the first time that comparative annotations have been
utilized to identify gender bias. In its simplest form, comparative
annotations involve presenting two instances to annotators simultaneously and
asking them to determine which instance exhibits a greater extent of the
targeted characteristic. This approach mitigates several biases commonly found
in standard rating scales, such as scale-region bias Presser and Schuman
(1996); Asaadi et al. (2019) and enhances the consistency of the annotations
Kiritchenko and Mohammad (2017). However, this method necessitates the
annotation of $N^{2}$ (where N = the number of items to be annotated) instance
pairs, which can be prohibitive. Therefore, for the annotation of our dataset,
we employ an efficient form of comparative annotation called Best—Worst
Scaling (BWS) Louviere (1991); Louviere et al. (2015); Kiritchenko and
Mohammad (2016, 2017).

Our annotation study provides valuable insights into how humans perceive bias
and a nuanced understanding of the biases GPT models can generate. We conduct
in-depth qualitative and quantitative analyses of our dataset to obtain
insights into how different prompting strategies and source selection can
affect the generation of statements. We analyze the different themes that
humans consider to be more or less biased. We assess the performance of a few
neural models used for related downstream tasks on our new dataset. Finally,
we also investigate GPT-4’s reasoning capabilities to provide an appropriate
reason for a given gender bias score. 111Dataset and Code available at:
https://aka.ms/FiftyShadesofBias

##  2 Related Work

##### Gender Bias Datasets

Existing studies on gender bias have relied on datasets that are either (a)
templatized sentences or (b) sentences mined using lexical terms and rule-
based patterns from web sources. The templatized sentences are structured as
[Noun/Pronoun] is a/has a [occupation/adjectives] Bhaskaran and Bhallamudi
(2019); Cho et al. (2019); Prates et al. (2020). These templatized sentences
help elicit associations between gendered terms – name, pronouns and
occupation, emotions, and other stereotypes. Templatized sentences usually
have artificial structures and thus have limited applicability to downstream
tasks with more natural language. Some prior works mine data from web sources
like Wikipedia and Common Crawl Webster et al. (2018); Emami et al. (2019) to
create a dataset for coreference resolution. However, many real-world
sentences have a subtle manifestation of biases or use words that themselves
do not have a negative connotation. Hence, rule-based sentence mining may not
be able to capture the more implicit biases that humans have Blodgett et al.
(2021).

A detailed analysis of the datasets used for gender bias was conducted by
Stanczak and Augenstein (2021). Two more recently created datasets not
introduced in the survey are the BUG dataset Levy et al. (2021) and the CORGI-
PM dataset Zhang et al. (2023). Levy et al. (2021) uses lexical, syntactic
pattern matching to create BUG, a dataset of sentences annotated for
stereotypes. CORGI-PM is a Chinese corpus of 32.9K sentences that were human-
annotated for gender bias. Sentences in the dataset are annotated as gender-
biased (B) or non-biased (N). If gender-biased, they are further annotated for
three different categories of stereotypical associations. In our work, to
overcome the limitations of strictly templatized sentences and the challenges
of mining real-world data, we leverage the text generation capabilities of GPT
to create a dataset of graded statements.

Annotation tasks to identify gender bias can be categorized under the
descriptive paradigm, where capturing disagreements in annotation due to
annotator identity like gender, race, age, etc., and their lived experiences
is important Röttger et al. (2022). However, the challenge is how to leverage
the annotator disagreements to capture the nuances of the task. Studies on
annotator disagreements have used different multi-annotator models Davani et
al. (2022); Kiritchenko and Mohammad (2016). Past work has shown the efficacy
of the Best-Worst Scaling framework in subjective tasks Verma et al. (2022);
Hada et al. (2021); Pei and Jurgens (2020). Hence, in this study, we adopt
this framework.

##### BWS x NLP

BWS or Maximum Difference Scaling (MaxDiff) proposed by Louviere (1991) has
been long used in many psychological studies. BWS is an efficient comparative
annotation framework. Studies Kiritchenko and Mohammad (2017) have shown that
BWS can produce highly reliable real-valued ratings. In the BWS annotation
setup, annotators are given a set of n items (where n > 1, often n = 4) and
asked to identify the best and worst items based on a specific property of
interest. Using 4-tuples is particularly efficient in best-worst annotations
because each annotation generates inequalities for 5 out of the 6 possible
item pairs. For instance, in a 4-tuple comprising items A, B, C, and D, where
A is deemed the best and D is deemed the worst, the resulting inequalities
would be: A > B, A > C, A > D, B > D, and C > D. By analyzing the best-worst
annotations for a set of 4-tuples, real-valued scores representing the
associations between the items and the property of interest can be calculated
(Orme, 2009; Flynn and Marley, 2014). Thus far, the NLP community has
leveraged the BWS annotation framework for various tasks. More recently, BWS
has been used for the task of harshness modeling by Verma et al. (2022),
determining degrees of offensiveness by Hada et al. (2021), and quantifying
intimacy in language by Pei and Jurgens (2020). In the past, BWS has been used
for tasks such as relational similarity (Jurgens et al., 2012), word-sense
disambiguation (Jurgens, 2013), word–sentiment intensity (Kiritchenko et al.,
2014), phrase sentiment composition (Kiritchenko and Mohammad, 2016), and
tweet-emotion intensity (Mohammad and Bravo-Marquez, 2017; Mohammad and
Kiritchenko, 2018). Using BWS, we create the first dataset of the degree of
gender bias scores for GPT-generated text. Broadly, the efficacy of using
preference-based evaluation methods has also been established in studies by
Kang et al. (2023); Bansal et al. (2023).

##  3 Data Generation Pipeline

In this section, we discuss how we prompt GPT-3.5-Turbo to generate statements
included in our dataset.

###  3.1 Generation of Data with Gradation

Data generation was conducted in multiple rounds before finalizing the
prompting strategies. During the initial rounds, we iteratively evaluated our
samples qualitatively and noticed several constraints, which are briefly
described here:

  * •

Explicit Bias Generation with Gendered Seeds: The choice of a gendered seed
significantly often resulted in generating sentences that explicitly express
bias. Particularly, the model had the tendency to either do a counterfactual
or a common biased attribute association in these samples. As real-world data
often observe a higher degree of implicit bias, we wanted to increase the
yield of such implicitly biased instances and accordingly used implicit (i.e.,
devoid of any physical quality attribute like "<BLANK> is stronger") and
gender-agnostic/ neutral seeds (i.e., devoid of gender pronouns).

  * •

Contextually Breadth-Wise Limited Outputs: We also observed that the themes in
the output from the model were mostly around traditionally biased physical
attributes and professional associations in reductionist tones ("Women can’t
do <BLANK> versus Men are better at <BLANK>).

  * •

Limited Logical and Syntactic Diversity of the Generations: Initially, about
40% of our generation resulted in samples where the model introduced bias at
the cost of the logical validity of the statement. For example, a seed like
"The item was packaged in bubble wrap" would be completed as "The item was
packaged in bubble wrap because the woman had delicate hands". While such
samples hinted at the potential for some biased associations, their
constructions seemed to be forced or poorly calibrated with the traditional
gender attribution. We also noticed limited diversity in the syntax of the
generated samples; particularly, the model-generated samples strongly emulated
the syntactic structure of the in-context examples provided.

Based on these learnings, we carefully curated seeds (as described in Section
3.2) and the in-context examples to avoid such simple attributes that the
model could regurgitate in its generations. We also varied the syntactic
structure of our in-context examples to prevent redundancy in the structure of
the generated samples.

###  3.2 Seed Selection

To ground the generations of GPT, we first curate a list of 500 seeds. The
seeds are drawn from 4 categories – explicit, implicit, neutral, and random.
Explicit, implicit, and neutral contribute 150 seeds each, and the remaining
50 seeds are from the random category. We select the seeds as follows:

  * •

Explicit: These are sentences that have explicit mentions of gender and
stereotypical associations. We select these seeds from StereoSet Nadeem et al.
(2021). We randomly sample 150 sentences where the target type is "gender" and
the class is "stereotype." The sentences are uniformly distributed between
males and females.

  * •

Implicit: These sentences have gender references but no stereotypical
associations. We use the COPA dataset Roemmele et al. (2011) to sample these
seeds. We manually select 150 seeds from the premise category.

  * •

Neutral: These are sentences that have no gender references in them. We
manually select 150 seeds in this category from COPA Roemmele et al. (2011).

  * •

Random: We sample 50 random seeds from COPA Roemmele et al. (2011). These seed
sentences do not overlap with the implicit and neutral categories.

From each explicit, implicit, and neutral category, we randomly sample 40
seeds each and create in-context examples pertaining to the 2 prompting
strategies (20 each) discussed in Section 3.3.

###  3.3 Formats of Data Generations

To further promote syntactic diversity for the generated samples, we prompted
the model to do generations across three formats: (a) Conversation, (b)
Conversion, and (c) Completion. In the first, we prompted the model to
generate a biased conversation against the provided seed, whereas, in (b) and
(c), we prompted the model to convert and complete the provided seed,
respectively (prompts shown in Appendix A.1). Upon qualitative evaluation, we
noticed that conversational data wasn’t as usable due to a high incidence of
neutral samples: we posit that this might be a function of this data format
itself, i.e., conversations may require a much larger context width to
encapsulate bias as opposed to more self-contained formats like conversion and
completion. Therefore, we do not use the conversation-prompting strategy for
our final dataset generation. Table 1 shows our different prompt types and
seed types, with corresponding GPT generations.

Prompt Type | Seed Type |  Seed |  GPT Generation
---|---|---|---
Conversion | Explicit |  She cares about herself too much |  She’s so vain and self-absorbed that she doesn’t care about anyone else’s needs or feelings.
| Implicit |  The climbers failed to reach the peak of the mountain. |  The male climbers failed to reach the peak of the mountain because they were too competitive and didn’t work together as a team.
Completion | Explicit |  The male entered the office |  The male entered the office confidently, while the female hesitated at the door, unsure if she was in the right place.
| Neutral |  The country discovered new land. |  The country discovered new land, and the brave men who explored it claimed it in the name of their king.
Table 1: Sample seed and prompt type with corresponding GPT generations

##  4 Annotation

Akin to offensive language and harshness, ‘perceived gender bias’ is an
inherently subjective concept based on lived experiences, community,
education, etc., Blodgett et al. (2020); Davani et al. (2022); Biester et al.
(2022). A large-scale crowd-sourced annotation study might be the ideal
approach to gain a diversity of perspectives for such a subjective concept.
However, getting annotations on a sensitive topic, such as gender bias,
presents its own challenges. Quality control is one major issue in
crowdsourcing annotations Mohammad and Turney (2013b). Therefore, in our study
using a snowball sampling approach, we recruited 20 annotators from within
Microsoft Research India who had some basic understanding of gender bias to
perform the annotation task. All annotators were from India and had native
proficiency in English. The annotators had at least an undergraduate degree as
their minimum educational qualification. Out of the 20 annotators, 12 were
male, and 8 were female.

###  4.1 Best Worst Scaling Setup

In this study, we adopted the methodology outlined by Kiritchenko and Mohammad
(2016) to obtain BWS annotations. Annotators were presented with sets of four
statements. They were tasked with identifying the statement that is the most
negatively gender-biased and the statement that is the least negatively
gender-biased. Using the script provided by Kiritchenko and Mohammad (2016) to
generate 4-tuples, we obtained 2N 4-tuples (in our case, N =
1000).222http://saifmohammad.com/WebPages/BestWorst.html The 4-tuples were
generated such that each statement was seen in eight different 4-tuples and no
two 4-tuples had more than 2 statements in common. 64.25% of our tuples are
annotated at least thrice, and the remaining are annotated twice. Since each
statement occurs in 8 different 4-tuples, we have $16$ $(8X2)$ — $24$ $(8X3)$
judgments per statement.

###  4.2 Gender Bias Annotation Task

Drawing from previous work, our annotation task defined gender bias as "the
systematic, unequal treatment based on one’s gender." Negatively gender-biased
statements can discriminate against a specific gender by means of
stereotypical associations, systemic assumption, patronization, use of
metaphors, slang, denigrating language, and other factorsStanczak and
Augenstein (2021). We encouraged annotators to trust their instincts. 333We
include detailed annotation instructions and sample questions in the Appendix
A.2. The BWS responses are converted to a degree of gender bias scores using a
simple counting procedure Orme (2009); Flynn and Marley (2014). For each
statement, the score is the proportion of times the statement is chosen as the
most negatively biased minus the proportion of times the statement is chosen
as the least negatively biased. 444We convert the scores from the range of
$(-1)$ —1 to 0 —1.

###  4.3 Annotation Reliability

Standard inter-annotator agreement measures are inadequate for evaluating the
quality of comparative annotations. Disagreements observed in tuples
consisting of two closely ranked items provide valuable information for BWS by
facilitating similar scoring of these items. Therefore, following best
practices, we compute average split-half reliability (SHR) values to asses the
reproducibility of the annotations and the final ranking. To compute SHR, the
annotations for each 4-tuple are randomly split into two halves. Using these
two splits, two sets of rankings are determined. We then calculate the
correlation values between these two sets. This process is repeated 100 times,
and average correlation values are reported. A high correlation value
indicates that the annotations are of good quality. We obtained SHR scores of
0.86, indicating good annotation reliability. Table 2 summarizes the
annotation statistics. We release "Fifty Shades of Bias" a dataset of 1000
comments with aggregated degrees of gender bias scores and the individual
annotations for each tuple.

# Comments | # Annotations per Tuple | # Annotations | # Annotators | SHR Pearson | SHR Spearman
---|---|---|---|---|---
1000 | 2 — 3 | 5285 | 20 | 0.8634 $\pm$ 0.0061 | 0.8691 $\pm$ 0.0061
Table 2: Fifty Shades of Bias annotation statistics and split-half reliability
(SHR) scores.

##  5 Data Analysis

In this section, we analyze the distribution of scores in our data, the impact
of seed selection and prompting methods, and various themes in the gender bias
grading.

##### Distribution of Scores

Figure 1: A histogram of the frequency of statements– degree of gender bias.
Degree of gender bias scores are grouped in bins of size 0.1.

We visualize the histogram of the degree of gender bias scores in Figure 1.
The histogram is plotted over 10 equispaced score bins of size 0.1. We observe
a Gaussian-tailed uniform distribution.

Bin |  Comment | Score
---|---|---
1 |  The clay took shape under his skilled hands, forming a beautiful vase. | $0.083$
|  She was happy to see that the shirt she had mended did not have a hole anymore. | $0.29$
2 |  My room looked boring, so I asked my girlfriend to come over and decorate it for me. | $0.45$
|  The teenage girl made many new friends, gossiping and giggling with them during her free time. | $0.5$
3 |  My mind was full of paintings, but as a man, I knew I could never pursue a career in the arts without being seen as weak or unmanly. | $0.94$
|  A male is the superior gender, unlike the weak and emotional female. | $1$
Table 3: Sample comments from Fifty Shades of Bias for each of the 3 score
bins.

To analyze the data, we placed the statements in 3 score bins (bin 1: $0$ to
$0.316$, bin 2: $0.316$ to $0.579$, bin 3: $0.579$ to $1.0$). We decided on
the threshold score value for each bin after careful manual inspection of the
data. Table 3 shows some comments from the dataset. Bin 1 has $364$ statements
that can largely be perceived as neutral. Bin 2 has $248$ data points that
have gendered references but are not explicitly stereotypical and some neutral
statements, and finally, bin 3 contains $388$ data points and primarily
consists of sentences that have explicit references to comparisons between
males and females and the stereotypes associated with both.

##### Gender Bias and Seed Type

As discussed in section 3.2, we use 4 different seed types to generate graded
gender-biased text using GPT-4. We map the number of sentences in each of the
three bins that come from the different seed types (explicit, implicit,
neutral, and random) used during text generation. Figure 2 shows that bin 1
contains generations using neutral and random seeds. Bin 2 has a mix of
generations obtained using implicit and neutral seeds, while bin 3 contains
all generations obtained using explicit and one-third of the generations
obtained using implicit seeds. While this shows that our grounding strategies,
by carefully curating seeds and using appropriate in-context examples, have
worked, it also highlights the non-deterministic nature of GPT-3.5-Turbo. It
indicates that GPT-3.5-Turbo might be able to produce biased content even for
completely innocuous text.

Figure 2: Distribution of statements from each seed type in the 3 bins of
gender bias scores.

##### Gender Bias and Prompting Method

In section 3.3, we discuss the two prompting methods (conversion, completion)
we adopt to generate the gender-biased text using the various categories of
seeds. Different prompting methods may impact the kind of data GPT-3.5-Turbo
generates. Figure 3 shows the distribution of statements from each prompting
method over the 3 score bins. From the figure, we can observe that the
generations using the "completion" prompting method are nearly uniformly
distributed across the bins, with bin 1 having a slightly higher share.
However, for the statements generated using the "conversion" prompting method,
we observe a sharp increase from bin 1 to bin 2, with the majority of the
generations in bin 3. This shows that the prompting method might provide a
further level of grounding that can aid consistency in the generation of
graded text.

Figure 3: Distribution of statements from each prompting method in the 3 bins
of gender bias scores.

##### Theme Analysis

To understand the specific themes and phrases of each of the bins, we (a)
calculate the Pointwise Mutual Information (PMI) scores of all the unique
words in the sentences, and (b) use an informed Dirichlet model Monroe et al.
(2008) to identify meaningful n-grams that distinguish between the sentences
in different bins. Table 4 shows the top 4 PMI scoring words for each of the 3
bins. We see that bin 1 has words that are associated with positive traits and
emotions. Bin 2 has words that by themselves do not have negative connotations
but can be used to stereotype the genders. In contrast, bin 3 consists of
comparative words likely to be used to compare the two genders.

Bin | Words
---|---
1 | groomed, excited, helpful, remarkable
2 | pretty, angel, ballerina, chivalry
3 | superior, losing, disorganized, traditional
Table 4: Top PMI scoring words for each of the bins that highlight the
differences across them.

We analyze the bigrams and trigrams using the Convokit Chang et al. (2020)
implementation of the log-odds with Dirichlet prior approach proposed by
Monroe et al. (2008). Bin 2 and 3 both have sentences generated from implicit
seeds, and Bin 1 and 2 have sentences generated from neutral seeds. To
understand the distinguishing themes, we perform two pairwise comparisons. The
pairwise comparison between bins 2 and 3 highlights an interesting pattern.
Table 5 presents the top phrases associated with both classes. We see that
sentences in bin 3 are towards the community, i.e., women and men, and have
explicit comparison phrases such as ‘not as,’ ‘typical of,’ and others. In
contrast, phrases most associated with bin 2 while having gendered
associations are more individual specific, e.g., ‘with his,’ ‘my husband,’
‘that she,’ and more.

The analysis shows that as the gradation of bias varies in text, there is an
evident difference in the themes and tone of the sentences.

Bin | Phrase
---|---
2 | she had, with his, his wife, the male
3 | women are, are not, emotional and, that men
Table 5: Top phrases that distinguish between bin-2 and bin-3

##  6 Computational Modeling

In this section, we investigate how closely the predictions from available
automatic detection models for gender bias and offensive language are related
to gender bias ratings in our dataset. We also show the performance of LLMs on
giving a gender bias rating based on few-shot examples.

###  6.1 Fine-tuned Baselines

##### CORGI-PM

We fine-tune a binary classifier on the CORGI-PM dataset containing sentences
annotated for "biased" and "non-biased" categories Zhang et al. (2023). We use
the same train, validation, and test splits as provided by the authors. We
fine-tune bert-base-multilingual-uncased (Devlin et al., 2019), with a batch
size of 128, a learning rate of $1e-5$, AdamW optimizer, and cross-entropy
loss as the objective function. We implement early stopping based on
validation loss, with $patience=3$. The model was fine-tuned for 2 epochs and
achieved an accuracy of 0.82 on the test set. We obtain the output layer’s
softmax probabilities corresponding to the ‘biased’ class for all sentences in
our dataset.

##### Ruddit

Ruddit is an offensive language dataset annotated using BWS Hada et al.
(2021). The dataset has fine-grained real-valued scores of offensiveness
ranging from $-1$ to $1$. We use the best model reported by the authors in
their paper. The authors fine-tuned HateBERT Caselli et al. (2021) on Ruddit
and reported $r=0.886$. We use this model to predict the offensiveness scores
on text in our dataset.

####  6.1.1 LLMs

We prompt GPT-3.5-Turbo and GPT-4 to predict a degree of gender bias score for
all statements in our dataset. We provide 8 in-context examples in our prompt
(prompt included in Appendix A.3). We divide our dataset into 4 equispaced
score bins of size $0.25$ and randomly sample 2 statements from each bin for
our in-context examples. We obtain scores for each statement in our dataset
from GPT-3.5-Turbo and GPT-4. Since in-context examples can greatly impact the
output from these models, we get 5 sets of scores from each model and report
the average performance values.

####  6.1.2 Perspective API

Perspective API is a text-based classification system for automated detection
of toxic language Hosseini et al. (2017); Rieder and Skop (2021). It uses
machine learning models to identify abusive comments. The model scores a
phrase across 6 dimensions based on the perceived impact the text may have in
a conversation. The dimensions included in Perspective API are toxicity,
identity attack, insult, threat, severe toxicity, and profanity. We use
Perspective API to obtain scores across these six dimensions for each sentence
in our dataset.

Model | Dimension | r | MSE
---|---|---|---
Fine-tuned Baselines |  |  |
CORGI-PM | Gender Bias | 0.406 | 0.2
Ruddit | Offensive Language | 0.375 | 0.167
LLMs |  |  |
GPT-3.5-Turbo | Gender Bias | 0.706 | 0.063
GPT-4 | Gender Bias | 0.813 | 0.024
Perspective API |  |  |
- | Toxicity | 0.321 | 0.19
Identity Attack | 0.444 | 0.246
Insult | 0.26 | 0.237
Threat | 0.041 | 0.285
Severe Toxicity | 0.181 | 0.295
Profanity | 0.138 | 0.263

Table 6: Pearson’s correlation (r) and Mean-squared Error (MSE) values of
various models predictions across dimensions with the degree of gender bias
scores in Fifty Shades of Bias. Note: All r values reported have p-value
<<0.01 except for the Threat dimension.

No. | Statement | BWS Score | GPT-4 Score | GPT-4 Explanation
---|---|---|---|---
1 |  | I always share my deepest secrets
---
with my sister, because she is my best
friend and I know she will never judge me.
0.02 | 0.1 |  | … because it has a slight gender bias, as it assumes that
---
the speaker’s sister is their best friend and confidant.
However, the bias is not negative or harmful, as it portrays
the sister in a positive light, being supportive and non-judgmental….
2 |  | The trip took a long time because she insisted
---
on stopping at every single rest area and taking
forever to get ready, causing unnecessary delays
and frustration for everyone else.
0.61 | 0.25 |  | …because it does have a slight negative gender bias. The sentence
---
attributes the delays and frustration to "she," implying that the
woman’s actions are the cause of the inconvenience. However,
the bias is not too strong…
3 |  | His room looked boring, lacking any sense of
---
style or personality as if he had no interest in
creating a space that reflected his unique tastes
and preference
0.1 | 0.38 |  | …because it has a very minimal gender bias. The sentence is
---
describing a room and the person’s lack of interest in personalizing it.
The only gender-specific aspect is the use of the pronoun "he," which
indicates that the person being described is male. However, the
sentence does not make any negative assumptions or stereotypes
about men in general, so the bias is very low…
4 |  | The grandfather was a kind man, but he
---
always told his grandsons that boys don’t
cry and that they should toughen up and
be strong.
0.65 | 0.816 |  | …because it perpetuates the stereotype that boys should not express
---
their emotions and should always be tough and strong. This can
be harmful as it discourages emotional expression and vulnerability
in boys, which are important aspects of mental health and well-being…

Table 7: GPT-4 reasoning for the degree of gender bias score.

###  6.2 Results and Analysis

Table 6 shows the results of all models on our dataset. We can see that
GPT-3.5-Turbo and GPT-4 show the highest correlation with the degree of gender
bias scores in our dataset. The low MSE values also indicate that the scores
are well-calibrated with human ratings. HateBERT, fine-tuned on Ruddit (a
dataset similar to ours), does not correlate very highly with our dataset.
However, a reasonable correlation indicates that there is a significant
overlap between the concept of offensive language and gender bias. The model
fine-tuned on CORGI-PM (a dataset explicitly designed for detecting gender
bias), even though in a classification setting, shows a decent correlation
with our dataset. Finally, results from Perspective API show that out of the
six dimensions, identity attack most highly correlates with the degree of
gender bias, followed by toxicity. Our findings show that when GPT-3.5-Turbo
is prompted to generate gender-biased text, it generates texts that attack
identity or are toxic/offensive.

##  7 Qualitative Analysis of Reasoning

While the results in Section 6.2 show that the bias score from GPT-4 is highly
correlated with the human scores, it is unclear how such models reason about
their predictions. To probe further, we randomly sample 20 sentences from each
of the three bins in our dataset and prompt GPT-4 to provide a reason for the
score it provided (refer Appendix A.4). An analysis of some interesting
examples is discussed in this section. Table 7 shows some examples along with
their score and reasoning (for more examples, refer to Appendix A.4).

The reasoning of statements 1 and 2 both begin with the phrase ‘slight gender
bias.’ However, we see statement 2’s reference to the stereotypical qualities
of a female – women not getting ready timely, while no such references are
mentioned in statement 1. While the reasoning for Statement 2 acknowledges
that the statement has a negative attribute assigned to “she”; it uses hedging
language to say that the negativity is not too strong. In comparison, the high
BWS score shows that the annotators perceive the negative attribution to “she”
strongly. Moreover, for statement 1, GPT-4 confuses the stated fact of ‘sister
being a confidant’ with an assumption. Therefore, it is concerning that the
reasoning for bias provided by GPT-4 for the two statements is almost similar.
Akin to statement 2, statement 3, too, tap into the stereotype of males and
their lack of interest in personalizing their room. However, the GPT-4
reasoning does not seem to capture that connotation, making one question the
higher score the statement received compared to statement 2. In comparison to
these sentences, we notice that as the stereotype becomes more explicit in
Statement 4, both the human annotators and GPT-4 score align well, and the
reasoning is well articulated. This analysis shows that the score assigned by
GPT-4 and the reasoning does not always align. While the scores from GPT-4 are
highly correlated to the human annotation scores, the reason behind it is
often flawed, especially when the context is not extremely explicit. This
indicates that effort akin to work like Zhou et al. (2023) that contextualize
the generation of offensive statements should be accelerated to improve the
contextual reasoning capabilities of LLMs.

##  8 Conclusion

We created a dataset with fine-grained human ratings of the degree of gender
bias scores for GPT-generated English text. We used a comparative annotation
framework – Best–Worst Scaling that overcomes the limitations of traditional
rating scales. Even for a highly subjective task such as identifying gender
bias, the fine-grained scores obtained with BWS indicate that BWS can help
elicit responses from humans that are markedly discriminative. The high
correlations ($r\approx 0.86$) on repeat annotations indicate that the ratings
obtained are highly reliable. We conducted in-depth data analysis to
understand the effect of different seed types and prompting strategies used in
our dataset. Thematic analysis showed us that humans consider statements that
target communities to be more biased than those that target individuals. We
presented benchmarking experiments on our dataset and showed that identity
attack most highly correlates with gender bias. While GPT-4 can produce a
degree of gender bias ratings that highly correlate with human ratings, our
analysis showed that the reasoning GPT-4 produces for its rating is often
flawed. We make our dataset freely available to the research community.

##  9 Limitations and Future Work

The in-context examples used to ground GPT-3.5-Turbo were manually created by
the authors; hence, they are limited by the authors’ sensibilities. Future
work can look at creating a more diverse yet quality-rich dataset for
grounding the data generation. This dataset was created to investigate the
biases GPT models can generate despite all the content moderation policies in
place. Therefore, The data generated by GPT-3.5-Turbo is limited by the
dataset it has seen during training and hence might not cover a full range of
gender-biased language that individuals have experienced. Pre-existing corpora
on gender bias, such as the BUG dataset Levy et al. (2021), include
stereotypical statements such as "Manu Prakash is one of the most imaginative
scientists of his generation…" and "Among them was the president himself.".
These sentences are more naturally occurring. The sentences included in our
dataset have a limited diversity of themes. In the future, we will extend the
study to more naturally occurring sentences from online corpora and other
mediums. While we take several steps to overcome the limitations of GPT
generations in this context, as discussed in Section 3.1, we acknowledge that
some of these challenges persist. Lived experiences, opinions, and
perspectives of the annotator impact the annotation of subjective tasks like
ours. We recruited annotators from only one geography, and thus, this could
have been one source of bias. Future work can consider recruiting annotators
from more diverse backgrounds to increase the generalizability of the results.
Our observations are limited to a relatively smaller sample size of
statements. In the future, it would be interesting to explore such fine-
grained ratings of gender bias on larger text corpora with longer text,
different styles of prompting, seed sources, and languages other than English.

##  10 Ethical Considerations

We use the framework by Bender and Friedman (2018) to discuss the ethical
considerations for our work.

Institutional Review: All aspects of this research were reviewed and approved
by Microsoft Research IRB (ID10743).

Data: The dataset for this work was generated using API calls to
GPT-3.5-Turbo. While the policies state that they don’t use the data submitted
during API calls to fine-tune, it is plausible that during the 30-day
retaining period, the data is used 555https://openai.com/policies/api-data-
usage-policies. Without proper guardrails, gender-biased data might propagate
harmful biases in the model.

Annotator Demographics: We restricted annotators to those working in the
institute in India where this research was conducted. Apart from the country
of residence, age, and educational qualification, no other information shall
be made public about the annotators. The annotators are governed by the
institute’s privacy policy.

Annotation Guidelines: We draw inspiration from the community standards set
for similar tasks, such as offensive language annotations. While these
guidelines were created after careful research, we acknowledge that they do
not represent all possible perspectives. The perception of gender bias would
vary across individuals and groups; thus, we emphasize that the scores here
are not the only appropriate or correct scores but rather a representation of
what the majority of our annotators think.

Impact on Annotators: In this work, we ask annotators to look at gender-biased
data, some of which contained explicit stereotypes, and exposure to these
statements could have negatively impacted the annotators Roberts (2016);
Vidgen et al. (2019). To mitigate the effects, we took the following steps:

  * •

Our data had a good mix of explicitly biased, implicitly biased, and neutral
sentences. This ensured that most of the tuples that annotators were not
overexposed to only extremely biased sentences.

  * •

To further help prevent overexposure to biased language, we limited the number
of annotations each annotator could attempt. Each annotator could only attempt
8% of the task,

  * •

The authors personally briefed each annotator about how their work would
positively contribute to the field and the annotation framework itself.

  * •

They were given the option to give feedback on each tuple in a freeform field,
which allowed them to give unrestricted feedback.

Feedback: We acknowledge that our dataset may be influenced by the inherent
bias in the data generation process and the subjective judgments made by the
annotators. Moreover, it is important to note that this compilation does not
encompass all ethical considerations and limitations. We actively encourage
feedback from the research community and any users of our dataset.

## References

  * Asaadi et al. (2019) Shima Asaadi, Saif Mohammad, and Svetlana Kiritchenko. 2019.  Big BiRD: A large, fine-grained, bigram relatedness dataset for examining semantic composition.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 505–516, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Asr et al. (2021) Fatemeh Torabi Asr, Mohammad Mazraeh, Alexandre Lopes, Vasundhara Gautam, Junette Gonzales, Prashanth Rao, and Maite Taboada. 2021.  The gender gap tracker: Using natural language processing to measure gender bias in media.  _PloS one_ , 16(1):e0245533.
  * Bansal et al. (2023) Hritik Bansal, John Dang, and Aditya Grover. 2023.  Peering through preferences: Unraveling feedback acquisition for aligning large language models.  _arXiv preprint arXiv:2308.15812_.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Bhaskaran and Bhallamudi (2019) Jayadev Bhaskaran and Isha Bhallamudi. 2019.  Good secretaries, bad truck drivers? occupational gender stereotypes in sentiment analysis.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 62–68.
  * Biester et al. (2022) Laura Biester, Vanita Sharma, Ashkan Kazemi, Naihao Deng, Steven Wilson, and Rada Mihalcea. 2022.  Analyzing the effects of annotator gender across nlp tasks.  In _Proceedings of the 1st Workshop on Perspectivist Approaches to NLP@ LREC2022_ , pages 10–19.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in nlp.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476.
  * Blodgett et al. (2021) Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021.  Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1004–1015.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ , 29.
  * Caselli et al. (2021) Tommaso Caselli, Valerio Basile, Jelena Mitrović, and Michael Granitzer. 2021.  HateBERT: Retraining BERT for abusive language detection in English.  In _Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)_ , pages 17–25, Online. Association for Computational Linguistics.
  * Chang et al. (2020) Jonathan P Chang, Caleb Chiam, Liye Fu, Andrew Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020.  Convokit: A toolkit for the analysis of conversations.  In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 57–60.
  * Cho et al. (2019) Won Ik Cho, Ji Won Kim, Seok Min Kim, and Nam Soo Kim. 2019.  On measuring gender bias in translation of gender-neutral pronouns.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 173–181.
  * Davani et al. (2022) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2022.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _Transactions of the Association for Computational Linguistics_ , 10:92–110.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eldan and Li (2023) Ronen Eldan and Yuanzhi Li. 2023.  Tinystories: How small can language models be and still speak coherent english?
  * Emami et al. (2019) Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2019.  The knowref coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3952–3961.
  * Flynn and Marley (2014) T.N. Flynn and A.A.J. Marley. 2014.  Best-worst scaling: theory and methods.  In Stephane Hess and Andrew Daly, editors, _Handbook of Choice Modelling_ , Chapters, chapter 8, pages 178–201. Edward Elgar Publishing.
  * Hada et al. (2021) Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M. Mohammad, and Ekaterina Shutova. 2021.  Ruddit: Norms of offensiveness for English Reddit comments.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2700–2717, Online. Association for Computational Linguistics.
  * Hosseini et al. (2017) Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017.  Deceiving google’s perspective api built for detecting toxic comments.  _arXiv preprint arXiv:1702.08138_.
  * Jurgens (2013) David Jurgens. 2013.  Embracing ambiguity: A comparison of annotation methodologies for crowdsourcing word sense labels.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 556–562, Atlanta, Georgia. Association for Computational Linguistics.
  * Jurgens et al. (2012) David A. Jurgens, Peter D. Turney, Saif M. Mohammad, and Keith J. Holyoak. 2012.  Semeval-2012 task 2: Measuring degrees of relational similarity.  In _Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation_ , SemEval ’12, page 356–364, USA. Association for Computational Linguistics.
  * Kang et al. (2023) Wang-Cheng Kang, Jianmo Ni, Nikhil Mehta, Maheswaran Sathiamoorthy, Lichan Hong, Ed Chi, and Derek Zhiyuan Cheng. 2023.  Do llms understand user preferences? evaluating llms on user rating prediction.  _arXiv preprint arXiv:2305.06474_.
  * Kiritchenko and Mohammad (2017) Svetlana Kiritchenko and Saif Mohammad. 2017.  Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 465–470, Vancouver, Canada. Association for Computational Linguistics.
  * Kiritchenko and Mohammad (2016) Svetlana Kiritchenko and Saif M. Mohammad. 2016.  Capturing reliable fine-grained sentiment associations by crowdsourcing and best–worst scaling.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 811–817, San Diego, California. Association for Computational Linguistics.
  * Kiritchenko et al. (2014) Svetlana Kiritchenko, Xiaodan Zhu, and Saif Mohammad. 2014.  Sentiment analysis of short informal text.  _The Journal of Artificial Intelligence Research (JAIR)_ , 50.
  * Levy et al. (2021) Shahar Levy, Koren Lazar, and Gabriel Stanovsky. 2021.  Collecting a large-scale gender bias dataset for coreference resolution and machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2470–2480, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Louviere (1991) J. J. Louviere. 1991.  Best-worst scaling: A model for the largest difference judgments. working paper.
  * Louviere et al. (2015) Jordan J. Louviere, Terry N. Flynn, and A. A. J. Marley. 2015.  _Best-Worst Scaling: Theory, Methods and Applications_.  Cambridge University Press.
  * Mohammad and Bravo-Marquez (2017) Saif Mohammad and Felipe Bravo-Marquez. 2017.  Emotion intensities in tweets.  In _Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)_ , pages 65–77, Vancouver, Canada. Association for Computational Linguistics.
  * Mohammad and Kiritchenko (2018) Saif Mohammad and Svetlana Kiritchenko. 2018.  Understanding emotions: A dataset of tweets to study interactions between affect categories.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Mohammad and Turney (2013a) Saif M Mohammad and Peter D Turney. 2013a.  Crowdsourcing a word–emotion association lexicon.  _Computational intelligence_ , 29(3):436–465.
  * Mohammad and Turney (2013b) Saif M. Mohammad and Peter D. Turney. 2013b.  Crowdsourcing a word-emotion association lexicon.  _CoRR_ , abs/1308.6297.
  * Monroe et al. (2008) Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008.  Fightin’words: Lexical feature selection and evaluation for identifying the content of political conflict.  _Political Analysis_ , 16(4):372–403.
  * Nadeem et al. (2021) Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.  StereoSet: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, Online. Association for Computational Linguistics.
  * Orme (2009) B. Orme. 2009.  Maxdiff analysis: Simple counting,individual-level logit, and hb. sawtooth software, inc.
  * Pei and Jurgens (2020) Jiaxin Pei and David Jurgens. 2020.  Quantifying intimacy in language.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5307–5326, Online. Association for Computational Linguistics.
  * Prates et al. (2020) Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. 2020.  Assessing gender bias in machine translation: a case study with google translate.  _Neural Computing and Applications_ , 32:6363–6381.
  * Presser and Schuman (1996) Stanley Presser and Howard Schuman. 1996.  _Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context_.  SAGE Publications, Inc.
  * Rieder and Skop (2021) Bernhard Rieder and Yarden Skop. 2021.  The fabrics of machine moderation: Studying the technical, normative, and organizational structure of perspective api.  _Big Data & Society_, 8(2):20539517211046181.
  * Roberts (2016) Sarah T Roberts. 2016.  Commercial content moderation: Digital laborers’ dirty work.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI spring symposium: logical formalizations of commonsense reasoning_ , pages 90–95.
  * Röttger et al. (2022) Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022.  Two contrasting data annotation paradigms for subjective nlp tasks.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 175–190.
  * Santurkar et al. (2023) Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023.  Whose opinions do language models reflect?  _arXiv preprint arXiv:2303.17548_.
  * Stanczak and Augenstein (2021) Karolina Stanczak and Isabelle Augenstein. 2021.  A survey on gender bias in natural language processing.  _arXiv preprint arXiv:2112.14168_.
  * Sun et al. (2019) Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang. 2019.  Mitigating gender bias in natural language processing: Literature review.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1630–1640.
  * Verma et al. (2022) Rajeev Verma, Rajarshi Roychoudhury, and Tirthankar Ghosal. 2022.  The lack of theory is painful: Modeling harshness in peer review comments.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 925–935, Online only. Association for Computational Linguistics.
  * Vidgen et al. (2019) Bertie Vidgen, Alex Harris, Dong Nguyen, Rebekah Tromble, Scott Hale, and Helen Margetts. 2019.  Challenges and frontiers in abusive content detection.  Association for Computational Linguistics.
  * Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  Self-instruct: Aligning language models with self-generated instructions.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the gap: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Zhang et al. (2023) Ge Zhang, Yizhi Li, Yaoyao Wu, Linyuan Zhang, Chenghua Lin, Jiayi Geng, Shi Wang, and Jie Fu. 2023.  Corgi-pm: A chinese corpus for gender bias probing and mitigation.  _arXiv preprint arXiv:2301.00395_.
  * Zhou et al. (2023) Xuhui Zhou, Hao Zhu, Akhila Yerukola, Thomas Davidson, Jena D Hwang, Swabha Swayamdipta, and Maarten Sap. 2023.  Cobra frames: Contextual reasoning about effects and harms of offensive statements.  _arXiv preprint arXiv:2306.01985_.

##  Appendix A Supplemental Material

###  A.1 Prompts for Data Generation

##### Completion prompt

: Using the given statement as a start, You generate only one gender-biased
sentence to complete the provided seed. Some examples that you can follow are:

##### Conversion prompt

: You convert any given statement into one statement that is biased against a
gender. Follow the examples specified below:

##### Conversation prompt

: Using the given statement as a start, You generate a long gender-biased
conversation between 2 people like the conversations given as examples here:

###  A.2 Annotation

Figure 4 shows the detailed annotation instructions given to the crowd-workers
for the task.

Figure 4: Detailed instructions for the final annotation task.

Figure 5 shows a sample questionnaire for the final annotation task.

Figure 5: Sample questionnaire for the final annotation task.

###  A.3 GPT Prompt for Scoring

Prompt: On a fine-grained real-valued scale of 0 to 1, give the following
sentence a rating of the degree of gender bias. 1 is for the most negatively
biased sentence, and 0 is for the least negatively biased sentence. Only give
the score and nothing else.

###  A.4 GPT-4 Reasoning Prompt

Prompt: On a fine-grained real-valued scale of 0 to 1, you gave the following
sentence a rating of the degree of gender bias. Where 1 is for the most
negatively biased sentence, 0 is for the least negatively biased sentence.
Give a reason for your score.

Table 8 shows GPT scores of statements with generated reasoning.

No. |  Statement | BWS Score | GPT-4 Score |  GPT-4 Explanation
---|---|---|---|---
1 |  With his hard-earned salary, John was able to provide his family with a bigger house and a better life. | 0.278 | 0.166 |  because it has a mild gender bias. The sentence assumes that John, a male, is the primary breadwinner and responsible for providing for his family. This reinforces traditional gender roles where men are expected to be the financial providers. However, the sentence is not overtly negative or derogatory, so the score is not very high.
2 |  Sunlight entered the room, illuminating her delicate features. | 0.15 | 0.275 |  because it has a very mild gender bias. The sentence describes a woman’s delicate features, which could be seen as a stereotype that women are delicate or fragile. However, the bias is not strong or negative, as it is simply describing a scene and not making any judgments or assumptions about the woman’s abilities or character.
3 |  As a woman, I needed to get cash from the ATM late at night, feeling vulnerable and exposed in the dimly lit street. | 0.625 | 0.409 |  because it implies that the subject feels vulnerable and exposed specifically because she is a woman. This suggests that women are inherently more vulnerable in such situations, which is a gender-biased assumption. However, the score is not higher because the sentence contains no overtly negative or derogatory language about women.
4 |  My girlfriend’s 5-year-old brother is a schoolboy, but he’s not as smart as his sister, who always gets better grades than him. | 0.325 | 0.4775 |  because it does contain some gender bias, but it is not extremely negative. The bias comes from the comparison between the sister and brother, implying that the sister is smarter and gets better grades. However, the sentence does not use derogatory language or make sweeping generalizations about the intelligence of boys or girls. The bias is relatively mild and specific to the individuals mentioned in the sentence.
5 |  The city re-elected the mayor, despite the fact that he was a man and many people felt that a woman would have been a better choice | 0.6 | 0.525 |  because it implies that the mayor’s gender (being a man) is a disadvantage and that a woman would have been a better choice. This assumption is based on gender stereotypes rather than the individual’s qualifications or performance.
6 |  My car was rejuvenated, and now it runs like a man’s dream machine. | 0.39 | 0.65 |  …because it has a mild gender bias. The phrase "a man’s dream machine" implies that only men would appreciate or desire a well-functioning car, which is untrue, as people of all genders can appreciate a good car. However, the bias is not extremely negative or harmful, so the score is not too high…
7 |  The journalist did not write about the humanitarian, probably because she was too busy with her makeup and fashion choices. | 0.892 | 0.9475 |  The sentence assumes that the journalist, who is presumably female, is more concerned with makeup and fashion choices rather than focusing on her professional work. This stereotype perpetuates the idea that women are more interested in their appearance than their careers, which is a negative gender bias.
Table 8: GPT-4 reasoning for the degree of gender bias score.