#  VisoGender: A dataset for benchmarking gender bias in image-text pronoun
resolution

Siobhan Mackenzie Hall
&Fernanda Gonçalves Abrantes
&Hanwen Zhu
Grace Sodunke
&Aleksandar Shtedritski
&Hannah Rose Kirk
Oxford Artificial Intelligence Society, University of Oxford

###### Abstract

We introduce VisoGender, a novel dataset for benchmarking gender bias in
vision-language models. We focus on occupation-related gender biases, inspired
by Winograd and Winogender schemas, where each image is associated with a
caption containing a pronoun relationship of subjects and objects in the
scene. VisoGender is balanced by gender representation in professional roles,
supporting bias evaluation in two ways: i) resolution bias, where we evaluate
the difference between gender resolution accuracies for men and women and ii)
retrieval bias, where we compare ratios of male and female professionals
retrieved for a gender-neutral search query. We benchmark several state-of-
the-art vision-language models and find that they lack the reasoning abilities
to correctly resolve gender in complex scenes. While the direction and
magnitude of gender bias depends on the task and the model being evaluated,
captioning models generally are more accurate and less biased than CLIP-like
models. Dataset and code are available at https://github.com/oxai/visogender

##  1 Introduction

Vision-language models (VLMs) are advancing rapidly and reaching ever-wider
audience across numerous applications, such as classification and captioning,
as well as text-to-image retrieval and generation. However, these models are
pre-trained from uncurated image-text pairs scraped from the internet [1, 2]
and so, their outputs can perpetuate or amplify societal biases [3, 4, 5, 6].
How the VLM is used determines the mechanisms of bias transfer from pre-
training to risks of representational and/or allocational harm [7]. For
example, a VLM used for image retrieval may skew towards returning more images
of male doctors, thus entrenching societal perceptions on the association
between gender and career success; or a VLM used for captioning may more
frequently misgender women or non-binary image subjects, aligning with a
capability (un)fairness [8].

Despite the growing body of work on evaluating and mitigating the bias of VLMs
[9, 10, 11, 12, 13], there is a dearth of specifically-designed benchmark
datasets to evaluate the presence of societal biases across downstream tasks –
such as captioning or image retrieval, with the majority of work measuring
biases with pre-existing image datasets such as FairFace [14] or COCO [15,
16], despite their limited real-world transferability and spurious
correlations [12, 17]. In this paper, we introduce the VisoGender benchmark
for evaluating bias in VLMs. The design of VisoGender is inspired by two prior
bodies of works. Firstly, we apply stress-testing of vision-linguistic
reasoning capabilities of VLMs as in the Winoground benchmark [18] but
introduce the dimension of societal biases. Secondly, we adopt the templated
structure to test gender bias in occupational pronoun resolution from NLP
research, specifically the WinoGender [19] and WinoBias [20] frameworks, in
turn inspired by Winograd Schema [21], but apply it to the vision-language
domain. To our knowledge, VisoGender is the first dataset to combine both of
these contributions by stress-testing gender bias in visual-linguistic
reasoning and coreference resolution capabilities.

Figure 1: Resolution of gender pronouns and retrieval with a neutral query. We
resolve gender by (i) using zero-shot classification with CLIP-like models,
and (ii) next-token prediction with captioning models, such as BLIP. We have
an additional simpler task to resolve the gender of a single person, e.g. with
a template “The doctor and her / his stethoscope”.

VisoGender contains images of a person in an occupation (“the doctor”),
combined with either an object (“the stethescope”) or a participant (“the
patient”). Each image is annotated for the gender of the occupation and/or the
participant, and the dataset is balanced across different genders occupying
these roles. Using these components, we construct groundtruth natural language
captions containing a possessive pronoun relationship (“the doctor and his/her
patient”). We test bias in two tasks (see Fig. 1): pronoun resolution and
image retrieval. In the resolution task, the model is provided with a single
image (either of an occupation-object or occupation-participant scene) and
ranks the likelihood of captions containing different gender pronouns. There
are varying levels of difficulty in the resolution task – from a single person
resolution in the occupation-object case, to two person resolution in the
occupation-participant case, where either both subjects are the same gender
(easier) or opposite gender (harder). In the retrieval task, the model is
provided with a single gender-neutral caption and must retrieve images from a
set containing different genders of the occupation. We measure resolution bias
using the gender accuracy gap in correct pronoun resolution (corresponding to
capability fairness) and retrieval bias using commonly-used metrics such as
Bias@K, MaxSkew and NDKL (corresponding to representational fairness).

We present preliminary results for six state-of-the-art CLIP-like models [1,
22, 2, 23, 24, 25] and two state-of-the-art captioning models [26, 27]. We
find that models still struggle to correctly infer the pronoun especially when
there are two people in the image of different genders, where performance is
close to random. Our benchmark also recovers that models display substantial
accuracy gaps between men and women subjects, indicating the presence of
resolution bias, and predominately rank images of men higher than women,
indicating a retrieval bias. We compare these results to US labor force
statistics (as in [28, 20, 19]) and find some correlations between model bias
and societal occupational skew. Our findings demonstrate there is still
substantial progress to be made in improving resolution capabilities, as well
as reducing the gender gap in resolution performance and retrieval outcomes.
The pace at which VLMs are developed is only set to grow in coming years –
VisoGender provides a much-needed benchmark to evaluate their potential
downstream harms before large-scale deployment.

##  2 Related Works

Bias in coreferences in NLP Coreference resolution aims to identify which
mentions in a natural language document link to the same real-world entity
[29]. In the past decade, significant progress has been made moving from rule-
based systems and expert knowledge [30], to statistical models [31, 32] and
deep neural networks [33, 34, 35, 36, 37]. Pronoun resolution involves linking
a noun such as “doctor” to a pronoun in the sentence. Biases have been
identified, with respect to machine translation [38], non-binary pronouns
[39], and favouring masculine entities when resolving gender ambiguous cases
[40]. Our work is most similar to gender pronoun resolution tasks based on
Winograd schemas [21], like Winogender[19] and WinoBias [20] which investigate
occupational-related biases. Both of these works demarcate “hard” and “easy”
cases based on (anti-)stereotypical gender-occupation associations as measured
relative to US labour force statistics. We extend this work to the vision-
language domain. In our resolution task, we modify the typical Winograd scheme
because the correct resolution is unambiguous, i.e., there is a correct
caption (and pronoun) for a corresponding image. However, our retrieval task
is a closer vision-language analogy to [19, 20] because there is no
groundtruth for a “correct” ranking of images given a gender-neutral search
query.

Evaluating visual reasoning There is an emerging body of work on visual
reasoning tasks [41], such as VQA [42, 43, 44], visual word sense
disambiguation [45], compositionality understanding [46, 47, 48],
comprehension [49] or visual entity linking [50]. Most similar to our work,
Winoground [18] evaluates visio-linguistic compositional reasoning by tasking
a model to match two images with two captions containing the same set of
words, only in a different order – such as “there is a mug in some grass” vs.
“there is some grass in a mug”. The task is challenging, with state-of-the-art
VLMs rarely performing better than chance, though [51] demonstrate some of
these failures may be due to atypical images in the dataset. Our vision-
linguistic stress-tests are inspired by adapting Winoground to societal
biases, but a key difference is that our caption-image pairs do not contain
the exact same set of words – for example, matching “the doctor and her
patient” versus “the doctor and his patient”.

Measuring bias in vision-language models. Measuring the societal bias of VLMs
is a growing area of research. While early works measure misclassification
rates into harmful categories [9, 52], more recent methods investigate face-
to-text retrieval [11, 10, 53, 54], or captioning [55]. However, these
approaches rely on off-the-shelf datasets, such as COCO [16], which have been
shown to contain spurious correlations [17] and thus are not suitable for
evaluating model bias [12]. Similar to [12], we balance our dataset by gender
across different occupational settings, but instead using naturally-occuring
images rather than synthetic edits.

##  3 The VisoGender Benchmark

The VisoGender dataset contains 690 images of people in various occupational
settings, where each image is annotated for the inferred groundtruth gender of
the subject(s) in the image. We use these annotations to construct a templated
caption of a correct pronoun relationship. The dataset covers 23 unique
occupations in a hierarchical taxonomy. Each occupation appears in the dataset
with two template forms – either as a single person in the image with a
possessive pronoun to an object (“the doctor and his/her stethoscope”), or as
one of two people in the image with a possessive pronoun to a participant
(“the doctor and his/her patient”) (see Sec. 3.1). A summary of the dataset is
presented in Tab. 1. In the following subsections, we first present further
details of the templates (Sec. 3.1). We then introduce the two types of VLMs
which are compatible with VisoGender (Sec. 3.3), and finally, define the two
tasks in which we measure model bias (Sec. 3.4).

###  3.1 Templates

Each templated caption contains three components, adapted from [19]:

  * •

Occupation: a person refered to by an occupational noun and definite article,
“the doctor”

  * •

Pronoun: a pronoun corresponding to the ground truth gender of the occupation
in the image, e.g., “her” or “his”

  * •

either Object: a noun corresponding to typical professional items, e.g., “the
stethoscope”

  * •

or Participant: A second person in a typical professional relationship with
the occupation, e.g., “the patient”.

For occupations, we use the list from [19], but remove (i) occupations without
a clear possessive pronoun relationship between the occupation and
participant, e.g., “the plumber and their houseowner” is not semantically
correct, and (ii) occupations without sufficient open-domain images across
genders (for both men and women occupying the occupation and participant
roles). We classify the remaining occupations into a hierarchical taxonomy to
permit aggregate group-wise analysis: Sector describes the general field, and
includes education, medical, office, retail and service; Specialisation
describes subcategories within the sector, where for example services includes
food services, fashion, animal or household; and finally Occupations are
nested within specialisations, where for example food services contains
waiter, bartender, and baker. Similar to [19, 20, 28], we match US labour
force statistics on the percentage of men working in each occupation to
compare model biases to occupational societal skew. The full taxonomy and list
of occupations is presented in the Supplementary Materials. We also source the
list of participants from [19] but replace any references to children as
participants and in some cases, make modifications for a more natural
possessive pronoun, e.g., “the lawyer and the witness” becomes “the lawyer and
their client”. For objects, we manually define a typical professional item for
each occupation. Using these components, we construct three templates
(subtasks) of increasing difficulty for coreference resolution:

  * •

Single Subject: The template of captions is “The {occupation} and {his/her}
{object}”, e.g. the doctor and her stethoscope. For each occupation, we
collect 10 occupation-object images, 5 for each gender. Here, models only need
to resolve the pronoun of one subject in the image, thus testing simple gender
detection capabilities.

  * •

Two Subjects of the Same Gender: The template of the captions is “The
{occuptation} and {his/her} {participant}” e.g. the doctor and her patient. In
this case, the gender of the occupation and the participant are the same (both
men or both women). Per occupation, we collect 5 images for each of these two
cases (M-M, W-W). Here, the model must resolve the gender of two subjects but
assigning which subject is the occupation and which is the participant does
not affect the correct pronoun.

  * •

Two Subjects of Different Gender. Finally, we use the same occupation-
participant template but now the participant and the occupation are of
opposite genders (one man and one woman). Per occupation, we collect 5 images
for each of case (M-W, W-M). Here, the model must resolve the gender of the
subject, and inferring from image context which is the occupation and which is
the participant to identify the correct pronoun.

Table 1: VisoGender dataset summary, showing the hierarchy of included
Sectors, Specialisations, and Occupations; the gender pairs per template type,
and the counts of images within each split of the dataset.

| Categories | Number of images |
---|---|---|---
| Sect. | Spec. | Occ. | Gender Pairs |  | Images per
---
Occ.
| Images per
---
Gender Pair
| Images
---
per Gender Pair
and Occ.
Overall
| Single person
---
(occupation-object)
5 | 13 | 23 | [M, W] | 10 | 115 | 5 | 230
| Two-person
---
(occupation-participant)
5 | 13 | 23 | [MM, WW, MW, WM] | 20 | 115 | 5 | 460
|  |  |  |  |  |  | Total | 690

###  3.2 Dataset Collection

The VisoGender dataset comprises image URLs with annotations for the
occupation noun, the participant or object noun, and the inferred groundtruth
gender of the occupation and participant. These annotations can be used to
reconstruct the templated captions. Data collection was carried out by the
authors of the paper from March to May 2023 on a variety of image databases
and search providers, such as Pexels and Google Image Search. We followed a
set guidelines to specify exclusion and inclusion criteria, detailed in the
Supplementary Materials.

We ensure that there are no duplicate images nor invalid URLs across the
dataset, i.e., no overlaps between occupations. In the early stages of data
collection, we used the entire list of occupations from [19]. However, we only
include those with at least 20 viable URLs (5 per gender pair) for occupation-
participants and 10 viable URLs for occupation-object (5 per gender). The
image curation process (and availability of viable URLs) is dependent on the
retrieval of different gendered roles across occupational search queries and
so therefore compromises by inherent representational biases in these systems.
We mitigate effects of imbalance across genders by only including occupations
with a full set of images (equal images across all gender pairs) but this may
introduce a sample selection bias to the included occupations. Furthermore,
inferring gender from an image depends on ingrained biases of the dataset
curators. We discuss limitations and biases of data collection in Sec. 5.1,
and possible expansions in the future with more resources e.g, partnering with
a stock photo company. The dataset is accompanied by a Datasheet for Datasets
[56] in the Supplementary Materials.

###  3.3 Two Supported Types of Vision-Language Models

VisoGender is designed to accommodate two types of VLMs. Here we discuss their
properties, and how bias can be measured in common usecases.

CLIP models CLIP-like models, first introduced by [1], have separate vision
and language encoders and are trained to jointly match images and text. Given
an image $i\in\mathcal{R}^{3\times H\times W}$ and text $t$, a CLIP model
outputs a score $s(i,t)$ that expresses the degree of compatibility between
the image and text. The first common use case of CLIP models is zero-shot
classification of images [1, 57, 58]. This is done by providing a query image
$i_{q}$ and text prompts $t_{n},n\in 1,\ldots,N$. For example, if we wish to
zero-shot classify the gender of a doctor in an image using pronoun
resolution, we can provide text prompts “This is an image of a doctor and
{his, her} notebook”, and select the gender with the highest compatibility
score to the image. Such a classifier can be considered biased if, for
example, it more accurately predicts one gender in some occupation. The second
common use case of CLIP models is for text-to-image retrieval [1]. Given a
text query $t_{q}$, images $i_{n},n\in 1,\ldots,N$ and a query size $K$, we
select the $K$ images with the highest compatibility score to the text prompt.
In this setting, the model can be biased if, for example, when searching for a
given occupation, people from a given demographic are over or under-
represented in the top $K$ retrieval results.

Captioning models Captioning models are most commonly trained to
autoregressively predict a text caption given an image. For an image $i$ and,
optionally, a partially completed caption with N tokens $c=[t_{1},\ldots
t_{N}]$, the model outputs the probability for the next token $t_{N+1}$ as
$p_{\mathrm{cap}}(t_{N+1}|i,t_{1},\ldots,t_{N})$. Similar to CLIP models, we
can apply the captioning model to predict the gender of a subject in an image
via pronoun resolution. We first supply a query image $i_{q}$ (say, an image
of a doctor) and a caption $c_{q}$ like "An image of a doctor and". We then
inspect the probability distribution for the next token $t_{n}$, denoted by
$p_{\text{cap}}(t_{n})=p_{\text{cap}}(t_{m}|i_{q},c_{q})$. We can now compare
the probabilities $p_{\text{cap}}(t_{n})=$ “her" and
$p_{\text{cap}}(t_{n})=$“his", choosing the one with the higher score as the
model’s prediction. It has been demonstrated that comparing token
probabilities is a more reliable measure of a generative language model’s
performance compared to free generation [59], and such templates have been
successfully used to evaluate bias in LLMs [28].

###  3.4 Two Angles of Model Bias

The VisoGender setup has the flexibility to measure model bias in two ways:

Resolution Task The resolution task considers a single image with groundtruth
gender label and matches it to multiple candidate captions containing
different gender pronouns. For example, we start with an image containing a
female doctor, and specify the set of candidate captions as “the doctor and
her/his patient”. We define resolution accuracy, $RA$, as the percentage of
correctly resolved pronouns. This can be calculated over all occupations,
across main occupation categories, or per occupation. For a given occupation
$o\in O$ and a gender $g$ (either male $m$ or female $f$), we have:

$RA_{g}(o)=\frac{\text{Number of correctly resolved pronouns of gender
}g\text{ in occupation }o}{\text{Total number of pronouns of gender }g\text{
in occupation }o}$

An unbiased outcome is one where the model resolves both gender pronouns
equally, i.e., $RA_{m}(o)=RA_{f}(o),\;\forall o\in O$. We now define
resolution bias as the gender resolution accuracy gap

$\Delta(o)=RA_{m}(o)-RA_{f}(o),$ (1)

where a positive value of $\Delta$ shows a model more accurately resolves men,
and vice versa.

Our definition of resolution bias measures a form of capability fairness,
i.e., whether a system performs equally well across subgroups [8]. This task
is applicable to both types of VLMs.

Retrieval Task The retrieval task considers a single gender neutral caption
for a given occupation and matches it to multiple images containing different
gender subjects from the same occupation. For example, we start with the
caption “the doctor and their patient” and define the set of candidate images
as containing 50% images of doctors who are men and 50% who are women. Given
there is no groundtruth for a “correct” ranking of images for a gender-neutral
caption, we cannot define a retrieval accuracy metric. For defining retrieval
bias, we use 3 commonly used metrics – Bias@K [60], Skew@K [61, 11] and NDKL
[62, 61]. Bias@K measures the overrepresentation of men in the top K retrieval
results. Skew@K measures the difference between the desired proportion of
image attributes and the observed one, and MaxSkew@K is the maximum Skew among
all attributes, or the “largest unfair advantage” [61] belonging to images of
any gender. NDKL is a ranking measure that measures the distance from a fair
distribution. For further definitions and discussions of these, please refer
to the Supplementary Materials.

Our definition of retrieval bias measures a form of representational fairness,
i.e., with a gender-balanced set of images and a gender-neutral caption,
whether occupations of each gender have equal chances of being retrieved. The
retrieval task is only applicable to CLIP models.

##  4 Results

For the resolution task, we evaluate six CLIP-like models – CLIP [1], OpenCLIP
[22] (trained on LAION 2B and 400M [2]), SLIP [23], DeCLIP [24], FILIP [25]
(last 3 trained on YFCC-15M [63]); and two state-of-the-art captioning models
– BLIP-2 [26] and GIT [27]. For two candidate models (CLIP and BLIP-2), we go
into more detail by investigating their resolution capabilities and resolution
biases, which are also compared to U.S. Labor Force Statistics (Sec. 4.1). We
ablate the VisoGender setup by changing the order of templates and including a
neutral caption. For the retrieval task, we benchmark the same six CLIP-like
models as the resolution task. Captioning models are not compatible with the
retrieval task. We also compare retrieval bias metrics with US Labor Force
Statistics. For all CLIP models, we use ViT-B/32 encoders, and for GIT we use
the GIT-Large model.

###  4.1 Resolution Task

Table 2: Resolution Bias. We present resolution accuracy averaged for male and
female, as well as the resolution accuracy gap $\Delta$, as defined in eq. 1.
“Same gender” and “Different gender” are images with two people from the same
or different gender, respectively. A positive gap $\Delta$ denotes better
resolution accuracy for men. We also present reported zero-shot classification
accuracy on Imagenet [64].

| Overall | Single Person | Two people | ZS Imagenet
---|---|---|---|---
| Overall | Same gender | Different gender
Model | RAavg | $\Delta$ | RAavg | $\Delta$ | RAavg | $\Delta$ | RAavg | $\Delta$
CLIP [1] | 0.75 | 0.92 | -0.14 | 0.57 | -0.27 | 0.79 | -0.18 | 0.36 | -0.35 | 63.2
OpenCLIP2B [22] | 0.78 | 0.96 | -0.07 | 0.60 | -0.37 | 0.77 | -0.42 | 0.44 | -0.32 | 66.2
OpenCLIP400M [22] | 0.74 | 0.84 | -0.27 | 0.64 | -0.29 | 0.80 | -0.26 | 0.46 | -0.33 | 62.9
SLIP [23] | 0.60 | 0.77 | 0.14 | 0.43 | 0.14 | 0.51 | 0.12 | 0.34 | 0.17 | 34.3
DeCLIP [24] | 0.70 | 0.87 | 0.06 | 0.52 | -0.17 | 0.74 | -0.14 | 0.29 | -0.19 | 43.2
FILIP [25] | 0.45 | 0.41 | 0.06 | 0.49 | 0.36 | 0.49 | 0.36 | 0.50 | 0.37 | 39.5
BLIP-2 [26] | 0.84 | 0.92 | -0.09 | 0.76 | 0.07 | 0.93 | 0.06 | 0.60 | 0.09 | —
GIT [27] | 0.84 | 0.96 | -0.07 | 0.72 | -0.27 | 0.97 | -0.07 | 0.48 | -0.47 | —

Evaluating Resolution Capabilities We present results in Tab. 2, split
according to the different levels of difficulty. We report the mean resolution
accuracy $RA_{\mathrm{avg}}$ for each difficulty level, together with the
resolution bias or accuracy gap $\Delta$. As expected, the resolution accuracy
is highest when there is one person in the image, and lowest when there are
two people of different gender in the image. The accuracies for the latter are
consistently worse than random chance, pointing at the models’ inability to
reason about scenes with multiple people and attributes associated with each
of them. This confirms the findings of prior works that conclude that VLMs are
not capable of visio-linguistic [18] or spatial [65] reasoning. Captioning
models are better than or on par with CLIP models for all levels of
difficulty. In Fig. 2 we see that BLIP-2 outperforms CLIP on all gender splits
of the dataset. From Tab. 2 we also see that models with better zero-shot
classification accuracy on Imagenet [64] tend to have a better overall
resolution accuracy.

Figure 2: Resolution accuracy. Left: RA for different gender pair
combinations. Right: Resolution bias (Gap).  Figure 3: Bias Relative to U.S.
data. We compare the gender accuracy gap to U.S. Labor Force statistics data.
While there are no patterns for CLIP, BLIP-2 shows a bias similar to the real-
world data.

Evaluating Resolution Bias From LABEL:tab:big_table_summary we see that models
tend to exhibit a larger resolution accuracy gap as we go to more “difficult”
subtasks, such as two people with different genders where there is higher
variation and almost _random_ predictions across models. In Fig. 2 we compare
the resolution bias, or accuracy gap, for CLIP and BLIP-2. We see that (i)
CLIP shows a larger accuracy gap, and (ii) CLIP is more biased towards
correctly resolving pronouns for women, whereas BLIP-2 correctly resolves
pronouns for men more often. For further analysis and per-occupation results,
see the Supplementary Materials.

To interpret the results in a real-world context, we compare U.S. Labor Force
Statistics on the proportion of males and females in occupations with
resolution bias in Fig. 3. We measure the correlation in the absolute values
(with Pearson’s R) and correlation in ranked values (with Kendall-Tau), i.e.,
testing for the monotonicity of relationship between model bias and societal
skew. While we see no pattern for the bias of CLIP, the accuracy resolution
gap of BLIP-2 correlates with the U.S. proportions – for occupations with
fewer women, such as “engineer”, the model correctly resolves men more often
than women, and vice versa.

Figure 4: Template Flipping where the subject of the template refers to either
the occupation or the participant.

###  4.2 Ablations

Template Flipping We change the subject of the prompt sentences for images
with two people by reordering the {participant} and {occupation}, e.g., “The
doctor and his patient” becomes “The patient and her doctor”. We compare both
templates in Fig. 4. While we observe similar trends in the two settings, the
resolution accuracy of CLIP is worse when the pronoun refers to the
participant.

Table 3: Neutral Pronoun Resolution. We measure resolution rates into neutral
pronouns. We also show $\Delta_{N}$, which is the difference between the ratio
of male and female images that were resolved with a neutral pronoun.

| Overall | Single Person | Two people
---|---|---|---
| Overall | Same gender | Different gender
Model | Rneutral | $\Delta_{N}$ | Rneutral | $\Delta_{N}$ | Rneutral | $\Delta_{N}$ | Rneutral | $\Delta_{N}$
CLIP [1] | 0.17 | 0.07 | 0.12 | 0.26 | 0.09 | 0.20 | 0.17 | 0.31 | 0.02
BLIP-2 [26] | 0.01 | 0.01 | -0.02 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.01

Neutral Pronoun Resolution Here we attempt to move away from binary gender
classification and introduce a third, neutral pronoun – “their”, which is
always grammatically correct. In Tab. 3 we see that while BLIP-2 almost never
chooses the neutral pronoun, it is selected by CLIP in 17% of all images and
31% of images containing two people of different genders. We also see that for
the more difficult settings, the neutral pronoun is selected more frequently,
with 31% in the “two people, different gender” setting, which corresponds to
almost random chance (33%). Finally, we see that images, where the ground
truth pronoun is male, tend to be resolved as neutral more often.

Figure 5: Retrieval bias by occupation for CLIP. Occupations are ranked by the
Bias@5 values. Positive Bias@K shows more images of men were retrieved for the
given occupation.

###  4.3 Retrieval bias

We evaluate VLMs on retrieval bias in LABEL:tab:big_table_retrieval. We see
that _all_ models have positive Bias@5 and Bias@10 values, which suggests that
images of men in professional settings are retrieved more often than images of
women, despite the candidate images always being gender-balanced. In Fig. 5,
we see that most occupations (15 out of 23) are skewed towards men in both
Bias@5 and Bias@10. However, we do not find a significant relationship between
retrieval bias and the U.S. Labor Force Statistics. For further analysis,
please refer to the Supplementary Materials.

Table 4: Retrieval Bias. We present mean and standard deviation across all
occupations. Positive Bias@K shows more images of men were retrieved.

| Bias@5 | Bias@10 | MaxSkew@5 | MaxSkew@10 | NDKL
---|---|---|---|---|---
Model | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$
CLIP [1] | 0.11 | 0.38 | 0.16 | 0.22 | 0.27 | 0.15 | 0.18 | 0.13 | 0.19 | 0.07
OpenCLIP2B [22] | 0.10 | 0.44 | 0.08 | 0.23 | 0.29 | 0.17 | 0.18 | 0.11 | 0.18 | 0.07
OpenCLIP400M [22] | 0.17 | 0.47 | 0.11 | 0.22 | 0.33 | 0.18 | 0.16 | 0.13 | 0.19 | 0.07
SLIP [23] | 0.06 | 0.52 | 0.00 | 0.24 | 0.32 | 0.21 | 0.17 | 0.12 | 0.19 | 0.09
DeCLIP [24] | 0.11 | 0.40 | 0.15 | 0.26 | 0.28 | 0.16 | 0.20 | 0.14 | 0.17 | 0.07
FILIP [25] | 0.01 | 0.43 | 0.03 | 0.26 | 0.29 | 0.16 | 0.17 | 0.13 | 0.18 | 0.07

##  5 Discussion

From our preliminary benchmarking with VisoGender, we summarise several key
findings:

Models struggle to resolve pronouns in the presence of both genders We found
that all CLIP-like models show close to random performance on gender
resolution when there are people of both genders in the scene. This hints at
insufficient visuo-linguistic capabilities in current VLMs.

Captioning models have a higher accuracy and smaller accuracy gap between
genders We find that captioning models outperform CLIP models on all subtasks.
We attribute this to the way resolution is done in captioning models – the
gender of the subject is extracted using the start of the template and next
token prediction. Meanwhile, CLIP models need to rely on a global cls text
feature, which seems to not capture the nuanced difference between entities in
the sentence.

Resolution and retrieval bias are not in the same direction Across models,
there is not a consistent pattern of bias direction – CLIP models are more
accurate at resolving feminine pronouns, while BLIP models are more accurate
for masculine pronouns. In contrast, we find that all CLIP models are
predominately biased towards retrieving images of men. This highlights a risk
of representative harm in deploying CLIP models in image search systems.

###  5.1 Limitations

Subjective assessment when creating datasets The dataset was compiled by a
diverse team (in terms of gender and heritage) to potentially mitigate
stereotypical perceptions but still inherently relies on subjective
perceptions of inferred binary gender. We acknowledge that the identification
of gender is a complicated societal matter, in that visual presentations may
not reflect self-identity and gender identities occupy a fluid spectrum,
misspecified by binary distinctions. We advocate that this work is extended to
include more genders and avoid erasure of non-binary individuals represented
across occupations [66]. The codebase is designed to be flexible to include
neopronouns in the future.

Stacking biases from the internet We source images from a variety of search
platforms (such as Google Image Search) and image hosting sites. While we
balance included occupations across genders, those that we leave out are not
“missing at random”, that is due to biases that already exist in images on the
internet. We could not find enough images for some occupations, e.g., there
were not 5 images of a female plumber and a male client that met our criteria
for data accessibility and format.

Dataset size The dataset is small compared to those typically used in VLM
training. However, this dataset is only intended for evaluation purposes. It
was beyond our means to partner with a StockImage provider, such as Getty
Images [67], but this could be an avenue in future work to expand dataset size
and to counteract some of the aforementioned image availability biases. Future
work could also augment the dataset with synthetic data from generative VLMs
[12, 68].

##  6 Conclusion

We introduced VisoGender, a novel dataset for benchmarking societal biases in
VLMs for both pronoun resolution and retrieval settings. On some parts of the
benchmark, we demonstrate that current state-of-the-art models perform no
better than random chance, and that they do not perform equally well for
resolving gender of men and women, nor give equal retrieval likelihood to
images of male or female professionals. There is significant headroom for
improvement both in the reasoning abilities of VLMs, and in the gender gap of
their abilities, when it comes to complex scenes with multiple humans. We hope
this work encourages the benchmarking of future VLMs, so the risk of
downstream harms and negative biases can be measured, compared and mitigated.

##  7 Acknowledgements

The authors would like to extend a thank you to following people for their
feedback and insight that was a valuable contribution to the development of
VisoGender: Max Bain, Hugo Berg, Seb Wilkes, Juliana Mota, Daniel Kochin and
Avishkar Bhoopchand. This work has been supported by the Oxford Artificial
Intelligence student society, the EPSRCCentre for Doctoral Training in
Autonomous Intelligent Machines & Systems [EP/S024050/1] (A.S.), the Economic
and Social Research Council Grant for Digital Social Science [ES/P000649/1]
(H.R.K.), and the Clarendon Fund in partnership with the St Cross College
Scholarship, Oxford (F.G.A). For computing resources, the authors are grateful
for support from Jonathan Caton and the Google Cloud and the CURe Programme
under Google Brain Research.

## References

  * [1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [2] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv preprint arXiv:2210.08402, 2022.
  * [3] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  arXiv preprint arXiv:2110.01963, 2021.
  * [4] Solon Barocas and Andrew D Selbst.  Big data’s disparate impact.  California law review, pages 671–732, 2016.
  * [5] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  Science, 356(6334):183–186, 2017.
  * [6] Dirk Hovy and Shannon L Spruit.  The social impact of natural language processing.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591–598, 2016.
  * [7] Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach.  Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1004–1015, 2021.
  * [8] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al.  Ethical and social risks of harm from language models.  arXiv preprint arXiv:2112.04359, 2021.
  * [9] Sandhini Agarwal, Gretchen Krueger, Jack Clark, Alec Radford, Jong Wook Kim, and Miles Brundage.  Evaluating clip: towards characterization of broader capabilities and downstream implications.  arXiv preprint arXiv:2108.02818, 2021.
  * [10] Ching-Yao Chuang, Varun Jampani, Yuanzhen Li, Antonio Torralba, and Stefanie Jegelka.  Debiasing vision-language models via biased prompts.  arXiv preprint arXiv:2302.00070, 2023.
  * [11] Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain.  A prompt array keeps the bias away: Debiasing vision-language models with adversarial learning.  arXiv preprint arXiv:2203.11933, 2022.
  * [12] Brandon Smith, Miguel Farinha, Siobhan Mackenzie Hall, Hannah Rose Kirk, Aleksandar Shtedritski, and Max Bain.  Balancing the picture: Debiasing vision-language datasets with synthetic contrast sets.  arXiv preprint arXiv:2305.15407, 2023.
  * [13] Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jernite.  Stable bias: Analyzing societal representations in diffusion models, 2023\.
  * [14] Kimmo Kärkkäinen and Jungseock Joo.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  In IEEE Winter Conference on Applications of Computer Vision (WACV), 2021.
  * [15] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [16] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer, 2014.
  * [17] Nicole Meister, Dora Zhao, Angelina Wang, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky.  Gender artifacts in visual datasets.  arXiv preprint arXiv:2206.09191, 2022.
  * [18] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5238–5248, 2022.
  * [19] Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme.  Gender bias in coreference resolution.  arXiv preprint arXiv:1804.09301, 2018.
  * [20] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang.  Gender bias in coreference resolution: Evaluation and debiasing methods.  arXiv preprint arXiv:1804.06876, 2018.
  * [21] Hector Levesque, Ernest Davis, and Leora Morgenstern.  The winograd schema challenge.  In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
  * [22] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev.  Reproducible scaling laws for contrastive language-image learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2818–2829, 2023.
  * [23] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 529–544. Springer, 2022.
  * [24] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  arXiv preprint arXiv:2110.05208, 2021.
  * [25] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu.  Filip: fine-grained interactive language-image pre-training.  arXiv preprint arXiv:2111.07783, 2021.
  * [26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [27] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  arXiv preprint arXiv:2205.14100, 2022.
  * [28] Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano.  Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models.  Advances in neural information processing systems, 34:2611–2624, 2021.
  * [29] Kevin Clark and Christopher D Manning.  Improving coreference resolution by learning entity-level distributed representations.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, 2016.
  * [30] Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky.  Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task.  In Proceedings of the fifteenth conference on computational natural language learning: Shared task, pages 28–34, 2011.
  * [31] Anders Björkelund and Jonas Kuhn.  Learning structured perceptrons for coreference resolution with latent antecedents and non-local features.  In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 47–57, 2014.
  * [32] Greg Durrett and Dan Klein.  Easy victories and uphill battles in coreference resolution.  In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1971–1982, 2013.
  * [33] Sam Joshua Wiseman, Alexander Matthew Rush, Stuart Merrill Shieber, and Jason Weston.  Learning anaphoricity and antecedent ranking features for coreference resolution.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2015.
  * [34] Sam Wiseman, Alexander M Rush, and Stuart M Shieber.  Learning global features for coreference resolution.  arXiv preprint arXiv:1604.03035, 2016.
  * [35] Kevin Clark and Christopher D Manning.  Deep reinforcement learning for mention-ranking coreference models.  arXiv preprint arXiv:1609.08667, 2016.
  * [36] Kevin Clark and Christopher D Manning.  Improving coreference resolution by learning entity-level distributed representations.  arXiv preprint arXiv:1606.01323, 2016.
  * [37] Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer.  End-to-end neural coreference resolution.  arXiv preprint arXiv:1707.07045, 2017.
  * [38] Shahar Levy, Koren Lazar, and Gabriel Stanovsky.  Collecting a large-scale gender bias dataset for coreference resolution and machine translation.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2470–2480, 2021.
  * [39] Yang Trista Cao and Hal Daumé III.  Toward gender-inclusive coreference resolution.  arXiv preprint arXiv:1910.13913, 2019.
  * [40] Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge.  Mind the gap: A balanced corpus of gendered ambiguous pronouns.  Transactions of the Association for Computational Linguistics, 6:605–617, 2018.
  * [41] Wangchunshu Zhou, Yan Zeng, Shizhe Diao, and Xinsong Zhang.  Vlue: A multi-task benchmark for evaluating vision-language models.  arXiv preprint arXiv:2205.15237, 2022.
  * [42] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh.  Vqa: Visual question answering.  In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015.
  * [43] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017.
  * [44] Wei-Lun Chao, Hexiang Hu, and Fei Sha.  Being negative but constructively: Lessons learnt from creating better visual question answering datasets.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 431–441, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [45] Sunjae Kwon, Rishabh Garodia, Minhwa Lee, Zhichao Yang, and Hong Yu.  Vision meets definitions: Unsupervised visual word sense disambiguation incorporating gloss information.  arXiv preprint arXiv:2305.01788, 2023.
  * [46] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.  Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901–2910, 2017.
  * [47] Ben Bogin, Shivanshu Gupta, Matt Gardner, and Jonathan Berant.  COVR: A test-bed for visually grounded compositional generalization with real images.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9824–9846, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * [48] Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi.  A corpus of natural language for visual reasoning.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223, 2017.
  * [49] Nan Ding, Sebastian Goodman, Fei Sha, and Radu Soricut.  Understanding image and text simultaneously: a dual vision-language machine comprehension task.  arXiv preprint arXiv:1612.07833, 2016.
  * [50] Arjun Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, and Siva Reddy.  Words aren’t enough, their order matters: On the robustness of grounding visual referring expressions.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6555–6565, Online, July 2020. Association for Computational Linguistics.
  * [51] Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald.  Why is winoground hard? investigating failures in visuolinguistic compositionality.  arXiv preprint arXiv:2211.00768, 2022.
  * [52] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi.  What does clip know about a red circle? visual prompt engineering for vlms.  arXiv preprint arXiv:2304.06712, 2023.
  * [53] Fanjie Kong, Shuai Yuan, Weituo Hao, and Ricardo Henao.  Mitigating test-time bias for fair image retrieval.  arXiv preprint arXiv:2305.19329, 2023.
  * [54] Jialu Wang, Yang Liu, and Xin Eric Wang.  Are gender-neutral queries really gender-neutral? mitigating gender bias in image search.  arXiv preprint arXiv:2109.05433, 2021.
  * [55] Kaylee Burns, Lisa Anne Hendricks, Kate Saenko, Trevor Darrell, and Anna Rohrbach.  Women also snowboard: Overcoming bias in captioning models.  arXiv preprint arXiv:1803.09797, 2018.
  * [56] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  Communications of the ACM, 64(12):86–92, 2021.
  * [57] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 709–727. Springer, 2022.
  * [58] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer.  Lit: Zero-shot transfer with locked-image text tuning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123–18133, 2022.
  * [59] Jennifer Hu and Roger Levy.  Prompt-based methods may underestimate large language models’ linguistic generalizations.  arXiv preprint arXiv:2305.13264, 2023.
  * [60] Jialu Wang, Yang Liu, and Xin Eric Wang.  Are gender-neutral queries really gender-neutral? mitigating gender bias in image search.  arXiv preprint arXiv:2109.05433, 2021.
  * [61] Sahin Cem Geyik, Stuart Ambler, and Krishnaram Kenthapadi.  Fairness-aware ranking in search & recommendation systems with application to linkedin talent search.  In Proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining, pages 2221–2231, 2019.
  * [62] Ke Yang and Julia Stoyanovich.  Measuring fairness in ranked outputs.  In Proceedings of the 29th international conference on scientific and statistical database management, pages 1–6, 2017.
  * [63] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [64] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252, 2015.
  * [65] Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach.  Reclip: A strong zero-shot baseline for referring expression comprehension.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [66] Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff M Phillips, and Kai-Wei Chang.  Harms of gender exclusivity and challenges in non-binary representation in language technologies.  arXiv preprint arXiv:2108.12084, 2021.
  * [67] Getty images.  https://www.gettyimages.co.uk/.
  * [68] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell.  Diversify your vision datasets with automatic diffusion-based augmentation.  arXiv preprint arXiv:2305.16289, 2023.
  * [69] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [70] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela.  Flava: A foundational language and vision alignment model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15638–15650, 2022.
  * [71] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang.  Groupvit: Semantic segmentation emerges from text supervision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134–18144, 2022.

Supplementary Material

###### Contents

  1. 1 Introduction
  2. 2 Related Works
  3. 3 The VisoGender Benchmark
  4. 4 Results
  5. 5 Discussion
  6. 6 Conclusion
  7. 7 Acknowledgements
  8. A Retrieval Bias Metrics
  9. B Fine-Grained Resolution Bias Results
  10. C Ablations on Retrieval Bias Results
  11. D Results for Additional VLMs
  12. E Results Compared to U.S. Labor Force Statistics
  13. F VisoGender Dataset criteria
  14. G VisoGender Occupational Taxonomy
  15. H How To Guide for Benchmarking a VLM
  16. I VisoGender Data Clause
  17. J VisoGender Datasheet

##  Appendix A Retrieval Bias Metrics

Here we define the retrieval bias metrics reported in the paper – Bias@K,
MaxSkew@K, and NDKL.

Bias@K [60] measures the proportions of male and female images in the
retrievals of a search result with a given text query. For an image $I$, we
define a function $g(I)\in\\{0,1\\}$ that denotes the (inferred) groundtruth
gender label of the image:

$g(I)=\begin{cases}1&\text{if the gender of the subject in $I$ is male}\\\
-1&\text{if the gender of the subject in $I$ is female}.\end{cases}$

Given a set of _K_ retrieved images $\mathcal{R}_{K}(q)$ for a query $q$ of a
certain occupation, we define the gender bias metric as:

$\mathrm{Bias@K}(q)=\frac{1}{K}\sum_{I\in\mathcal{R}_{K}(q)}g(I),$

and let $\mathrm{Bias@K}$ be the average of $\mathrm{Bias@K}(q)$ over all
occupations $q\in Q$. $\mathrm{Bias@K}$ is positive if the retrieval model
skews toward male professionals in retrieved images, and negative if female.

Note that Bias@K approximates a form of representational fairness, and is only
applicable to CLIP-like models.

Skew@K [61, 11] measures the difference between the desired proportion of
image attributes in $\mathcal{R}_{k}(q)$ for the query $q$ and the actual
proportion. Let the desired proportion of images containing a professional of
gender $A$ in the set of retrieved images be $p_{d,q,A}\in[0,1]$ and the
actual proportion be $p_{\mathcal{R}(q),q,A}\in[0,1]$. The resulting Skew@K of
$\mathcal{R}(q)$ for a gender $A\in\mathcal{A}$ is:

$\mathrm{Skew}_{A}\mathrm{@K}(q)=\ln{\frac{p_{\mathcal{R}_{K}(q),q,A}}{p_{d,q,A}}},$

where $p_{d,q,A}$ is the proportion of gender $A$ of occupation $q$ in our
dataset, which always equals $0.5$ (gender balanced).

A disadvantage of Skew@K is that it only measures bias with respect to a
single gender at a time and must be aggregated to give a holistic view of the
bias over all attributes. Following [11], we take the maximum value of Skew@K
among all attribute labels $A$ of the retrieved images for the text query $q$:

$\mathrm{MaxSkew@K}(q)=\max_{A\in\mathcal{A}}\mathrm{Skew}_{A}\mathrm{@K}(\mathcal{R}(q)),$

which gives us the “largest unfair advantage” [61] belonging to images within
a given gender. Here, a MaxSkew@K of 0 for the attribute gender and a given
text query $q$ implies that men and women are equally represented in the
retrieved set of $K$ images $\mathcal{R}_{K}(q)$. $\mathrm{MaxSkew@K}$ is the
average of $\mathrm{MaxSkew@K}(q)$ over all occupations $q\in Q$.

NDKL [62, 61] (normalised discounted cumulative KL-divergence) measures the
distance of the retrieval model from a fair distribution, in a weighted
average over $K$. Let $D_{\mathcal{R}_{K}(q)}$ and $D$ denote the binary
distribution of occupational gender over the top $K$ retrieved images and the
desired distribution, respectively. Then the NDKL for occupation $q$ is
defined as:

$\mathrm{NDKL}(q)=\frac{1}{Z}\sum_{i=1}^{K}\frac{1}{\log_{2}(i+1)}D_{\mathrm{KL}}(D_{\mathcal{R}_{K}(q)}\|D),$

where $Z$ is a normalising factor $Z=\sum_{i=1}^{K}\frac{1}{\log_{2}(i+1)}$.

##  Appendix B Fine-Grained Resolution Bias Results

In this section, we present fine-grained resolution bias results by sector
(Fig. 6), by specialisation (Fig. 7) and by occupation (Fig. 8).

#### Bias per sector

Fig. 6 shows that CLIP is better at resolving women for most sectors, except
for the medical sector, where CLIP performs better for men in some
specialisations. BLIP-2, on the other hand, is better at resolving men in
office environments.

#### Bias per specialisation

Fig. 7 shows that the specialisations within a sector show similar trends for
CLIP, but we observe inter-sector variations for BLIP-2. For example, within
the service sector, food and household services are biased towards men, and
fashion and animal services are biased towards women. Similarly, for office
jobs, only legal are biased towards women, and financial, structure, and
general office jobs are biased towards men.

#### Bias per occupation

Fig. 8 shows several occupations with zero accuracy resolution gap – auditor
for CLIP, and doctor and lawyer for BLIP-2. For most occupations, CLIP is
biased toward correctly resolving women, and the largest gaps we observe are
for accountant, architect, engineer, and painter. BLIP-2 is most biased
towards correctly resolving men in the auditor, broker and waiter occupations,
and women in the counselor, paralegal and veterinarian occupations.

Figure 6: Resolution accuracy. Resolution bias (Gap) Sector Figure 7:
Resolution accuracy. Resolution bias (Gap) per Specialisation Figure 8:
Resolution accuracy. Resolution bias (Gap) per Occupation

##  Appendix C Ablations on Retrieval Bias Results

We evaluate retrieval bias when reversing the order in which {occupation} and
{participant} occur. We present results in LABEL:tab:retrieval_flipped. While
most bias measures are of similar magnitude, Bias@5 and Bias@10 for SLIP are
negative – meaning more women are retrieved. Meanwhile, as reported in the
main paper, using the original template, these bias measures were positive.

Table 5: Retrieval Bias with Flipped Templates. We present mean and standard deviation across all occupations, when the pronoun refers to the participant in the occupation-participant pair. Positive Bias@K shows more images of men were retrieved. | Bias@5 | Bias@10 | MaxSkew@5 | MaxSkew@10 | NDKL
---|---|---|---|---|---
Model | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$
CLIP [1] | 0.13 | 0.41 | 0.10 | 0.23 | 0.29 | 0.16 | 0.17 | 0.14 | 0.19 | 0.08
OpenCLIP2B [22] | 0.04 | 0.36 | 0.06 | 0.24 | 0.26 | 0.13 | 0.16 | 0.13 | 0.16 | 0.05
OpenCLIP400M [22] | 0.23 | 0.40 | 0.10 | 0.25 | 0.30 | 0.18 | 0.16 | 0.16 | 0.18 | 0.08
SLIP [23] | -0.18 | 0.49 | -0.06 | 0.26 | 0.35 | 0.18 | 0.17 | 0.14 | 0.21 | 0.09
DeCLIP [24] | 0.15 | 0.49 | 0.13 | 0.27 | 0.35 | 0.16 | 0.20 | 0.14 | 0.19 | 0.08
FILIP [25] | -0.01 | 0.38 | 0.03 | 0.29 | 0.27 | 0.14 | 0.19 | 0.15 | 0.17 | 0.06

##  Appendix D Results for Additional VLMs

| Overall | Single Person | Two people | ZS Imagenet
---|---|---|---|---
| Overall | Same gender | Diff. gender
Model | RAavg | $\Delta$ | RAavg | $\Delta$ | RAavg | $\Delta$ | RAavg | $\Delta$
ALIGN [69] | 0.52 | 0.78 | -0.33 | 0.26 | -0.02 | 0.42 | -0.01 | 0.10 | 0.03 | 85.5
FLAVA [70] | 0.73 | 0.91 | -0.06 | 0.54 | -0.29 | 0.75 | -0.22 | 0.34 | -0.35 | –
GroupViT [71] | 0.64 | 0.79 | -0.16 | 0.50 | -0.09 | 0.64 | -0.11 | 0.34 | -0.07 | –
(a) Resolution Bias. We present resolution accuracy averaged for male and
female; resolution accuracy gap $\Delta$ as defined in the main paper and
zero-shot classification accuracy on Imagenet [64].

| Bias@5 | Bias@10 | MaxSkew@5 | MaxSkew@10 | NDKL
---|---|---|---|---|---
Model | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$ | Mean | $\sigma$
ALIGN [69] | 0.10 | 0.42 | 0.06 | 0.30 | 0.29 | 0.16 | 0.22 | 0.14 | 0.18 | 0.06
FLAVA [70] | 0.10 | 0.47 | 0.11 | 0.20 | 0.31 | 0.18 | 0.16 | 0.11 | 0.20 | 0.07
GROUPVIT [71] | 0.18 | 0.37 | 0.16 | 0.25 | 0.28 | 0.16 | 0.18 | 0.17 | 0.18 | 0.07
(b) Retrieval Bias. We present mean and standard deviation across all
occupations for two-person images.

Table 6: Additional bias results.

We repeat the analysis in the main paper on additional VLMs – ALIGN [69],
FLAVA [70] and GroupViT [71]. These models, similar to the other CLIP models
evaluated, are biased towards resolving women images more accurately than men,
i.e., have a negative gender accuracy gap (see Tab. 6(a)). The exception is
ALIGN in the two-person case, with a slightly positive in the gender accuracy
gap. The performance is significantly worse in the two-person case, with worst
performance for resolution accuracy coming from the setting of different
gender subjects. FLAVA has highest overall resolution accuracy, along with the
biggest gender gap scores relative to the other two models. Similar to CLIP
models evaluated, we find that the models have a bias toward retrieving images
of men in occupational settings (see Tab. 6(b)).

##  Appendix E Results Compared to U.S. Labor Force Statistics

Table 7: Pearson’s correlation coefficiants and Kendall $\tau$ when the resolution accuracy results compared to the U.S. Labor Force Statistics (Male Proportion) |  | Pearson’s r | Kendall $\tau$
---|---|---|---
Occupation | Participant | CLIP | BLIP-2 | CLIP | BLIP-2
Same pair | M | M | -0.45 | 0.40 | -0.31 | 0.36
W | W | -0.23 | 0.12 | -0.24 | 0.05
Diff. pair | M | W | -0.28 | 0.42 | -0.28 | 0.22
W | M | -0.05 | -0.53 | -0.06 | -0.42

The U.S. Labor Force Statistics publishes an annual report of population
demographics across occupations. We record the male/female proportion in the
U.S. labour force by matching each VisoGender occupation to either a single
U.S. occupation category or by taking a mean across any relevant categories.
For VisoGender jobs that cover multiple U.S. categories, we also report the
standard deviation. The statisitcs used are predominantly from 2022, with the
exception of medical - specialist occupation which is taken from 2020. This
mapping and accompanying metadata is available in our GitHub repository:
https://github.com/oxai/visogender.

###  E.1 Evaluating Resolution Bias Relative to U.S. Skew

In Fig. 9 and Fig. 10 we compare the resolution accuracy of CLIP and BLIP-2
per occupation to the proportions of U.S. men working in that occupation. The
correlation coefficients for each occupation-participant pair is summarised in
Tab. 7. There is no correlation between the resolution accuracy scores and the
proportions of the men in the U.S. Labor Force occupation statistics, with the
exception of CLIP same pair (M-M) and BLIP-2 different pair (W-M) showing some
correlation, that is not significant.

Figure 9: Resolution accuracy. CLIP: Mapping of Resolution Accuracy for the
same and diff. gender pairs to the U.S. Labor Force Statistics (Male
Proportion) Figure 10: Resolution accuracy. BLIP-2: Mapping of Resolution
Accuracy for the same and diff. gender pairs to the U.S. Labor Force
Statistics (Male Proportion) Figure 11: Retrieval bias. Comparision of the
$\rm Bias@10$ scores to the to the U.S. Labor Force Statistics (Male
Proportion).

###  E.2 Evaluating Retrieval Bias Relative to U.S. Skew

In Fig. 11 we compare the retrieval bias of CLIP to the U.S. male proportions
for each occupation. We find that for CLIP, OpenCLIP-400M, and OpenCLIP-2B,
the $\mathrm{Bias@10}$ metric (skew toward men) is slightly positively
correlated with the proportion of male in each occupation in the U.S., in both
Pearson’s $r$ and Kendall’s $\tau$. Although this correlation is not
significant, it reflects potential bias in the methods or dataset of CLIP and
OpenCLIP that warrants future investigation.

##  Appendix F VisoGender Dataset criteria

Data collection was carried out by the authors of the paper from March to May
2023 on a variety of image databases and search providers, such as Pexels and
Google Image Search. We followed a set of guidelines to specify exclusion and
inclusion criteria:

  1. 1.

There is either only one or two people in the image, depending on the task,

  2. 2.

The image does not contain any children,

  3. 3.

The image is a photograph (in colour or black and white) but not e.g., a
cartoon,

  4. 4.

The image does not contain stock photo watermarks,

  5. 5.

The image can be accessed by a URL,

  6. 6.

The image is under a Creative Commons licence, and does not fall under a
specific clause disallowing its use for machine learning purposes.

##  Appendix G VisoGender Occupational Taxonomy

The taxonomy, as indicated in Fig. 12, is characterised according to a three
tier system: Sector (the general field), Specialisation (subcategories within
the Sector) and Occupation (job categories nested within the Specialisation).
Tab. 8 outlines the number of images per Sector and Specialisation. category.

Figure 12: Taxonomy of the VisoGender dataset based on Winogender occupations [19] Table 8: Dataset statisctics. Number of images per sector and specialisation. |  | Single-person | Two-person
---|---|---|---
|  | {Occuptation}–{Object} | {Occuptation}–{Participant}
Sector | Education | 20 | 40
Medical | 40 | 80
Office | 90 | 180
Retail | 20 | 40
Service | 60 | 120
Specialisation | General courses | 10 | 20
Institutional | 10 | 20
Mental heath | 10 | 20
Hospital worker | 30 | 60
Legal | 20 | 40
Financial | 40 | 80
General office | 10 | 20
Structure | 20 | 40
Customer facing | 20 | 40
Food service | 30 | 60
Fashion | 10 | 20
Animal | 10 | 20
Household | 10 | 20

##  Appendix H How To Guide for Benchmarking a VLM

A VLM can be evaluated on our benchmark using the provided code (
https://github.com/oxai/visogender). The benchmark scores are saved in json
files, as depicted below.

To perform well on VisoGender, the scores should be optimised as follows:

  * •

Resolution bias:

    * –

resolution accuracies should be as close to 100% as possible indicating a high
capability in performing gender coreference resolution

    * –

the gender gap score should be as close to zero as possible to ensure the
model is not biased towards either gender

  * •

Retrieval bias: all scores should be as close to zero as possible to
demonstrate a model is not biased towards either gender.

⬇

{

"visogender_blip2_results": {

"metadata": {"experiment_desc": "CAPTIONING", "model_name": "blipv2"},

"resolution_bias": {

"all_images": {"overall_accuracy": 0.84},

"single_person_images": {"RA_avg": 0.92, "gender_gap": -0.09},

"two_person_images": {"RA_avg": 0.76, "gender_gap": 0.07},

"two_person_images_same_gender": {"RA_avg": 0.93, "gender_gap": 0.06},

"two_person_images_diff_gender": {"RA_avg": 0.60,"gender_gap": 0.08}

}

}

}

⬇

{

"visogender_clip_results": {

"metadata": {"experiment_desc": "CLIP","model_name": "clip"},

"resolution_bias": {

"all_images": {"overall_accuracy": 0.75},

"single_person_images": {"RA_avg": 0.92, "gender_gap": -0.14},

"two_person_images": {"RA_avg": 0.57, "gender_gap": -0.27},

"two_person_images_same_gender": {"RA_avg": 0.79, "gender_gap": -0.18},

"two_person_images_diff_gender": {"RA_avg": 0.36, "gender_gap": -0.35}

},

"retrieval_bias": {

"bias@5": {"mean": 0.13, "sigma": 0.35},

"bias@10": {"mean": 0.16, "sigma": 0.22},

"maxskew@5": {"mean": 0.25, "sigma": 0.15},

"maxskew@10": {"mean": 0.18, "sigma": 0.13},

"NDKL": {"mean": 0.18, "sigma": 0.07}

}

}

}

##  Appendix I VisoGender Data Clause

The Visogender dataset only contains URLs referencing images that (at the time
of curation) are under Creative Commons and/or royalty free licences allowing
for their use and distribution. No images are stored directly.

The URLs are curated manually, and at the time of collection in April - May
2023, they did not point to any images containing harmful or disturbing
imagery, nor any images containing children.

The associated metadata per image is provided by manual labelling.

The authors confirm that, to the best of their knowledge, they are using all
intellectual property in accordance with their licences, and the use of the
data does not violate any rights. The authors do not take responsibility for
any licences that change with time.

This dataset will be maintained by the authors. If you would like to
contribute to the dataset, please contact the authors. In the event that there
are any issues with the dataset, or any specific links to images, or
associated images, please contact the authors and offending information will
be removed immediately.

##  Appendix J VisoGender Datasheet

We present a Datasheet for the VisoGender dataset [56] which is available on
GitHub: https://github.com/oxai/visogender/. The information in the datasheet
is up to date as of June 2023. Any amendements to the datasheet subsequent to
this version will be made on GitHub.

###  J.1 Motivation

#### For what purpose was the dataset created?

The dataset was created as a benchmark to evaluate vision-language models
(VLM). Specifically, it is designed to measure a model’s abilities and biases
for gender pronoun coreference resolution and image retrieval in an
occupational setting.

#### Who created the dataset (e.g., which team, research group) and on behalf
of which entity (e.g., company, institution, organisation)?

The authors of the paper, as a part of the Oxford Artificial Intelligence
Society Research Labs.

#### Who funded the creation of the dataset?

N/A

#### Any other comments?

None

###  J.2 Composition

#### What do the instances that comprise the dataset represent (e.g.,
documents, photos, people, countries)?

The dataset comprises images of people in a professional setting. There is at
least one person in the image which represents the professional person as
referred to by their occupation.

#### How many instances are there in total (of each type, if appropriate)?

Single person: 230 | Two people in the image: 460 | Total number of images: 690

#### Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set?

This images were curated from Google image search (Creative Commons Licence)
and Pexels images which are royalty free. In most cases, we took the first 5
images per gender-pair and occupation meeting the criteria and guidelines. Our
restrictive licensing criteria and resource constraints (i.e., limiting
partnerships with StockPhoto providers) meant that the set of total images to
chose from was limited. In some cases, there were insufficient images that met
the criteria. However, compared to all images on the internet of individuals
in professional settings, this is a non-random subsample.

#### What data does each instance consist of?

Each image contains one person, as referred to by their occupation, or, in the
case of two people: one person referred to by their occupation and another
referred to by a participation noun, relative to the interaction with the
professional. In the case of one person in the image, they are accompanied by
an object.

#### Is there a label or target associated with each instance?

Yes, the labels refer to the professional person and their participant (if
applicable). There is an object label as well, in the case of a single person
being present. There are also labels related to the occupation, based on the
taxonomy.

#### Is any information missing from individual instances?

No.

#### Are relationships between individual instances made explicit (e.g.,
users’ movie ratings, social network links)?

N/A

#### Are there recommended data splits (e.g., training,
development/validation, testing)?

This is purely an evaluation set. It is not intended for training purposes.

#### Are there any errors, sources of noise, or redundancies in the dataset?

At the time of initial release, there are no errors, redundancies or sources
of noises to the best of the authors’ knowledge, based on internal review.

#### Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?

The dataset contains URLs that reference images that (at the time of curation)
are under Creative Commons and/or royalty free licences that allow for their
use and distribution.

#### Does the dataset contain data that might be considered confidential
(e.g., data that is protected by legal privilege or by doctor–patient
confidentiality, data that includes the content of individuals’ non-public
communications)?

The data collected (URLs) does not contain any personal, or identifiable
information and the associated metadata is provided by manual labelling, and
is based on Google and Pexels image search query tags.

#### Does the dataset contain data that, if viewed directly, might be
offensive, insulting, threatening, or might otherwise cause anxiety?

The URLs are curated manually, and at the time of collection, they did not
point to any images containing harmful or disturbing imagery, nor any images
containing children. Any URL endpoints that change to become problematic or
are determined to infringe on privacy will be removed immediately.

#### Does the dataset identify any subpopulations (e.g., by age, gender)?

The images are of people with identifiable features that can infer such
information. However, no identifiable or personal information is shared with
the dataset.

#### Is it possible to identify individuals (i.e., one or more natural
persons), either directly or indirectly (i.e., in combination with other data)
from the dataset?

The images depict real people. However, the authors are not aware of any
metadata associated with these public images that can be used to identify the
individuals.

#### Does the dataset contain data that might be considered sensitive in any
way (e.g., data that reveals race or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)?

Only race can potentially be inferred from the images. To the best of the
authors’ knowledge, these public images were not collected alongside this
sensitive information.

#### Any other comments?

None.

###  J.3 Collection Process

#### How was the data associated with each instance acquired?

Search queries such as “a photo of a {gendered adjective} {occupation} and a
{gendered adjective} {participant}” or “a photo of a {gendered adjective}
{occupation} and a {object}” were used on Google image search (with the
Creative Commons licence filter) and Pexels image search.

#### What mechanisms or procedures were used to collect the data (e.g.,
hardware apparatuses or sensors, manual human curation, software programs,
software APIs)?

Image search queries, and manual labelling using a shared Google Sheets
document.

#### If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?

N/A

#### Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

The authors of the paper were responsible for the data collection, and no
compensation was provided. No external parties were involved.

#### Over what timeframe was the data collected?

Two months in 2023 (March to May).

#### Were any ethical review processes conducted (e.g., by an institutional
review board)?

No, as the images are under open licences, and no human subjects were
recruited for the purpose of the study.

#### Did you collect the data from the individuals in question directly, or
obtain it via third parties or other sources (e.g., websites)?

Third party websites, with appropriate licences allowing us to use the images.

#### Were the individuals in question notified about the data collection?

No, as we are collating images collated by third parties. However it is
assumed, based on the associated licences, that the individuals depicted were
made aware that their images were taken for public access.

#### Did the individuals in question consent to the collection and use of
their data?

Please see the previous question.

#### If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses?

N/A

#### Has an analysis of the potential impact of the dataset and its use on
data subjects (e.g., a data protection impact analysis) been conducted?

No.

#### Any other comments?

The authors confirm that, to the best of their knowledge, they are using all
intellectual property in accordance with their licences, and the use of the
data does not violate any rights. The authors do not take responsibility for
any licences that change with time.

###  J.4 Preprocessing/cleaning/labelling

#### Was any preprocessing/cleaning/labelling of the data done (e.g.,
discretisation or bucketing, tokenization, part-of-speech tagging, SIFT
feature extraction, removal of instances, processing of missing values)?

Manual labelling of the data was conducted at the time of the data collection.
Images were also sorted into an occupational taxonomy we created.

#### Was the “raw” data saved in addition to the preprocessed/cleaned/labelled
data (e.g., to support unanticipated future uses)?

The only data that was collected is present in the open version of VisoGender.

#### Is the software that was used to preprocess/clean/label the data
available?

N/A

#### Any other comments?

No.

###  J.5 Uses

#### Has the dataset been used for any tasks already?

The dataset is completely novel and, as of June 2023, has only been used in
the original VisoGender paper

#### Is there a repository that links to any or all papers or systems that use
the dataset?

Yes.

#### What (other) tasks could the dataset be used for?

The dataset should only be used as an evaluation dataset for binary gender
bias vision-language models. It is not advised to use this dataset for
training purposes, as the binary nature of the labels risks erasure of non-
binary people represented in these professional roles.

#### Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labelled that might impact future uses?

We only collect URLs which link to images hosted on third party sites. There
is the potential that these links become deprecated, or image licences change
over time. The authors do not take responsibility for any changes to image
links, or their associated licences, but will immediately remove any
problematic images in the event these are identified.

#### Are there tasks for which the dataset should not be used?

This should not be used for the training of models. It is solely intended as
an evaluation dataset.

#### Any other comments?

None.

###  J.6 Distribution

#### Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organisation) on behalf of which the dataset was
created?

This dataset is publicly available and it is encouraged that developers of
VLMs use it to assess their models’ propensities for gender bias.

#### How will the dataset be distributed (e.g., tarball on website, API,
GitHub)?

GitHub.

#### When will the dataset be distributed?

At the time of the paper being published in 2023.

#### Will the dataset be distributed under a copyright or other intellectual
property (IP) licence, and/or under applicable terms of use (ToU)?

These URLs are distributed based on their royalty free/ Creative Commons
licences that the images occupy at the time of curation. The dataset is open
source, but we request that the dataset is cited in any subsequent work. The
citation can be found alongside the data on our GitHub repository:
https://github.com/oxai/visogender.

#### Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?

No, not at the time of data curation (March - May 2023)

#### Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances?

No.

#### Any other comments?

None.

###  J.7 Maintenance

#### Who will be supporting/hosting/maintaining the dataset?

The authors of the paper and the Oxford Artificial Intelligence Society.

#### How can the owner/curator/manager of the dataset be contacted (e.g.,
email address)?

The first author (Siobhan Mackenzie Hall) can be contacted by email
(siobhan.hall@nds.ox.ac.uk), or via a GitHub Issue:
https://github.com/oxai/visogender/issues.

#### Is there an erratum?

N/A at the time of publishing.

#### Will the dataset be updated (e.g., to correct labelling errors, add new
instances, delete instances)?

Yes. We encourage anyone that finds fault to contact the authors to amend or
remove any problematic URLs.

#### If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were the
individuals in question told that their data would be retained for a fixed
period of time and then deleted)?

N/A as the images are under licences.

#### Will older versions of the dataset continue to be
supported/hosted/maintained?

Yes, with the exception of any URLs that change with time and potentially link
to problematic images.

#### If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so?

Yes. We encourage anyone looking to expand the dataset to contact the authors
to discuss this development.

#### Any other comments?

None.