# SoUnD Framework: Analyzing (So)cial Representation in (Un)structured (D)ata

Mark Díaz
Google Research
<EMAIL_ADDRESS>
Sunipa Dev
Google Research
<EMAIL_ADDRESS>
Emily Reif
Google Research
<EMAIL_ADDRESS>
Emily Denton
Google Research
<EMAIL_ADDRESS>
Vinodkumar Prabhakaran
Google Research
<EMAIL_ADDRESS>

###### Abstract

The unstructured nature of data used in foundation model development is a
challenge to systematic analyses for making data use and documentation
decisions. From a Responsible AI perspective, these decisions often rely upon
understanding how people are represented in data. We propose a framework
designed to guide analysis of human representation in unstructured data and
identify downstream risks. We apply the framework in two toy examples using
the Common Crawl web text corpus (C4) and LAION-400M. We also propose a set of
hypothetical action steps in service of dataset use, development, and
documentation.

##  1 Introduction

Data is recognized as a core underlying factor contributing to machine
learning model behaviours that are potentially unfair or harmful to humans
[60]. Insights gleaned from systematic analysis of datasets can identify
potential harms and inform interventions to mitigate risk. Principled analysis
of the data underpinning pre-trained foundation models is particularly salient
given the increasing reach of such models and their use by researchers and
developers who lack the resources to develop computationally-intensive models
[14, 36].

At the same time, the large, unstructured nature of datasets that underpin
foundation models poses significant challenges for conducting analyses
required to make development, documentation, and use decisions. The open-ended
potential for downstream use, means that risks are wide-ranging and sometimes
lack clear methods of evaluation [79], complicating systematic study. Prior
systematic fairness audits and auditing tools have often focused on labeled
datasets and utilized aggregated and disaggregated analyses to identify class
imbalances (e.g., [67, 47, 50, 30]). Despite increased scrutiny of large
unstructured datasets [12, 28], methods of analysis remain less robust and
less systematic relative to labeled datasets, in part because labels provide a
crucial pointer to dataset features to evaluate for fairness and bias
concerns.

Our work closes this gap by contributing a conceptual framework to facilitate
standardized analysis of unstructured data, in service of responsible data use
and dataset development, use, and documentation. The framework (shown in
Appendix A) focuses on how people are represented in data, including the range
of data features that indicate social identity and influence the
representation of different social and cultural groups– from identity terms
that explicitly name social groups to recurring image objects implicitly
associated with those groups. Analyses are grouped according to who is in the
data, what is in the data, as well as associations between them. The
framework’s focus on whether and how people are depicted means that the core
analytical questions are not modality-specific and are extensible to new
modalities and combinations as they emerge. The framework is designed to
integrate into RAI workflows to support decision making for mitigation steps
(e.g., data filtering, rebalancing) for a range of goals, such as new dataset
development, existing dataset adaptation, and benchmark development.

##  2 Background

###  2.1 Dataset Transparency and Documentation

Transparency is a core focus in RAI, with a growing body of scholarship
focusing on increasing transparency of AI systems and datasets for a variety
of stakeholders. These range from developers who may build on pre-trained
models to system end-users who may be subject to algorithmic decision making
[53, 78]. At the dataset level, transparency serves to highlight critical
information both about the contents of a dataset as well as the processes that
underpin how a dataset was created. To this end, a range of work brings
structured approaches to documenting both dataset content and development
processes [10, 33, 28, 26, 44, 65, 75, 61]. However as massive, unstructured
datasets increasingly become the norm in ML development, structured frameworks
are needed to help summarize key characteristics of the data they capture.
Dodge et al. [28] offer an expansive audit of C4 [62], which inspires a more
structured approach for disentangling the contents of web-crawled data that
feature heavily in ML datasets. Our work aims to standardize approaches to
support existing transparency and documentation efforts by enabling the
identification and communication of potential social risks associated with
data.

###  2.2 Dataset Audits

Datasets underpinning ML training and testing have been at the center of a
range of ethical controversies relating to privacy, consent, unfair system
performance, representational harms, and harmful applications [60]. Against
this backdrop, prominent ML datasets have been subject to close scrutiny, with
empirical examinations and audits uncovering a range of problematic content
that itself is a cause of harm (e.g., copyright violations; representational
harms such as misgendering) or that can lead to downstream harms. For example,
both image and text datasets have been shown to contain co-occurrence
statistics that mirror harmful social biases and stereotypes [32, 39]; image
datasets have been found to include problematic sexual imagery, including
depictions of sexual violence and non-consensual sexual content, and racial
and ethnic slurs within image labels and captions [11, 12]; text datasets have
been found to contain biased and harmful sentiments towards marginalized
identity groups [43, 28] and to exclude perspectives from marginalized groups
[28]. Dataset audits can close documentation gaps [28], be used to make data
filtering or re-balancing decisions [66], and, in some extreme cases, lead to
the deprecation of datasets, such as MegaFace [48] and Tiny Images [76].
Organizing prior individual audits, we present a principled framework that
supports dataset auditing to both shape dataset development decisions, as well
as flag downstream model evaluations to prioritize.

###  2.3 Standardizing Responsible AI Workflows

Evaluating datasets in a structured way and effectively communicating results
to various stakeholders remains an important challenge for RAI. As Sambasivan
et al. [69] show, data work often takes a backseat to work focused on
developing state of the art models and algorithms. In addition to lower
incentive to engage in data work as a component of robust evaluation, current
approaches to data documentation are “largely ad hoc and myopic in nature”
[38] and practitioners face difficulty in understanding why documentation is
needed, how best to document, and, ultimately, what to document [19]. A range
of development toolkits and checklists have been proposed to address these
challenges, including documentation frameworks such as Data and Model Cards
[33, 58, 61], internal auditing frameworks [64], analysis tooling such as Know
Your Data111https://knowyourdata.withgoogle.com/ and WIMBD [29], and impact
assessment frameworks [72]. RAI audits in particular require support to
determine what to measure and how to measure it to avoid risks that can
compound through development [69]. Mitchell et al. [57] give a high-level
framework for measuring large, unstructured datasets and we extend this by
configuring our framework around social representation and demonstrate how the
results of an audit might be used to distill dataset decisions.

##  3 Framework

In this section, we introduce a framework for systematic evaluation, anchoring
on risk and harm associated with representations of humans in data. The
framework supports analyses for a variety of goals ranging from dataset
development to third-party audits. Our framework also identifies a set of
components that guide the operationalization of each analysis and the
interpretation of their results.

###  3.1 Framework analyses

Figure 1 demonstrates the framework’s conceptual structure. We organize the
framework around high-level questions about human-centered considerations in
data: namely, Who is in the data, What is in the data, and How are the two
associated? This structure also allows the analyses to focus on data questions
at different levels of complexity with respect to corresponding downstream
harms, as well as to prevent an over-focus on optimizing isolated analyses or
metrics.

Our framework is designed to be general-purpose and extensible, thus it is not
exhaustive of every single possible analysis within each section. While we use
text, image, and image-text datasets as references for developing and
describing the framework, it can be adapted to other modalities with
appropriate changes. For instance, the dialect analysis in text could be
adjusted to include elements of (or complemented with) analysis of accent in
speech data. Analyses can also be modified, added, or removed as the field’s
collective sociotechnical understanding about relevant social biases evolve
over time, while preserving the overall framework structure. For example,
salient social identity term lists may iteratively change as best practices
respond to social shifts, or as global socio-cultural contexts are
increasingly integrated into RAI considerations. Analyses can also be updated
alongside our understanding of salient social risks, the human social
characteristics they are connected to, and our technical means of analyzing
them. However, the motivating questions remain stable.

Figure 1: Framework conceptual structure. The combination of Human Factor and
Content Factor analyses provided disaggregated associations. Analyses
documentation and results are on the right.

####  3.1.1 Who is in the data?

In asking who is in the data, we consider several human factors of data that
include measuring the presence of people in data along with social
characteristics.

Presence of People: The analyses of people’s presence specifically tally
whether individuals or identifying information appear in data. This includes
calculations of personally-identifiable information and face or person
detection.

Extending to new modalities, the analyses implicitly ask which data
characteristics can indicate the presence of a person, such as faces or bodies
in visual data or voice in audio data. Results guide more focused, follow up
analyses that assess depictions of social groups.

Social Characteristics: The analyses of social characteristics in data center
on data characteristics that are often associated with social identity and may
be used as proxies for social identity. Some proxies appear directly in data,
such as pronouns, while others, such as perceived age or gender expression in
images, must be inferred, frequently using predictive methods (e.g., Lanitis
et al. [51]). These include analyses of dialect, linguistic style, skin tone,
and voice pitch. Social characteristic analyses provides insight into the
over- and under-representation of specific social groups, which has been
associated with disparities in performance [80, 16] and general problems for
class prediction [46]. Because these characteristics are social in nature,
their measurement must be localized. For example, social identity terms and
gender presentation vary across social and cultural contexts.

####  3.1.2 What is in the data?

The second high-level grouping of analyses focuses on content that may
heighten the sensitivity of human representation as a whole.

Content: This group of analyses is focused on content characteristics that
relate to harmful or undesirable outcomes that are independent of specific
people or social groups. Analyses include calculating the distribution of
topics in text, as well as sexual content in images. Topic distribution
provides a bird’s-eye view of the composition of the data and can give an
indication of sexually explicit or sensitive topics contained in a dataset.
Topic distributions can give clues to subtle downstream biases. For example,
models trained primarily on news data have been shown to exhibit biases
against particular country names and professions [42].

Provenance: Data provenance can indicate what is likely to be contained in the
data. This includes values, norms, and perspectives in data ascertained
through metadata, such as the geographic distribution of sources and their
publication dates. For example, source URL domains point to the range of
content represented in web-scraped data, which offers insight into document
content, such as linguistic and cultural content, as well as the prevalence of
machine-generated text [28]. The geographic, cultural, and social
representation in data can have implications for downstream models. For
example, image classifiers trained on datasets sourced predominantly from
western countries have lower rates of accuracy when applied to images from
non-western countries [74]. Data recency can have particular impacts on models
supporting low-resource languages, which can disproportionately rely on
religious or historical texts due to data scarcity (e.g., Ahmadi and Masoud
[3]).

####  3.1.3 Human $\times$ Content Associations

The final section of the focuses on associations between human and content
factors in data. Each standalone section of the framework is valuable,
however, associations reveal how people are depicted. Associations
disaggregate analyses within and across modalities, such as social identity
terms and topics in text (for language datasets) or occurrences between
objects detected in images and identity terms in associated text (for
multimodal datasets). Associations can reveal stereotype-aligned correlations,
which can amplify the stereotypes and propagate exclusionary norms [24, 79,
81, 39].

While highly specific combinations of analyses can be run (e.g., an evaluation
of queer depictions in Spanish-language medical literature from a specific
year), the structure of the framework facilitates analyses beginning with the
most general question (i.e., are people depicted?) followed by more specific
inquiries to provide a tractable entry point for RAI analyses.

###  3.2 Framework Components

Next, we outline additional framework components that guide analysis results
reporting and general analysis planning. The Output and Action fields are
provided to capture the results of a given analysis and any mitigation actions
decided in response. The Taking Action section discusses in more depth the
process of making mitigation decisions. In addition to a research-backed
motivation related to downstream risks, each analysis includes additional
fields to support planning:

Analysis Object indicates whether an analysis is calculated on data directly
(i.e., tokens in text data; source urls) or if an analysis applies to an
inference produced by an intermediate classifier (e.g., inferred document
topic; predicted age of person in an image). This distinction highlights which
analyses are dependent on predictive models and therefore susceptible to
biases that those predictive models may themselves exhibit. The distinction
between “Image” and “Inferred image signals” is particularly important since
few of the analyses in the framework are applied to image data directly.

Effort indicates estimated time and cost of an analysis based on current
techniques and available tooling, which, in turn, reflect bias toward use for
English language and Western data. In non-Western contexts, effort is likely
higher for technical implementation as well as making determinations about
which identity groups should be prioritized.

Dependencies indicates intermediate resources needed to conduct an analysis,
such as identity term lists or classifiers which produce inferred signals.
While the framework does not dictate a required implementation for any
analysis, we point to example classifiers and term lists that may be used.

##  4 Taking Action

The framework is not meant to be exhaustively implemented for every use case,
since all possible Association analyses would produce an intractably large
number of results. Moreover, which mitigation actions to take depends highly
on context: the downstream effects of dataset filtering and rebalancing are,
in many cases, still an open question, and there are a range of other actions
that may be applicable. Finally, while this framework can be used to uncover
previously unanticipated biases, we expect that in practice, practitioners
will have high-level aims informed by institutional policies and previously
documented data biases. An exhaustive review of mitigation actions is beyond
the scope of this work, however we describe key considerations that inform the
actions a user should take. We also include a selection of guiding questions
and considerations at the start of the full framework.

Important considerations are 1) dataset purpose (e.g., training or evaluation
data) 2) analysis goals (e.g., auditing a third-party dataset or developing a
new evaluation benchmark), and 3) development steps at which interventions can
be applied (e.g., data collection, model evaluation, documentation).

Dataset Purpose: The framework can be applied to a range of dataset types
including pre-training or fine-tuning datasets. It can also be used for
understanding and evaluating model-generated data. Suitable framework actions
depend on the purpose of the dataset. For example, when analyzing pre-training
data, it may be unclear how changes to data distributions will impact model
performance, potentially making other mitigations more desirable. In contrast,
data used for benchmark development stands to be used as a repeated measure of
model robustness and performance. Thus, actions that might require additional
costs or resources may be more easily justified to meet evaluation goals.

Analysis Goals
---
Dataset Development |  Developing a dataset for training or evaluation through new data collection and/or adaptation of existing datasets
Use Decisions |  Making decisions regarding appropriate use of a dataset, whether for training or evaluative purposes
Model Understanding |  Investigating potential roots of or explanations for model behavior
Auditing |  Auditing a dataset, whether third-party or internal to a development project, for example to fill documentation gaps, ensure legal or institutional compliance, or to foster greater public awareness
Table 1: A non-exhaustive list of data analysis goals.

Analysis Goals: A range of goals can motivate framework use– each of which
brings attention to different actions. Table 1 lists common goals of dataset
analyses. For example, developing a new dataset from web-scraped data raises
potential decisions to collect additional data or adjust filtering criteria in
the data collection process. In contrast, conducting an audit of a third-party
dataset for compliance purposes brings focus to documentation and data use
decisions.

Development Phase |  Actions |  Description
---|---|---
Data Collection/ Processing |  Addition |  Rebalancing distributions across an entire dataset or within specified categories with additional (potentially synthetic) data
Removal |  Filtering data to remove unwanted content
Augmentation |  Augmenting data, such as through data tagging [6] to allow a model to learn undesirable content while controlling its production downstream
Flagging |  Flagging analysis results for further downstream evaluation or documentation
Non-Use |  Not using the dataset, for example if applying analyses to different candidate datasets to decide which to use
Model Evaluation |  Addt’l Benchmarking |  Selection of additional evaluation benchmarks
Benchmark Creation |  Development of novel benchmarks to evaluate identified concerns
Documentation |  Warning |  Documentation of general or use case-specific limitations
Non-Use |  Documentation of cases where the dataset should not be used at all
Packaging and Release |  Licensing |  Development of appropriate licensing and terms of use specifications
Access |  Development of limited access policies
Table 2: A non-exhaustive list of actions that may be taken to address social
risks identified in data.

Development Phase: Each development phase affords different actions to address
data concerns. Table 2 shows common actions by development phase. For example,
during data collection, toxic content biased across social identity groups
might be addressed by modifying the dataset or by adjusting model evaluation
planning. Alternatively, documentation can flag concerns for public
consumption. Moreover, concerns may be addressed through explicit dataset
release decisions.

The decision to pursue an action such as the ones listed above will depend on
analysis goals, available resource play a key role in determining the
mitigation actions that are available. Direct action on a dataset may not be
possible, for example if there are cost constraints or if data sources and
filtering techniques used to develop a third-party dataset are not clearly
defined or known.

##  5 Applying the Framework

In order to showcase how the framework can guide evaluations, we provide two
toy examples applying it to C4 [62] and LAION-400M [73]– two large,
unstructured datasets available under a CC-BY 4.0 license. We apply the
analyses from the standpoint of an individual or team seeking to repurpose the
dataset for their own use. That is, our planned output is a training dataset
and our goal is to develop derivative datasets from C4 and LAION-400M while
assessing representational biases that have been previously identified in text
and image datasets. We do not present every possible analysis; instead, we
focus on a few key results. We do this for two reasons. First, not all
analyses are relevant for understanding a specific social group or modality.
Thus, in practice, only a selection of analyses will be conducted. Second,
some analyses are not yet technically feasible and techniques to achieve them
are themselves the subject of research (e.g., detecting hateful symbols and
memes in images [55]). Finally, because we primarily focus on how to use
results across analyses to inform risk mitigation, we do not delve into
technical details or performance metrics for the classifiers we use.

Dataset Analysis Output: C4 Age Terms X Top Tokens:
Analysis Object: Text Task: For social identity terms in text, calculate top
word co-occurrences for each. Effort: Low Dependencies: None Action: Flag
associations. Consider rebalancing harmful associations that emerge in model
evaluations Output: Ageing anti-aging, wrinkles, collagen, elasticity, age-
related Old Age age-related, middle-aged, degeneration, dementia, ripe Elderly
dementia, ageing, older, alzheimer, frail Age Terms X Topic:
Task: Calculate the distribution of topics within text, disaggregated by
social identity terms or inferred social identity signals Analysis Object:
Inferred Text Signals + Text Effort: Low Dependencies: Topic Classifier
Action: Flag problematic associations Output: Ageing Medical Lit, Skin Care,
Face Care, Anti-Ageing, Nutrition Old Age Medical Lit, Books/Lit, Health
Cond., History, Retirement Elderly Medical Lit, Retirement, Assisted Living,
Geriartics, Health Cond. Figure 2: Sample dataset analyses output for C4.

###  5.1 Evaluating Age Representation in C4

We motivate our analyses based on prior research that identifies age bias as
an issue for ML and AI development [25, 31] and calls for increased
representation of older adults in AI datasets [59]. With this in mind, we turn
to assessing how older adults are depicted in C4 using Association analyses.
We assess the results of age depictions via top-associated tokens and topics
with age-related terms.

Output: We see in Figure 2 the tokens most associated with old age terms,
which occur 110,000 times in the dataset. These include dementia, and
degeneration, both of which can render negative sentiment. We see related
associations in the topics disproportionately associated with old age terms.
These include health topics, including medical conditions and assisted living,
as well as skin and face care beauty products, which likely point to content
covering anti-ageing products and discussion.

Action: In line with prior work, we find both low and skewed representation of
older age in text. Some work has been conducted on decoupling adjective
associations from select identity terms [24], however broader sentential
context surrounding age-related terms may still carry negative or stigmatized
sentiment. Filtering or removing data stands to worsen older adult
underrepresentation in ML datasets, however it is among the lowest cost
options. Toward the goal of developing training data, other actions may be
taken. If a data collection or generation pipeline is feasible to pursue,
targeted data collection or synthetic data can be used to rebalance the data,
ideally through ablation studies to understand how model performance changes.
Analysis results can also be flagged to evaluate the downstream model and the
data it generates for similar biases to assess model limitations. Contingent
on these evaluations, documentation warnings or non-use in certain cases may
also be necessary.

Dataset Analysis Output: LAION-400M Queer Identity Terms X Sexual Imagery:
Analysis Object: Text | Inferred Image Signals Task: Calculate co-occurrences between social identity terms and sexual imagery. Effort: Low Dependencies: Visual Content Classifier Action: Data filtering or tagging. Possible rebalancing Output: Queer Identity Terms X Topic:
Task: Calculate the distribution of topics by queer identity terms Analysis
Object: Inferred Text Signals, Text Effort: Low Dependencies: Topic Classifier
Action: Follow up analyses of sexual content Output: Lesbian Adult Videos,
LGBT Porn, Porn Gay LGBT, LGBT Porn, Online Images Bisexual LGBT, Porn, Online
Images Transgender LGBT, Social Issues, Discrimination Queer LGBT,
Events/Listings, Online Images Heterosexuality Porn, Adult Videos, Porn Figure
3: Sample dataset analyses output for LAION-400M Dataset.

###  5.2 Evaluating Queer Representation in LAION-400M

LAION-400M is a multimodal text and image dataset that features over 400
million image-text pairs extracted from Common Crawl, prominently used in
text-to-image generation. The dataset is unstructured and uncurated, though it
does feature NSFW tags, which were used to identify and filter a number of
illicit images. Text-image datasets are shown to reflect various societal
biases; for example, DALL-E and its variants reproduce gender and skin tone
biases from neutral prompts [21]. We apply the framework to LAION-400M to
analyze such multimodal and text associations.

Researchers have found unintended and undesirable associations with queer
identity terms in text datasets [27] and sexually explicit depictions of women
in LAION-400M [11, 12]. While explicit content can be used for specific
applications, unintentional inclusion risks accidental generation of explicit
content by downstream models. We assess a combination of these biases by
evaluating queer representation, again considering mitigations for adapting a
third-party training dataset. We look to the results of Association analyses
in text using the same topic classifier from our prior analyses, and we run
multimodal Association analyses using a classifier similar to [35], which
identifies sexual content in images. We analyze a collection of social
identity terms via webref entities.

Output: The top-associated tokens with each queer identity term, which can be
seen in Figure 3, largely reference other queer identity terms. Figure 3 shows
the top topics associated with a variety of sexual identities. Prominent among
these are those that seemingly refer to various sexual topics and activities.
This includes for "heterosexuality". Interestingly, the most frequent topic
for "heterosexuality" is "LGBT Porn" which suggests that the term is connected
to a subgenre of pornographic videos. Though not a sexual identity, the
generally derogatory term "transsexual" is also strongly associated with
sexual terms and topics.

Action: Considering dataset usage for T2I training in particular, there is
likely a very limited set of use cases in which the generation of sexual
content would be appropriate. Such use cases would likely entail very
specialized dataset curation and model development. Therefore, one could
consider filtering sexual content in order to both limit its downstream
production as well as to avoid the inclusion of published sexual content,
which has often been made public without sex worker consent [22]. Because
filtering may lead to removal of nearly half of the instances of some identity
terms, rebalancing may also be needed. Alternatively, sexual content in text
data could be augmented with tags to preserve a downstream model’s ability to
detect it while limiting its production.

##  6 Evaluating Representation in Data

In line with Mitchell et al. [57]’s call to establish practices for measuring
data, characterizing how people are represented in data is a necessary part of
identifying downstream risks. Yet, RAI lacks systematized guidance to address
the myriad ways that traces of social identity emerge in explicitly and
implicitly across data modalities. Up to this point, there has been little
guidance for triangulating social identity in datasets across different
analyses and approaches that use combinations of features to measure a latent
representation of social identity have not yet been widely applied in RAI
work.

Notably, our framework omits both a canonical list of social identities to
analyze as well as an exhaustive list of methods to measure a given social
identity. We do this because social identity is unstable nature [37]. Beyond
specific data modalities, the characteristics associated with social
identities change with context and, in a single context, the same features can
be associated with more than one identity group. For example, Hispanic
surnames are prevalent in both Latin America and the Philippines, meaning an
analysis of cultural diversity cannot rely on these features alone. Scholars
in RAI have shown that the axes along which discrimination occurs are
culturally specific [68].

Analyzing how people are depicted is additionally challenging because "good"
social representation can change with context. As Chasalow and Levy [20]
posit, representativeness is both time and place specific. The social
categories we attend to are shaped both by normative assumptions about what
should be measured as well as the existence of a name for or conception of a
social category. For example, Andrews et al. [5]’s research suggests that a
word list generated today to analyze disability representation in a dataset
would likely feature different terminology than a list generated 30 years ago.
As new language and visual cues emerge to describe existing or newly salient
social groups, our approaches to analyzing group representation must be
updated. Our framework aims to address these shifts with analyses that are
repeatable and updateable with emergent social identity cues.

###  6.1 The Role of Datasets in Assessing Harm

In developing this framework to support development decisions, we extend the
work of others advocating for more attention to data work [69], including the
growing focus on data-centric AI (DCAI) [45]. Building from DCAI’s focus on
understanding the data used and produced throughout ML development, our
framework sets a foundation for systematically analyzing social risks in data.
If DCAI is focused broadly on shifting focus from the model to the data, our
framework emphasizes human-centered angle within that focus. Some work in DCAI
does bring attention to data sources and the sociocultural views they
represent, such as those expressed through data annotation [26, 7, 56].
However, this work has limited application to unstructured data.

An important part of dataset evaluation is determining when a data
distribution is problematic. Future work in DCAI and RAI should explore the
effects of different distributions on model performance and output bias. This
work points to opportunities to use the framework to scaffold experiments to
study the effects of different data distributions on model performance across
contexts. In this way the framework is a concrete aid to what Jarrahi et al.
[45] calls "data benchmarking". Across iterative development of the same
model, as well as across developments of distinct models, the framework can
act as a consistent measuring stick for relating representations in data to
model fairness.

At the same time, dataset evaluations are just one component of RAI
evaluation. Much algorithmic fairness work focuses on data sources and dataset
pre-processing; however, as Hooker [40] argues, algorithm design choices, such
as optimization for privacy guarantees, compression techniques, and even
learning rate can contribute to model biases. Hooker also critically points
out that dataset evaluations rely on a priori decisions about which features
to evaluate and are inherently informed by human biases regarding what should
or should not be prioritized. As a result, dataset evaluations must be
considered alongside other approaches to mitigating risk and harm. For
example, Hooker turns to model compression techniques to isolate data points
at risk of exacerbated error rates as a way to guide further auditing. This
enables iterative error analysis that DCAI calls for.

###  6.2 Supporting RAI Goals

RAI development requires dataset analyses that integrate into existing RAI
workflows in ways that complement existing tools and frameworks. This includes
structured guidance to repeatedly apply analyses of human depictions in data.
Our framework supports the development of transparency artifacts by
standardizing results and by flagging benchmark tests to prioritize based on
problematic data distributions. In this context, the framework stands as a
structured auditing aid.

How and when to act on analysis results depends on the nature of the data
being used as well as risk mitigation goals. In other words, the same results
can warrant different action, depending on context. A primary use motivating
the design of this framework is to analyze datasets used for foundation
models. The high cost of model training limits opportunities to run
comprehensive studies to identify which mitigation strategies improve fairness
issues as well as their impacts model performance generally. For RAI, this
means making mitigation decisions with limited information about specific
impacts. This challenge is exacerbated by data cascades, which can compound to
produce out-sized, negative outcomes [69]. Yet, the range of potential
downstream risks warrants proactive decision making. While decision making at
the training data level is difficult when downstream applications are unclear,
the framework provides valuable information about the people represented in
data, and can be used to guide later evaluations of fine-tuning data or model-
generated data.

##  7 Conclusion

The open-ended nature of AI risks and harms poses challenges to RAI
practitioners seeking to not only identify risks, but also take appropriate
action to mitigate them, at times with limited information about how
downstream models will be fine-tuned or applied. In response, we propose a
standardized framework to evaluate unstructured datasets for downstream risk,
with a focus on human representation. Building from critical dataset audits
and concurrent work standardizing these efforts (such as by Elazar et al.
[29]), we organize our framework around social representation and provide
exemplar uses to demonstrate its application. Our framework is designed as a
general-purpose starting point that is extensible to other modalities and
application contexts, as needed.

## References

  * [1]
  * Abid et al. [2021] Abubakar Abid, Maheen Farooqi, and James Zou. 2021\.  Persistent anti-muslim bias in large language models. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 298–306.
  * Ahmadi and Masoud [2020] Sina Ahmadi and Mariam Masoud. 2020.  Towards machine translation for the Kurdish language.  _arXiv preprint arXiv:2010.06041_ (2020).
  * Allamanis [2019] Miltiadis Allamanis. 2019\.  The adverse effects of code duplication in machine learning models of code. In _Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software_. 143–153.
  * Andrews et al. [2022] Erin E Andrews, Robyn M Powell, and Kara Ayers. 2022\.  The evolution of disability language: Choosing terms to describe disability.  _Disability and Health Journal_ 15, 3 (2022), 101328\.
  * Anil et al. [2023] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023.  PaLM 2 Technical Report.  arXiv:cs.CL/2305.10403
  * Arhin et al. [2021] Kofi Arhin, Ioana Baldini, Dennis Wei, Karthikeyan Natesan Ramamurthy, and Moninder Singh. 2021\.  Ground-Truth, Whose Truth?–Examining the Challenges with Annotating Toxic Text Datasets.  _arXiv preprint arXiv:2112.03529_ (2021).
  * Bandy and Vincent [2021] Jack Bandy and Nicholas Vincent. 2021.  Addressing" documentation debt" in machine learning research: A retrospective datasheet for bookcorpus.  _arXiv preprint arXiv:2105.05241_ (2021).
  * Barocas et al. [2017] Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017.  The Problem With Bias: Allocative Versus Representational Harms in Machine Learning. SIGCIS.  http://meetings.sigcis.org/uploads/6/3/6/8/6368912/program.pdf
  * Bender and Friedman [2018] Emily M Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ 6 (2018), 587–604.
  * Birhane and Prabhu [2021] Abeba Birhane and Vinay Uday Prabhu. 2021.  Large image datasets: A pyrrhic win for computer vision?. In _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 1536–1546.
  * Birhane et al. [2021] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021\.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  _arXiv preprint arXiv:2110.01963_ (2021).
  * Bolukbasi et al. [2016] Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _Advances in neural information processing systems_ 29 (2016).
  * Bommasani et al. [2021] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Buolamwini and Gebru [2018] Joy Buolamwini and Timnit Gebru. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification. In _Conference on fairness, accountability and transparency_. PMLR, 77–91.
  * Caliskan et al. [2017] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017\.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ 356, 6334 (2017), 183–186.
  * Carlini et al. [2021] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021\.  Extracting Training Data from Large Language Models.. In _USENIX Security Symposium_ , Vol. 6.
  * Chang and Custis [2022] Jiyoo Chang and Christine Custis. 2022.  Understanding Implementation Challenges in Machine Learning Documentation.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_. 1–8.
  * Chasalow and Levy [2021] Kyla Chasalow and Karen Levy. 2021.  Representativeness in statistics, politics, and machine learning. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_. 77–89.
  * Cho et al. [2022] Jaemin Cho, Abhay Zala, and Mohit Bansal. 2022.  Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers.  _arXiv preprint arXiv:2202.04053_ (2022).
  * Cole [2020] Samantha Cole. 2020\.  These Nudes Do Not Exist and I Don’t Know Why This Startup Does Either. In _Vice_.  https://www.vice.com/en/article/qjdx7w/these-nudes-do-not-exist-and-i-dont-know-why-this-startup-does-either.
  * Crawford and Paglen [2021] Kate Crawford and Trevor Paglen. 2021.  Excavating AI: The politics of images in machine learning training sets.  _Ai & Society_ 36, 4 (2021), 1105–1116.
  * Dev et al. [2021] Sunipa Dev, Tao Li, Jeff M Phillips, and Vivek Srikumar. 2021\.  OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5034–5050.  https://doi.org/10.18653/v1/2021.emnlp-main.411
  * Díaz et al. [2018] Mark Díaz, Isaac Johnson, Amanda Lazar, Anne Marie Piper, and Darren Gergle. 2018.  Addressing age-related bias in sentiment analysis. In _Proceedings of the 2018 chi conference on human factors in computing systems_. 1–14.
  * Díaz et al. [2022] Mark Díaz, Ian Kivlichan, Rachel Rosen, Dylan Baker, Razvan Amironesei, Vinodkumar Prabhakaran, and Emily Denton. 2022\.  Crowdworksheets: Accounting for individual and collective identities underlying crowdsourced dataset annotation. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. 2342–2351.
  * Dixon et al. [2018] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and mitigating unintended bias in text classification. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_. 67–73.
  * Dodge et al. [2021] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021\.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  _arXiv preprint arXiv:2104.08758_ (2021).
  * Elazar et al. [2023] Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, et al. 2023\.  What’s In My Big Data?  _arXiv preprint arXiv:2310.20707_ (2023).
  * Friedler et al. [2019] Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth. 2019.  A comparative study of fairness-enhancing interventions in machine learning. In _Proceedings of the conference on fairness, accountability, and transparency_. 329–338.
  * Garcia de Alford et al. [2020] Adriana Solange Garcia de Alford, Steven K Hayden, Nicole Wittlin, and Amy Atwood. 2020.  Reducing Age Bias in Machine Learning: An Algorithmic Approach.  _SMU Data Science Review_ 3, 2 (2020), 11\.
  * Garg et al. [2018] Nikhil Garg, Londa Schiebinger, Dan Jurafsky, and James Zou. 2018.  Word embeddings quantify 100 years of gender and ethnic stereotypes.  _Proceedings of the National Academy of Sciences_ 115, 16 (2018), E3635–E3644.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021\.  Datasheets for datasets.  _Commun. ACM_ 64, 12 (2021), 86–92.
  * Gehman et al. [2020] Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _arXiv preprint arXiv:2009.11462_ (2020).
  * Google Cloud Vision [[n.d.]] Google Cloud Vision. [n.d.].  _Image Annotation_.  https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation
  * Han et al. [2021] Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, et al. 2021\.  Pre-trained models: Past, present and future.  _AI Open_ 2 (2021), 225–250.
  * Hanna et al. [2020] Alex Hanna, Emily Denton, Andrew Smart, and Jamila Smith-Loud. 2020\.  Towards a critical race methodology in algorithmic fairness. In _Proceedings of the 2020 conference on fairness, accountability, and transparency_. 501–512.
  * Heger et al. [2022] Amy K Heger, Liz B Marquis, Mihaela Vorvoreanu, Hanna Wallach, and Jennifer Wortman Vaughan. 2022\.  Understanding Machine Learning Practitioners’ Data Documentation Perceptions, Needs, Challenges, and Desiderata.  _Proceedings of the ACM on Human-Computer Interaction_ 6, CSCW2 (2022), 1–29.
  * Hendricks et al. [2018] Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018\.  Women also snowboard: Overcoming bias in captioning models. In _Proceedings of the European conference on computer vision (ECCV)_. 771–787.
  * Hooker [2021] Sara Hooker. 2021\.  Moving beyond “algorithmic bias is a data problem”.  _Patterns_ 2, 4 (2021), 100241.
  * Hovy and Prabhumoye [2021] Dirk Hovy and Shrimai Prabhumoye. 2021.  Five sources of bias in natural language processing.  _Language and Linguistics Compass_ 15, 8 (2021), e12432.
  * Huang et al. [2019] Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2019.  Reducing sentiment bias in language models via counterfactual evaluation.  _arXiv preprint arXiv:1911.03064_ (2019).
  * Hutchinson et al. [2020] Ben Hutchinson, Vinodkumar Prabhakaran, Emily Denton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020.  Social biases in NLP models as barriers for persons with disabilities.  _arXiv preprint arXiv:2005.00813_ (2020).
  * Hutchinson et al. [2021] Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021\.  Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 560–575.  https://doi.org/10.1145/3442188.3445918
  * Jarrahi et al. [2022] Mohammad Hossein Jarrahi, Ali Memariani, and Shion Guha. 2022\.  The Principles of Data-Centric AI (DCAI).  _arXiv preprint arXiv:2211.14611_ (2022).
  * Johnson and Khoshgoftaar [2019] Justin M Johnson and Taghi M Khoshgoftaar. 2019.  Survey on deep learning with class imbalance.  _Journal of Big Data_ 6, 1 (2019), 1–54.
  * Kearns et al. [2018] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018.  Preventing fairness gerrymandering: Auditing and learning for subgroup fairness. In _International conference on machine learning_. PMLR, 2564–2572.
  * Kemelmacher-Shlizerman et al. [2016] Ira Kemelmacher-Shlizerman, Steven M Seitz, Daniel Miller, and Evan Brossard. 2016.  The megaface benchmark: 1 million faces for recognition at scale. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4873–4882.
  * Khan et al. [2018] Mahrukh Khan, Muhammad Atif Tahir, and Zeeshan Ahmed. 2018.  Detection of violent content in cartoon videos using multimedia content detection techniques. In _2018 IEEE 21st International Multi-Topic Conference (INMIC)_. IEEE, 1–5.
  * Kleinberg et al. [2016] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016\.  Inherent trade-offs in the fair determination of risk scores.  _arXiv preprint arXiv:1609.05807_ (2016).
  * Lanitis et al. [2004] Andreas Lanitis, Chrisina Draganova, and Chris Christodoulou. 2004\.  Comparing different classifiers for automatic age estimation.  _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ 34, 1 (2004), 621–628.
  * Lee et al. [2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2021.  Deduplicating training data makes language models better.  _arXiv preprint arXiv:2107.06499_ (2021).
  * Lima et al. [2022] Gabriel Lima, Nina Grgić-Hlača, Jin Keun Jeong, and Meeyoung Cha. 2022.  The Conflict Between Explainable and Accountable Decision-Making Algorithms. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. 2103–2113.
  * Luccioni and Viviano [2021] Alexandra Sasha Luccioni and Joseph D Viviano. 2021.  What’s in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus.  _arXiv preprint arXiv:2105.02732_ (2021).
  * Mathias et al. [2021] Lambert Mathias, Shaoliang Nie, Aida Mostafazadeh Davani, Douwe Kiela, Vinodkumar Prabhakaran, Bertie Vidgen, and Zeerak Waseem. 2021\.  Findings of the WOAH 5 shared task on fine grained hateful memes detection. In _Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)_. 201–206.
  * Mishra and Gorana [2021] Abhilash Mishra and Yash Gorana. 2021.  Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing.  _arXiv preprint arXiv:2111.08723_ (2021).
  * Mitchell et al. [2022] Margaret Mitchell, Alexandra Sasha Luccioni, Nathan Lambert, Marissa Gerchick, Angelina McMillan-Major, Ezinwanne Ozoani, Nazneen Rajani, Tristan Thrush, Yacine Jernite, and Douwe Kiela. 2022.  Measuring Data.  _arXiv preprint arXiv:2212.05129_ (2022).
  * Mitchell et al. [2019] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model Cards for Model Reporting. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ _(FAT* ’19)_. Association for Computing Machinery, New York, NY, USA, 220–229.  https://doi.org/10.1145/3287560.3287596
  * Park et al. [2021] Joon Sung Park, Michael S Bernstein, Robin N Brewer, Ece Kamar, and Meredith Ringel Morris. 2021\.  Understanding the Representation and Representativeness of Age in AI Data Sets. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 834–842.
  * Paullada et al. [2021] Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2021\.  Data and its (dis)contents: A survey of dataset development and use in machine learning research.  _Patterns_ 2, 11 (2021), 100336.  https://doi.org/10.1016/j.patter.2021.100336
  * Pushkarna et al. [2022] Mahima Pushkarna, Andrew Zaldivar, and Oddur Kjartansson. 2022\.  Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.  http://jmlr.org/papers/v21/20-074.html
  * Raji et al. [2020a] Inioluwa Deborah Raji, Timnit Gebru, Margaret Mitchell, Joy Buolamwini, Joonseok Lee, and Emily Denton. 2020a.  Saving face: Investigating the ethical concerns of facial recognition auditing. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 145–151.
  * Raji et al. [2020b] Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020b.  Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In _Proceedings of the 2020 conference on fairness, accountability, and transparency_. 33–44.
  * Rostamzadeh et al. [2022] Negar Rostamzadeh, Diana Mincu, Subhrajit Roy, Andrew Smart, Lauren Wilcox, Mahima Pushkarna, Jessica Schrouff, Razvan Amironesei, Nyalleng Moorosi, and Katherine Heller. 2022.  Healthsheet: development of a transparency artifact for health datasets. In _2022 ACM Conference on Fairness, Accountability, and Transparency_. 1943–1961.
  * Russakovsky et al. [2014] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. 2014\.  ImageNet Large Scale Visual Recognition Challenge.  _CoRR_ abs/1409.0575 (2014).  arXiv:1409.0575 http://arxiv.org/abs/1409.0575
  * Saleiro et al. [2018] Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani. 2018\.  Aequitas: A bias and fairness audit toolkit.  _arXiv preprint arXiv:1811.05577_ (2018).
  * Sambasivan et al. [2021a] Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. 2021a.  Re-imagining algorithmic fairness in india and beyond. In _Proceedings of the 2021 ACM conference on fairness, accountability, and transparency_. 315–328.
  * Sambasivan et al. [2021b] Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo. 2021b.  “Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI. In _proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–15.
  * Sap et al. [2019] Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The Risk of Racial Bias in Hate Speech Detection. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 1668–1678.  https://doi.org/10.18653/v1/P19-1163
  * Scheuerman et al. [2019] Morgan Klaus Scheuerman, Jacob M Paul, and Jed R Brubaker. 2019\.  How computers see gender: An evaluation of gender classification in commercial facial analysis services.  _Proceedings of the ACM on Human-Computer Interaction_ 3, CSCW (2019), 1–33.
  * Schiff et al. [2020] Daniel Schiff, Bogdana Rakova, Aladdin Ayesh, Anat Fanti, and Michael Lennon. 2020.  Principles to practices for responsible AI: Closing the gap.  _arXiv preprint arXiv:2006.04707_ (2020).
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021\.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ (2021).
  * Shankar et al. [2017] Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017\.  No classification without representation: Assessing geodiversity issues in open data sets for the developing world.  _arXiv preprint arXiv:1711.08536_ (2017).
  * Srinivasan et al. [2021] Ramya Srinivasan, Emily Denton, Jordan Famularo, Negar Rostamzadeh, Fernando Diaz, and Beth Coleman. 2021.  Artsheets for Art Datasets.  (2021).
  * Torralba et al. [2008] Antonio Torralba, Rob Fergus, and William T Freeman. 2008\.  80 million tiny images: A large data set for nonparametric object and scene recognition.  _IEEE transactions on pattern analysis and machine intelligence_ 30, 11 (2008), 1958–1970.
  * Trinh and Le [2018] Trieu H Trinh and Quoc V Le. 2018.  A simple method for commonsense reasoning.  _arXiv preprint arXiv:1806.02847_ (2018).
  * Wagner et al. [2020] Ben Wagner, Krisztina Rozgonyi, Marie-Therese Sekwenz, Jennifer Cobbe, and Jatinder Singh. 2020\.  Regulating transparency? Facebook, twitter and the German network enforcement act. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 261–271.
  * Weidinger et al. [2021] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021\.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_ (2021).
  * Wilson et al. [2019] Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern. 2019\.  Predictive inequity in object detection.  _arXiv preprint arXiv:1902.11097_ (2019).
  * Zhao et al. [2018] Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018.  Gender bias in coreference resolution: Evaluation and debiasing methods.  _arXiv preprint arXiv:1804.06876_ (2018).

##  Appendix A Framework Introduction

Here we present a framework overview, detailed for text, image, and text-image
application. Rather than an exhaustive list of every possible analysis, the
Human-Content Associations section shows only social identity analyses for
each modality. At the end of the overview we provide a list of example tools
that can be used to meet the dependency requirements of the analyses. The full
framework features recommended risk mitigation actions, however, in practice,
mitigation steps should depend on development context and goals. The below
tables list common analysis goals and mitigation actions and the following
section on Guiding Considerations lists questions and prompts to guide
framework use.

Analysis Goals
---
Dataset Development |  Developing a dataset for training or evaluation through new data collection and/or adaptation of existing datasets
Use Decisions |  Making decisions regarding appropriate use of a dataset, whether for training or evaluative purposes
Model Understanding |  Investigating potential roots of or explanations for model behavior
Auditing |  Auditing a dataset to fill documentation gaps, ensure legal or institutional compliance, or to foster greater public awareness
Table 3: A non-exhaustive list of data analysis goals. Dev. Phase |  Actions |  Description
---|---|---
Data Collection/ Processing |  Addition |  Rebalancing distributions across an entire dataset or within specified categories with additional (potentially synthetic) data
Removal |  Filtering data to remove unwanted content
Augmentation |  Augmenting data, such as through data tagging [6] to allow a model to learn undesirable content while controlling its production downstream
Flagging |  Flagging analysis results for further downstream evaluation or documentation
Non-Use |  Not using the dataset, for example if applying analyses to different candidate datasets to decide which to use
Model Evaluation |  Addt’l Benchmarking |  Selection of additional evaluation benchmarks
Benchmark Creation |  Development of benchmarks to evaluate new concerns
Documentation |  Warning |  Documentation of general or use case-specific limitations
Non-Use |  Documentation of cases where the data should not be used
Packaging and Release |  Licensing |  Development of licensing and terms of use specifications
Access |  Development of limited access policies
Table 4: A non-exhaustive list of actions that may be taken to address social
risks identified in data.

###  A.1 Guiding Considerations

  * •

What are the planned deliverables of the data being analyzed?
The planned outputs help to inform decisions regarding resource investment in
analyses and potential mitigations. For example, a novel benchmark dataset may
warrant more extensive analysis than a pre-training dataset because it will be
repeatedly use to judge the performance of many models. Similarly, a fine-
tuning dataset may be the subject of more modification than a pre-training
dataset due to its smaller size and direct role in shaping model compliance
with a range of fairness and policy constraints.

  * •

What are the primary goals the analyses will support?
Compliance audits and use decisions may only warrant documentation, or
packaging and release actions; model understanding may simply be used to help
diagnose model behavior (i.e., no mitigation action on data); whereas the
development of a new dataset can involve many mitigation actions.

  * •

To what extent can completed or planned development steps be revisited or
modified
Additional analyses and mitigations require time and resources, as does
experimentation to compare which actions are most effective for a given
project. For example, budget may be limited for additional data collection,
while data filtering or tagging may still be possible.

  * •

To what extent is the dataset mutable? (i.e., can data be added, removed, or
changed?)
The mutability of a dataset narrows the available mitigation actions and the
degree to which data can be filtered or rebalanced.

  * •

Can new or additional model benchmarks be run?
When developing a new model, an ultimate consideration is whether biases
identified in data will persist through training and finetuning steps. This is
known to be true for a number of biases discovered in certain datasets. For
novel or recently discovered biases, relevant benchmarks may not exist or are
not feasible to create to test downstream models for the bias. This may place
emphasis on other mitigation actions, depending on the risk associated with
the identified bias.

  * •

Will the data or a model trained on the data be released or made available for
external use? If so, can the release terms (e.g., terms of use, licensing,
policies) be modified?
The use of data and models released externally cannot be as carefully
controlled as data and models used strictly within a single institution.
Safeguards against risks may be handled through terms of use or licensing
agreements in order to mitigate data or model limitations.

  * •

Are there institutional or legal requirements regarding what should be
analyzed in the data?
Legal and institutional compliance are a first step for planning analyses and
reporting results. These can be supplemented with parallel fairness analyses
of additional social groups, or analyzed more in depth to make mitigation
decisions (e.g., analyzing which data sources feature the strongest identified
biases in order to make filtering decisions)

  * •

Are there additional fairness or bias concerns a priori?
Beyond compliance concerns, consider analyses that involve groups or data
subjects that have been previously impacted by similar systems or within which
the context the data or model will be released.

  * •

What signals are available for analyzing social characteristics in the data?
The feasibility of running a given analysis depends on the available data
features. Aside from straightforward considerations, such as whether a dataset
contains faces that can be detected, varied social characteristics may map to
a single data feature (e.g., common surnames across disparate racial and
ethnic groups) or may not easily map to any data features (e.g., religion and
disability are two examples of social characteristics that are not always
’visible’ in images).

##  Appendix B Analysis Overview

###  B.1 Human Factors

####  B.1.1 Presence of People

  * •

Personally-Identifiable Information

  * •

Faces

####  B.1.2 Social Characteristics

  * •

Social Identity Terms

  * •

Pronouns

  * •

Hateful Terms in Text

  * •

Dialect

  * •

Perceived Social Identity in Images

  * •

Hateful Symbols in Images

###  B.2 Content Factors

####  B.2.1 Content

  * •

Offensive Speech

  * •

Topics

  * •

Sexual Imagery

  * •

Violent Imagery

####  B.2.2 Provenance

  * •

Top Sources

  * •

Source Geography

  * •

Source Publication Time

  * •

Language

  * •

Data Duplication

  * •

Dataset Overlap

###  B.3 Human X Content Associations

Text

  * •

Social Identity Terms X Top Tokens

  * •

Social Identity Terms X Offensive Speech

  * •

Social Identity Terms X Topic

Image

  * •

Perceived Social Identity Features X Top Image Features

  * •

Perceived Social Identity Features X Sexual Imagery

  * •

Perceived Social Identity Features X Violent Imagery

Text-Image

  * •

Social Identity Terms X Sexual Imagery

  * •

Social Identity Terms X Violent Imagery

  * •

Perceived Social Identity Features X Text-Image

  * •

Perceived Social Identity Features X Sexual Imagery

  * •

Perceived Social Identity Features X Violent Imagery

  * •

Perceived Social Identity Features X Offensive Speech

  * •

Perceived Social Identity Features X Topic

###  B.4 Analysis Dependencies

Below is a list of technical dependencies required for framework analyses
along with example classifiers that can be used.

PII Detection:

Google Data Loss Prevention API (InfoType Detector)

Face Detection:

Google Cloud Vision API (Image Annotation)

Sexual Imagery:

Google Cloud Vision API (Safe Search Annotation)

Violent Imagery:

Google Cloud Vision API (Safe Search Annotation)

Object Classifier:

Google Cloud Vision API (Object Localization)

Topic Classifier:

Google Cloud Natural Language API (Content Categories)

Offensive Speech:

Perspective API (Note: Not offensive speech, but Toxicity)

Language Classifier:

Google Cloud Natural Language API (Text Annotation

Dialect Classifier:

No comprehensive classification approaches.

Hateful Symbols:

No currently reliable method

Social Identity Features:

A number of published audits point to important limitations of social identity
classification systems (e.g., [16] and [71] audits of gender classification
for image, and [63] audit of gender and age classification for image),
therefore we encourage research and results interpretation with individual
usage contexts in mind.

##  Appendix C List of Framework Analyses

###  C.1 Human Factors

Analyses focused on identifying individuals, social identity groups, and
cultural groups in data.

Presence of People

####  C.1.1 Personally Identifiable Information (PII)

Personally identifiable information allows for an individual’s identity to be potentially inferred through direct or indirect means. It includes, but is not limited to, names, email addresses, addresses, and images or video with identifying characteristics. The presence of personally identifiable information in a dataset is not inherently problematic. However, even if a dataset is not intended for public use, publishing a model trained on a dataset containing personally identifiable information risks “leaking” that information through attack methods designed to recover specific training instances from a model [18]. Generative models that have overfit to the training distribution may also risk generating outputs that constitute a breach in privacy. Task: Identify data containing PII Analysis Object: Text | Inferred Text Signals Effort: Medium Dependencies: PII Detection Output: Proportion and list of data points containing PII Action: Flag PII to assess potential privacy violations that arise for model tasks or as a result of dataset publication

####  C.1.2 People in Images

As a subcategory of personally-identifiable information, faces in datasets are
at risk of “leaking” through attack methods designed to recover specific
training instances from a model [18]. The presence of PII is not inherently
problematic, however privacy risks must be evaluated in the context of
specific tasks. Task: Calculate the proportion of images that depict people
(e.g., via face detection) Analysis Object: Inferred Image Signals Effort: Low
Dependencies: Face Detection Output: Proportion of images that depict people
Action: Flag images of people

Social Characteristics

####  C.1.3 Social Identity Terms

Under-representation in datasets can contribute to representational harms [9],
such as strong under-representation of darker-skinned individuals in datasets
linked to under-performance in pedestrian detection [80] and data imbalances
generally causing issues for class prediction [46]. Attention to social
identity representation in datasets also helps to support intersectional
analyses which inherently focus on relatively smaller intersections of data
but require sufficient data points to conduct. Task: Calculate proportion of
text referencing different social identity groups, considering unitary and
intersectional groups Analysis Object: Text Effort: Low Dependencies: Social
Identity Term List Output: Frequency of text depicting unitary and
intersectional groups Action: Flag social identity representation

####  C.1.4 Pronoun Distribution

Pronoun distributions help provide a high-level snapshot of gender
representation in datasets to evaluate their connection to model performance
on gender bias benchmarks. Gender stereotypic associations have been well
documented in NLP [13, 17] and mitigating imbalances in gendered terms in
datasets has been shown to mitigate gender bias in benchmark performance [81].
Task: Count number of gender pronouns that occur Analysis Object: Text Effort:
Low Dependencies: Pronoun List Output: Distribution of pronouns Action: Flag
pronoun distribution

####  C.1.5 Hateful Terms in Text

When training datasets contain derogatory language, slurs, or offensive
language, there is a risk models trained on the data will apply these terms in
an inappropriate manner, e.g. describing people with such terms. Several
recent audits of image classification datasets and image-to-text datasets have
uncovered the presence racial and ethnic slurs and other derogatory and
hateful language [12, 11, 23]. Task: Count number of hateful terms that occur
Analysis Object: Text Effort: Low Dependencies: Hateful Term List Output:
Distribution of hateful terms, the groups they reference, and their
occurrences Action: Flag hateful terms

####  C.1.6 Dialect

Dialects vary across regions and cultures. Data that lacks manners of speaking
or dialects prominently used by subgroups can lead to erasure of those
dialects in resultant models, or categorization of non-dominant dialects as
incorrect, low quality, or even offensive [70]. Task: Calculate proportion of
documents representing languages and dialects aimed to be supported in
downstream applications Analysis Object: Inferred Text Signals Effort: High
Dependencies: Dialect Classifier Output: Proportion of utterances representing
different languages and dialects Action: Flag dialect representation. Consider
rebalancing strongly underrepresented dialects or qualifying downstream model
capabilities.

####  C.1.7 Social Identity in Images

Training datasets that under-represent certain social identity groups can
produce models that exhibit poor performance when presented with novel
depictions of those groups. For example, image classifiers trained on datasets
sourced predominantly from western countries have lower accuracy when applied
to images from non-western countries [74], and facial analysis systems trained
on datasets that skew heavily towards lighter skinned subjects have higher
error rates when applied to images depicting faces with darker skin tones
[16]. Task: Calculate proportion of images depicting social identity groups,
considering unitary and intersectional groups Analysis Object: Inferred Image
Signals Effort: Low Dependencies: Perceived Social Identity Classifiers
Output: Proportion of images depicting unitary and intersectional groups
Action: Flag social identity representation

####  C.1.8 Hateful Symbols in Images

Visual imagery associated with hate groups (e.g., swastikas) may go undetected
or undocumented in datasets. Identifying hateful content across modalities can
help mitigate the risk of unintentionally generating hateful or offensive
content. It is important to note that offensiveness of imagery is often
dependent on cultural context, and hence any action to address this issue
should account for this. Task: Calculate proportion of images that depict
known hateful symbols or text Analysis Object: Inferred Image Signals Effort:
Not yet possible by automated methods Dependencies: Hateful Symbol Classifier
Output: Proportion of images depicting known hateful symbols or text Action:
Flag images with hateful symbols

###  C.2 Content Factors

Analyses focused on identifying content that may heighten the sensitivity of
human depictions.

Content

####  C.2.1 Offensive Speech Distribution

Identifying the distribution of offensive content in training data can help to
understand the relationship between offensive speech in training data and
model outputs as well as help mitigate the risk of unintentionally generating
toxic content. Analyses of OWTC and OpenAI-WT suggest that toxic language
generation has links to toxic language in web text data and toxic training
examples that may be harder for a model to “forget” [34]. Task: Calculate the
distribution of toxicity Analysis Object: Inferred Text Signals Effort: Low
Dependencies: Offensive Speech Classifier or similar (e.g., Perspective
API222www.perspectiveapi.com) Output: Histogram of offensiveness across
documents Action: Flag toxic content

####  C.2.2 Topic Distribution

Topic distribution can provide a birds-eye view of the composition of the
dataset across various topics such as sports, games, finance, etc. More
importantly, it can also give an indication of sexually explicit or sensitive
topics they contain, and potential biases they bring. For example, models
trained primarily on news data have been shown to exhibit biases against
particular country names and professions [42]. Assessments of data
distributions across languages may highlight potential weaknesses across
languages in generative tasks. Task: Calculate the distribution of topics
Analysis Object: Inferred Text Signals Effort: Low Dependencies: Topic
Classifier (e.g., Google Cloud Content
Categories333https://cloud.google.com/natural-language/docs/categories)
Output: Histogram of topic distribution Action: Flag dominant topics

####  C.2.3 Sexual Imagery

Many datasets have been identified as unintentionally containing sexually
explicit content. For example, audits of image datasets such as Imagenet,
TinyImage, and LAION-400M found pornographic and explicit imagery,
predominantly depicting women [12, 11]. While some datasets may be explicitly
curated to contain sexually explicit content, unintentional inclusion of such
content can risk accidental generation of sexually explicit content by models
trained on the data. Sexually explicit image content also raises potential
ethical concerns regarding data sourcing since much publically available
pornographic content has been stolen from sex workers or captured in a non-
consentual manner [22]. Task: Calculate the proportion of images that depict
sexual content Analysis Object: Inferred Image Signals Effort: Low
Dependencies: Visual Content Classifier (e.g., Cloud Vision
API444https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation)
Output: Proportion of images depicting sexual content Action: Flag sexual
imagery. Consider filtering sexual imagery depending on downstream tasks and
to avoid problematically-sourced dataset content

####  C.2.4 Violent Imagery

Like sexual imagery, violent content may feature in datasets as a part of
efforts to moderate content, such as in the detection of cartoon content that
may be inappropriate for children [49]. For general purposes, inadvertent
generation of violent content may be undesirable or counter to institutional
or platform policies. Task: Calculate the proportion of images that depict
violent content Analysis Object: Inferred Image Signals Effort: Low
Dependencies: Visual Content Classifier (e.g., Cloud Vision
API555https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation)
Output: Proportion of images depicting violent content Action: Flag violent
imagery. Consider filtering violent imagery depending on downstream tasks

Provenance

####  C.2.5 Top Sources

The top source URL domains provide a high-level indication of the range of
content most represented in web-scraped data. This, in turn, provides insight
into document content, such as language and cultural content represented, as
well as the prevalence of machine-generated text [28]. Task: Count the top
websites from which tokens were collected Analysis Object: Metadata Effort:
Low Dependencies: None Output: List of the top websites ordered by tokens
collected Action: Flag top URLs

####  C.2.6 Source Geographic Spread

Source URL domains provide an indication of geographic, cultural, and social
representation. When training datasets have strong cultural skews, a resulting
model can exhibit poor performance when presented with culturally or
regionally specific depictions. For example, image classifiers trained on
datasets sourced predominantly from western countries have lower rates of
accuracy when applied to images from non-western countries [74]. Task:
Calculate the known proportions of total tokens from different geographic
regions (e.g., as determined through country-specific top-level domains such
as, .uk, .de, .gh, etc.) Analysis Object: Metadata Effort: Low Dependencies:
None Output: List of the top country-level domains with the proportion of data
sourced from each Action: Flag geographic spread. Consider rebalancing if
regions relatively underrepresented in the dataset are prioritized for
downstream uses.

####  C.2.7 Source Data Publication Time

The relevance of some data can change over time. For example, assessments of
truth or offensiveness of a statement can change in relation to when it is
evaluated. Data recency may have impacts on models supporting low-resource
languages, which can disproportionately rely on religious or historical texts
due to data scarcity [3]. Task: Identify the dates (years) of publication for
data obtained Analysis Object: Metadata Effort: Low Dependencies: None Output:
Histogram of data published per year Action: Flag publication time. Consider
rebalancing if prioritizing tasks likely to be relatively sensitive to data
publication time (e.g., Q&A factuality; use of contemporary slang in
generative language tasks)

####  C.2.8 Language

Languages vary across regions and cultures, however work in NLP frequently focuses on English, even when there is data available for other languages [41]. As with under-representation of data and classes in other contexts, poorly represented languages in a language dataset may lead to worsened performance of a downstream model when applied to that langauge. Task: Calculate proportion of documents representing languages aimed to be supported in downstream applications Analysis Object: Text | Inferred Text Signals Effort: Low Dependencies: Language Classifier (e.g., Google Cloud Natural Language API) Output: Proportion of utterances representing different languages Action: Flag language representation. Consider rebalancing strongly underrepresented languages or qualifying downstream model capabilities.

####  C.2.9 Data Duplication

Duplicate text has been identified in a number of datasets (e.g., [8]) and has been shown to worsen model performance on some tasks [4] as well as induce model memorization [52], which can carry privacy risk [18]. Task: Calculate the proportion of duplicated data points Analysis Object: Text | Image Effort: Low Dependencies: None Output: Proportions of duplicated data points Action: Filter duplicate data to reduce memorization and privacy risks and improve model performance

####  C.2.10 Dataset Overlap

There is growing awareness and concern of test set contamination, i.e. presence of training instances in the test set, especially with large internet sourced datasets. Explicitly examining and mitigating any overlap in training and evaluation splits is increasingly commonplace (e.g., [77, 15]. Task: Identify overlaps between the present dataset and relevant datasets and benchmarks Analysis Object: Text | Image Effort: Medium Dependencies: None Output: For benchmark datasets identified, list of percent overlap with the current dataset based on exact matches of target text Action: Filter dataset overlaps to preserve the validity of benchmark results

###  C.3 Human X Content Associations

Disaggregated analyses of how content is associated with human factors. Listed
below are a selection of all possible associations between Human Factors and
Content Factors. Disaggregated analyses enable evaluation of stereotypical or
otherwise harmful associations between depictions of people and dataset
content. For example, prior research has identified stereotype-aligned
associations between gender and activities in image datasets [81, 39],
associations between muslims and violence in language models [2], and
associations between women of color and explicit content in text datasets [54]
and image-text datasets [12].

Text

####  C.3.1 Social Identity Terms X Top Tokens

Analysis Object: Text Task: For social identity terms in text, calculate top
word co-occurrences for each Effort: Low Dependencies: None Output: List of
most frequent social identity terms and their respective top co-occurrences,
demarcating identity terms (e.g., woman, Muslim, trans) Action: Flag
associations. Consider rebalancing harmful associations that emerge in model
evaluations

####  C.3.2 Social Identity Terms X Topic

Task: Calculate the distribution of topics within text, disaggregated by
social identity terms or inferred social identity signals Analysis Object:
Inferred Text Signals + Social Identity Term List Effort: Low Dependencies:
Topic Classifier (e.g., Google Cloud Content
Categories666https://cloud.google.com/natural-language/docs/categories)
Output: Distribution of topics disaggregated by social identity Action: Flag
disproportionate associations between social identity terms and stereotypical
or sensitive topics

####  C.3.3 Social Identity Terms X Offensive Speech

Task: Calculate the distribution of toxicity within text, disaggregated by
social identity terms or inferred social identity signals Analysis Object:
Inferred Text Signals + Social Identity Term List Effort: Low Dependencies:
Offensive Speech Classifier or similar (e.g., Perspective
API777www.perspectiveapi.com Output: Distribution of toxicity, disaggregated
by social identity Action: Flag high toxicity. Consider rebalancing harmful
associations that emerge in model evaluations

Image

####  C.3.4 Perceived Social Identity Features X Top Image Features

Task: Calculate co-occurrences between perceived social identity features and
image features, such as objects and other people. Analysis Object: Inferred
Image Signals Effort: Low Dependencies: Object Classifier (e.g., Google Cloud
Vision API888https://cloud.google.com/vision/docs/object-localizer) Output:
List of perceived social identity features and the top co-occurrences for
each. Action: Flag stereotypical associations. Consider rebalancing harmful
associations that emerge in model evaluations.

####  C.3.5 Perceived Social Identity Features X Sexual Imagery

Task: Calculate co-occurrences between perceived social identity features and
sexual imagery. Analysis Object: Inferred Image Signals Effort: Low
Dependencies: Visual Content Classifier (e.g., Google Cloud Vision
API999https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation
Output: Proportion of images depicting sexual content disaggregated by social
identity features Action: Flag sexual content. Consider filtering sexual
content depending on downstream tasks and to avoid problematically-sourced
dataset content.

####  C.3.6 Perceived Social Identity Features X Violent Imagery

Task: Calculate co-occurrences between perceived social identity features and
violent imagery. Analysis Object: Inferred Image Signals Effort: Low
Dependencies: Visual Content Classifier (e.g., Cloud Vision
API101010https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation
Output: Proportion of images depicting violent content disaggregated by social
identity Action: Flag violent content. Consider filtering sexual content
depending on downstream tasks and to avoid problematically-sourced dataset
content

####  C.3.7 Perceived Social Identity Features X Hateful Symbols

Task: Calculate co-occurrences between perceived social identity features and
hateful symbols detected in images. Analysis Object: Inferred Image Signals
Effort: Not currently possible Dependencies: Hateful Symbol Classifier Output:
Proportion of images depicting sexual content disaggregated by social identity
Action: Flag hateful content. Consider filtering hateful content depending on
downstream tasks.

Text-Image

####  C.3.8 Social Identity Terms X Sexual Imagery

Analysis Object: Social Identity Term List + Inferred Image Signals Task:
Calculate co-occurrences between social identity terms and sexual imagery.
Effort: Low Dependencies: Visual Content Classifier (e.g., Cloud Vision
API111111https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation
Output: Proportion of images depicting sexual content disaggregated by social
identity term Action: Flag associations. Consider rebalancing harmful
associations that emerge in downstream model evaluations

####  C.3.9 Social Identity Terms X Violent Imagery

Analysis Object: Social Identity Term List + Inferred Image Signals Task:
Calculate co-occurrences between social identity terms and violent imagery.
Dependencies: Visual Content Classifier (e.g., Cloud Vision
API121212https://cloud.google.com/vision/docs/reference/rest/v1/images/annotate#safesearchannotation
Output: Proportion of images depicting violent content disaggregated by social
identity term Action: Flag associations. Consider rebalancing harmful
associations that emerge in downstream model evaluations Effort: Low

####  C.3.10 Perceived Social Identity Features X Top Text Tokens

Analysis Object: Inferred Image Signals + Text Task: Calculate co-occurrences
between perceived social identity signals and tokens in associated text.
Effort: Low Dependencies: Perceived Social Identity Classifiers Output: List
of most frequent co-occurrences for each social identity signal Action: Flag
associations. Consider rebalancing harmful associations that emerge in
downstream model evaluations

####  C.3.11 Perceived Social Identity Features X Offensive Speech

Analysis Object: Inferred Image Signals + Inferred Text Signals Task:
Calculate co-occurrences between perceived social identity signals and tokens
in associated text. Effort: Low Dependencies: Offensive Speech Classifier or
similar (e.g., Perspective API131313www.perspectiveapi.com Output:
Distribution of toxicity, disaggregated by social identity Action: Flag high
toxicity. Consider rebalancing harmful associations that emerge in model
evaluations

####  C.3.12 Perceived Social Identity Features X Topic

Analysis Object: Inferred Image Signals + Inferred Text Signals Task:
Calculate co-occurrences between perceived social identity signals and tokens
in associated text. Effort: Low Dependencies: Topic Classifier (e.g., Google
Cloud Content Categories141414https://cloud.google.com/natural-
language/docs/categories) Output: Distribution of toxicity, disaggregated by
social identity Action: Flag high toxicity. Consider rebalancing harmful
associations that emerge in model evaluations