Laura Weidinger<EMAIL_ADDRESS>arxiv.org/abs/123 001

# Sociotechnical Safety Evaluation of Generative AI Systems

Laura Weidinger  Google DeepMind, London N1C 4DN, United Kingdom  Maribeth
Rauh  Google DeepMind, London N1C 4DN, United Kingdom  Nahema Marchal  Google
DeepMind, London N1C 4DN, United Kingdom  Arianna Manzini  Google DeepMind,
London N1C 4DN, United Kingdom  Lisa Anne Hendricks  Google DeepMind, London
N1C 4DN, United Kingdom  Juan Mateos-Garcia  Google DeepMind, London N1C 4DN,
United Kingdom  Stevie Bergman  Google DeepMind, London N1C 4DN, United
Kingdom  Jackie Kay  Google DeepMind, London N1C 4DN, United Kingdom  Conor
Griffin  Google DeepMind, London N1C 4DN, United Kingdom  Ben Bariach  Google
DeepMind, London N1C 4DN, United Kingdom  Iason Gabriel  Google DeepMind,
London N1C 4DN, United Kingdom  Verena Rieser  Google DeepMind, London N1C
4DN, United Kingdom  William Isaac  Google DeepMind, London N1C 4DN, United
Kingdom

###### Abstract

Generative AI systems produce a range of risks. To ensure the safety of
generative AI systems, these risks must be evaluated. In this paper, we make
two main contributions toward establishing such evaluations. First, we propose
a three-layered framework that takes a structured, sociotechnical approach to
evaluating these risks. This framework encompasses capability evaluations,
which are the main current approach to safety evaluation. It then reaches
further by building on system safety principles, particularly the insight that
context determines whether a given capability may cause harm. To account for
relevant context, our framework adds human interaction and systemic impacts as
additional layers of evaluation. Second, we survey the current state of safety
evaluation of generative AI systems and create a repository of existing
evaluations. Three salient evaluation gaps emerge from this analysis. We
propose ways forward to closing these gaps, outlining practical steps as well
as roles and responsibilities for different actors. Sociotechnical safety
evaluation is a tractable approach to the robust and comprehensive safety
evaluation of generative AI systems.

###### keywords:

Evaluation, Sociotechnical, Generative AI, Multimodal

## Acknowledgements

We thank Simon Osindero, Sasha Brown, Matt Botvinick, Canfer Akbulut, Suresh
Venkatasubramanian, Victor Ojewale, Fernando Diaz, Olivia Wiles, Doug Fritz,
Courtney Biles, Nicklas Lundblad, Neil Rabinowitz, Jenny Brennan, Sunipa Dev,
Don Wallace, Ramona Comanescu, Mark Díaz, Michal Lahav, Alex Kaskasoli,
Isabela Albuquerque, Seliem El-Sayed, and Rida Qadri for their feedback and
contributions to this work.

###### Contents

  1. 1 Introduction
  2. 2 Framework for sociotechnical AI safety evaluation
    1. 2.1 Layer 1: Capability
    2. 2.2 Layer 2: Human interaction
    3. 2.3 Layer 3: Systemic impact
    4. 2.4 Summary
  3. 3 Current state of sociotechnical safety evaluation
    1. 3.1 Taxonomy of harm
      1. 3.1.1 Multimodality raises new evaluation challenges
    2. 3.2 Mapping the landscape
      1. 3.2.1 Limitations
    3. 3.3 Evaluation gaps
  4. 4 Closing evaluation gaps
    1. 4.1 Operationalising risks
      1. 4.1.1 Ensuring validity
    2. 4.2 Selecting evaluation methods
      1. 4.2.1 Capability evaluation methods
      2. 4.2.2 Human interaction evaluation methods
      3. 4.2.3 Systemic impact evaluation methods
    3. 4.3 Practical steps to closing the multimodal evaluation gap
      1. 4.3.1 Repurposing evaluations for new modalities
      2. 4.3.2 Transcribing non-text output for text-based evaluation
      3. 4.3.3 Model-driven evaluation may fill gaps
  5. 5 Discussion
    1. 5.1 Benefits of a sociotechnical approach
    2. 5.2 Roles and responsibilities
    3. 5.3 Limits of evaluation
      1. 5.3.1 Evaluation is incomplete
      2. 5.3.2 Evaluation is never value-neutral
    4. 5.4 Steps forward
      1. 5.4.1 Evaluations must be developed where they do not yet exist
      2. 5.4.2 Evaluations must be done as a matter of course
      3. 5.4.3 Evaluation must have real consequences
      4. 5.4.4 Evaluations must be done systematically, in standardised ways
      5. 5.4.5 Toward a shared framework for AI safety
  6. 6 Conclusion
  7. A Appendix
    1. A.1 Taxonomy of harm
    2. A.2 Evaluation methods per layer
      1. A.2.1 Capabilities layer
      2. A.2.2 Human interaction layer
      3. A.2.3 Systemic impact layer
    3. A.3 Case study: Misinformation
      1. A.3.1 Capability
      2. A.3.2 Human interaction
      3. A.3.3 Systemic impacts

### Reader’s guide

This is a long document. Depending on your background and interests, we
recommend different reading strategies:

  * •

Two-minute read: Look at figure 2.1 (p.2.1) that illustrates our three-layered
evaluation framework, and figures 3.1-3 (p.3.1) which depict the current state
of safety evaluations.

  * •

Ten-minute read: Read the abstract and skim section 2 (p.2), which introduces
our three-layered evaluation framework; look at figures 3.1-3 (p.3.1) which
depict the current state of safety evaluations.

  * •

Evaluators: Skim section 2 (p.2), where we introduce our three-layered
evaluation framework, and section 3 (p.3) where we survey the current state of
safety evaluation; dedicate most time to section 4 (p.4) on practical steps to
closing evaluation gaps, and to the case study (p.A.3) on evaluating
misinformation that puts our evaluation framework into practice. Read about
evaluation as a practice of responsible innovation in the discussion (p.5),
and about the limitations of specific evaluation methodologies (p.A.2).

  * •

People steering AI labs: Look at figure 2.1 (p.2.1) that illustrates our
three-layered evaluation framework, read section 3 (p.3), which outlines gaps
in the current state of safety evaluation of generative AI systems, and look
at figure 5 (p.5.1) that illustrates the roles & responsibilities of different
actors. Consider the limitations of evaluation methods laid out in the section
on evaluation methodologies (p.A.2) and our case study (p.A.3) on evaluating
misinformation that puts our evaluation framework into practice.

  * •

Public policy makers: Look at figure 2.1 (p.2.1) that illustrates our three-
layered evaluation framework, skim section 3 (p.3), which lays out the state
of evaluation today; and read the part on roles and responsibilities (p.5.2)
in the discussion section.

  * •

AI researchers: Consider the evaluation framework in section 2 (p.2), concrete
ways forward as introduced in section 4 and in the case study (p.A.3), and the
limitations and implications in the discussion section (p.5).

##  1 Introduction

Generative111By “generative” we refer to AI systems that generate novel output
rather than analysing existing data (Huang et al., 2022). We focus on
generative AI systems, which we define as models that input and output any
combination of image, audio, video, and text. This includes transformer-based
systems, such as large language models, diffusion-based systems, and hybrid
architectures., multimodal222By “multimodal” we refer to models that accept
and produce output in any combination of image, audio, and text. This includes
models that accept or produce output in more than one modality, such as
interleaved image and text data, or audiovisual data. AI systems333By “AI
system” we refer to a pre-trained base model or foundation model, potentially
“fine-tuned” by adapting it to particular datasets for specific performance
targets, including via practices such as RLHF. AI systems may also include
filters such as input or output filters. AI systems are ready for integration
into a product. are becoming increasingly widely used. Real-world applications
of generative AI systems are proliferating across domains, ranging from
medical applications (Singhal et al., 2023; Nori et al., 2023) to news and
politics (e.g. Bruell (2023)) and social interaction such as companionship
(e.g. Pentina et al. (2023); Griffith (2023)). Early systems produced output
in single modalities, such as image generation (Rombach et al., 2022; Ramesh
et al., 2021) and text capabilities, producing compelling natural language
(OpenAI, 2023b; Glaese et al., 2022; Anil et al., 2023). Increasingly,
generative AI systems in other modalities such as audio, including voice and
music (Oord et al., 2016; Dhariwal et al., 2020; Borsos et al., 2023;
Agostinelli et al., 2023; Huang et al., 2023), and video and audiovisual
capabilities are steadily improving (Du et al., 2023). Generative AI systems
are increasingly multimodal, and their integration into various aspects of
life is anticipated (Google Research, 2023).

In addition to creating benefits, generative AI systems pose risks of harm.
For individual modalities, these risks have been mapped out in different
taxonomies (Bommasani et al., 2022; Weidinger et al., 2021; Liu et al., 2023b;
Bird et al., 2023; Solaiman et al., 2023; Shevlane et al., 2023; Barnett,
2023; Shelby et al., 2023; Dinan et al., 2021) as well as in research on
individual risks or applications (e.g. Luccioni et al. (2023); Bianchi et al.
(2023); Birhane et al. (2021); Carlini et al. (2023a); Shevlane et al. (2023);
Khlaaf et al. (2022)). Complementing foresight research, observed instances of
harm from generative AI systems have been logged to identify risks that these
systems create (AI Incident Database, ; Organisation for Economic Co-operation
and Development, b). Now that risks from generative AI systems have been
identified, their impact on the overall safety of a generative AI system must
be understood. This requires evaluation.

The growing use of generative AI systems makes it both easier and more
pressing to evaluate potential risks of harm. As these technologies become
widely used and embedded, the risks they create are a public safety concern.
Accordingly, evaluating potential risks from generative AI systems is a
growing priority for AI developers (OpenAI, 2023c; Anthropic, 2023), public
policy makers (The White House, 2023), regulators (UK Task Force, ; EU AI Act,
2023; National Institute of Standards and Technology, 2021b, a), and civil
society (Electronic Privacy Information Center, ).

Evaluation is the practice of measuring AI system performance or impact.
Safety evaluation in particular focuses on evaluating risks of harm or
actualised impacts on people or broader systems. Evaluations can be
exploratory (such as open-ended probing of an AI system) or directed (such as
running a specific test). They include qualitative investigations, such as
studying how people actually attempt to use an AI system, as opposed to
assessing intended use cases. Exploratory evaluations may identify areas of
uncertainty or additional context, or give rise to novel directed evaluation
questions. Directed evaluations follow a series of steps, whereby a target –
such as a risk of harm – is selected, operationalised into an observable
metric, and measured. In any evaluation, the results are then judged against a
normative baseline, such as whether an AI system is “good”, “fair”, or “safe
enough”. Evaluation is never neutral: it rests on interwoven technical and
normative decisions, such as deciding what to evaluate in the first place, how
to measure it (see Operationalising risks), and what results indicate “good”
AI system performance (Bakalar et al., 2021; Bowker and Star, 2000). Safety
evaluation can form part of broader safety audits, which may additionally take
into account organisational governance structures or existing documentation
and more (Raji et al., 2020; Mökander et al., 2023; Costanza-Chock et al.,
2022).

Evaluation performs an important function by providing public safety
assurances. By systematically testing AI systems against potential risks of
harm, evaluation can make AI systems less opaque. Evaluation also sheds light
on, predicts, and quantifies the likelihood of potential downstream harms, and
can surface the factors and mechanisms that influence whether downstream harm
may occur. Evaluations can guide the development of AI systems, as well as
providing assurances on levels of AI system safety in different contexts. As a
result, the understanding of AI systems that evaluations provide is essential
for well-informed, responsible decision-making on AI system development and
deployment (Stilgoe et al., 2013). Further, evaluation of different risk areas
brings to light normative trade-offs that arise as AI systems are developed
and deployed in real-world settings. By performing these functions, evaluation
is a foundation for meaningful accountability on the responsible innovation
and deployment of generative AI systems.

In this paper, we make two main contributions: a sociotechnical framework for
safety evaluation and an empirical assessment of the current safety evaluation
landscape. While the priority of safety evaluations for generative AI systems
is clear, current approaches are often heterogeneous and ad hoc. The
evaluations that are being conducted differ between organisations and AI
systems (e.g. Glaese et al. (2022); Anthropic (2023); OpenAI (2023a); Mishkin
et al. (2022); Anil et al. (2023)), which makes them hard to compare and
reproduce. This can also mean that the evaluation of a given AI system misses
important risks that should be considered. We argue that a more systematic and
standardised approach to safety evaluation is necessary to ensure meaningful,
comparable, and comprehensive safety evaluation (c.f. Vogel and Manyika
(2023); Liang et al. (2022); National Institute of Standards and Technology
(2019)). As a step in that direction, we offer a sociotechnical framework to
guide safety evaluation of generative AI systems.

Our second main contribution is a review of the current state of safety
evaluations for generative AI systems, including the public release of a
repository of existing evaluations, and analysing gaps. These gaps are
tractable: we present practical steps toward closing them. We propose roles
and responsibilities for different stakeholders and discuss how currently
disparate communities with interests in the safety of AI systems can
intersect.

The paper proceeds as follows. In section 2, we outline our proposed framework
for sociotechnical safety evaluation across three layers that progressively
add context: the capability layer, the human interaction layer, and the
systemic impact layer. Section 3 surveys the current state of research and
practice in sociotechnical safety evaluation, identifying strengths and
limitations of existing approaches. Section 4 builds on the framework and
survey to discuss ways to close observed evaluation gaps. In section 5, we
conclude with a discussion of open questions for the field of safety
evaluations, including the varying roles and responsibilities between AI
developers and public sector stakeholders for conducting evaluations across
the layers, and the connections between the range of proposed approaches to
safety associated with generative AI systems.

##  2 Framework for sociotechnical AI safety evaluation

Recent research identified a sociotechnical gap in our understanding of the
safe development and deployment of AI systems (Lazar and Nelson, 2023; Shelby
et al., 2023; Mohamed et al., 2020; Selbst et al., 2019). This sociotechnical
gap arises where AI system safety is evaluated only with regard to technical
components of an AI system, i.e. individual technical artefacts such as data,
model architecture, and sampling strategies. While these are important aspects
of AI safety evaluation, they alone are insufficient to determine whether an
AI system is safe. Instead, an approach is needed that takes into account
human and systemic factors that co-determine risks of harm.

To close this gap, we apply a sociotechnical lens to AI safety evaluation.
Sociotechnical research has a long-standing history in expanding the frontiers
of AI system evaluation to include human and systemic factors (Raji et al.,
2020; Ekstrand et al., 2018; Dwork et al., 2012; Friedman and Nissenbaum,
1996; Barocas and Selbst, 2016). This approach is rooted in the observation
that AI systems are sociotechnical systems: both humans and machines are
necessary in order to make the technology work as intended (Selbst et al.,
2019). The interaction of technical and social components determines whether
risk manifests (Leveson, 2012). Consequently, AI evaluation requires a
framework that integrates these components and their interactions.

Similarly to other sociotechnical work (c.f. Raji et al. (2020); Rismani et
al. (2023)), our approach is further inspired by a system safety approach from
the discipline of safety engineering. System safety represents a historical
paradigm shift in safety engineering, from component-based approaches toward
systems thinking, taking into account broader contexts, interactions and
emergent properties of complex systems (Leveson, 2012). Component-based
approaches to safety emerged historically from industries dealing with
hazardous materials in constrained settings and processes. These approaches
isolate system components or steps in a process and assess individual failure
modes of each part. The sum of these assessments is then considered a
comprehensive safety evaluation of the entire system. This approach is not fit
for purpose in complex and versatile systems, such as where software and
people interact with great degrees of freedom (Hutchins, 1995; Leveson, 2012).
Here, a component-based approach to safety needs to give way to a system-based
approach to safety (Leveson, 2012). An analogous shift is required in the
safety evaluation of generative AI systems. We propose a framework that
accounts for this shift.

Specifically, we present a three-layered framework to structure safety
evaluations of AI systems. The three layers are distinguished by the target of
analysis. The layers are: capability evaluation, human interaction evaluation,
and systemic impact evaluation. These three layers progressively add on
further context that is critical for assessing whether a capability relates to
an actual risk of harm or not.

To illustrate these three evaluation layers, consider the example of
misinformation harms. Capability evaluation can indicate whether an AI system
is likely to produce factually incorrect output (e.g. Lin et al. (2022); Ji et
al. (2023)). However, the risk of people being deceived or misled by that
output may depend on factors such as the context in which an AI system is
used, who uses it, and features of an application (e.g. whether synthetic
content is effectively signposted). This requires evaluating human-AI
interaction. Misinformation risks raise concerns about large-scale effects,
for example on public knowledge ecosystems or trust in shared media. Whether
such effects manifest depends on systemic factors, such as expectations and
norms in public information sharing and the existence of institutions that
provide authoritative information or fact-checking. Evaluating misinformation
risks from generative AI systems requires evaluation at each of these three
layers.

The three layers in this framework interact and their boundaries are gradual.
Effects detected at one layer may be indicate related observations at the
next. For example, discrepancies in how an AI system performs for different
user groups can be identified at the layer of human interaction, and may
foreshadow disparate systemic impacts for these groups. A further illustration
of the gradual boundaries between the layers is that evaluation methods can
straddle multiple layers. For example, adversarial testing is a method for
evaluating capabilities. However, by focusing on the experience of the
adversarial tester, it can be a measure of human interaction: specifically of
the friction a person encounters when trying to use an AI system to malicious
ends. While interactions between these layers may extend beyond the failure of
individual system components and be complex, they are often still within the
control of AI system developers.

In addition, there are feedback loops within and between layers. For example,
societal context may feed back into system capabilities via the opinions and
demographics of human annotators, as annotated data is used to adapt AI
systems to particular contexts. Note that the layers are not ordered by
importance nor in any chronological order. Rather, evaluations at each layer
can be performed simultaneously and asynchronously. We now introduce the
three-layered framework in detail.

###  2.1 Layer 1: Capability

Evaluation at this layer targets AI systems and their technical
components.444These include training data; model components, such as model
architectures and classifiers; the model itself, such as pre-training
embeddings; and model outputs, such as images. These are routinely evaluated
in isolation, including tests of AI system behaviour in response to novel
tasks or environments, or testing individual technical artefacts, such as the
data that an AI system is trained on. It also includes evaluating processes by
which these artefacts are created, such as the aggregation mechanisms in
processes that are used to adapt an AI system to a particular task. In
addition to assuring the safety of an AI system, evaluations at this layer are
often performed to guide iterative model development ("hill-climbing").

While evaluation at this layer does not assess downstream harm per se, it can
provide indication of whether a component, output, or AI system is likely to
cause downstream harm. Several risks of harm can be evaluated by measuring
capabilities through the outputs of an AI system. This includes, for example,
the extent to which an AI model reproduces harmful stereotypes in images or
utterances (representation harms (Bianchi et al., 2023)), makes factual
errors, or displays advanced capabilities that present safety hazards. The
extent to which model performance deteriorates when prompted in different
languages, about different groups, or in different domains can also be
evaluated at this layer and can be indicative of the likely distribution of
potential downstream harm. Capabilities also include metrics that are designed
to track efficiency and may shed light on potential downstream environmental
impact, such as energy use at inference (Kaack et al., 2022). Capabilities can
be assessed against fixed, automated tests or probed dynamically by human or
automated adversarial testers (see Selecting evaluation methods).

Evaluations at this layer can also concern the data on which a model is
trained. Using tools to visualise clusters or associations in the training
data can reveal diversity and representativeness of the data, or the presence
of sensitive data such as private information (Dodge et al., 2021; Choi et
al., 2023; Wang et al., 2022; Kreutzer et al., 2022). Similar tools can be
used to assess the learned associations of a trained AI system (Caliskan et
al., 2017; Steed and Caliskan, 2021).

Other components that can be analysed at this layer include filters and
techniques used to reduce output that may relate to a particular risk of harm,
such as filters for safety harms in images (Rando et al., 2022) or toxic
language (Perspective API, ). However, such filters have limitations (Rando et
al., 2022) that can aggravate representation harm by disproportionately
filtering out content from some groups (Welbl et al., 2021).

Capability evaluation is critical, but insufficient, for a comprehensive
safety evaluation. It can serve as an early indicator of potential downstream
harms, but to assess whether or not a capability relates to risks of harm
requires taking into account context – such as who uses the AI system, to what
end, and under which circumstances. This context is assessed at subsequent
layers.

Figure 2.1: A sociotechnical framework for safety evaluation comprises three
layers.

###  2.2 Layer 2: Human interaction

This layer centres the experience of people interacting with a given AI
system. Assessing AI system safety requires evaluating not just the AI system
in isolation but also effects on people interacting with AI systems, and the
human–AI dyad. This includes usability: does the AI system perform its
intended function at the point of use, and how do experiences differ between
user groups? This layer also centres potential externalities: does human–AI
interaction lead to unintended effects on the person interacting or exposed to
AI outputs? Evaluation at this layer acknowledges that AI system safety
depends on who uses an AI system, with what goal in mind, and in what context.
This layer shifts the lens to the humans interacting with an AI system and is
key to a human-centred approach to AI development (Liao and Vaughan, 2023;
Tahaei et al., 2023; Vaughan and Wallach, 2021).

In addition to testing AI system capabilities, the functionality of an AI
system in the context of a concrete application must be assessed (Raji et al.,
2022a). This includes testing how different people actually use the system, as
real-world use often deviates from intended use cases. User groups are
heterogeneous, and safety evaluation requires not only assessing whether an AI
system works but also for whom it works well (Wang et al., 2023). To assess
usability in practice, human interaction with AI systems needs to be evaluated
“in the wild”, i.e. in a real-world application context for an AI system such
as a hospital (Sendak et al., 2020) or police units (Marda and Narayan, 2020).
Evaluating the human–AI interaction can also reveal how easy it is to use a
model for malicious ends (Roy and Umbach, 2023).

Human-centred testing can shed light on potential externalities created by
specific use cases or applications of AI systems. To assess a risk of harm,
directed psychology or human–computer interaction experiments can be
performed. Under controlled, safe conditions, potential harmful outcomes to
people interacting with AI can be studied, such as overreliance on AI systems
(Chiesurin et al., 2023) or overtrust (e.g. due to AI systems endowed with
anthropomorphic cues, Glikson and Woolley (2020)). Some effects may only
manifest over time and require longitudinal evaluation. For example, one
experiment found that repeated exposure to AI content increases its
persuasiveness but only up until a certain point (Cacioppo and Petty, 1980).
Regarding another risk area, it has also been hypothesised that increased
feelings of social isolation due to overuse of technology may only show up
after frequent exposure to an AI system, in between a user’s interactions
(Turkle, 2011). Evaluation at this layer can also assess harms to data
annotators, as they are exposed to harmful model outputs, including via
surveys or interviews (Stoev et al., 2023; Gray and Suri, 2019). In addition,
human interaction evaluations may reveal disparate harm profiles for different
modalities. For example, one set of evaluations found that users more readily
believe synthetic misinformation that is presented in video as opposed to text
modalities (Sundar et al., 2021).

Evaluations at this layer can also identify psychological mechanisms by which
harms may occur to a person interacting with an AI system. For example, they
may identify cognitive biases that influence people coming to believe
misinformation (Jerit and Zhao, 2020), or how AI systems influence or persuade
humans over the course of an interaction, such as when co-writing a text
(Jakesch et al., 2023; Hohenstein et al., 2023).

Finally, evaluation that considers an AI system in the context of use can
assess the overall performance of the human–AI dyad, such as quality of
outcomes on AI-assisted computer coding tasks compared to a human–human
baseline (Vasconcelos et al., 2023).

While this layer provides critical context by adding human interaction to the
evaluation, it remains insufficient for a comprehensive AI safety assessment.
It provides limited insights on the potential broader impacts that an AI
system may have when deployed at scale, and does not consider risks and
impacts on broader systems such as society, economic impacts, or the natural
environment. Assessing these effects requires analysing the broader systems
into which an AI system is deployed, at the third and final layer of our
sociotechnical framework for safety evaluation.

###  2.3 Layer 3: Systemic impact

The third target of evaluation is the impact of an AI system on the broader
systems in which it is embedded, such as society, the economy, and the natural
environment. Widely used AI systems shape, and are shaped by, the societies in
which they are used (Matias, 2023; Wagner et al., 2021). Detecting the effects
from these interactions requires evaluation at the system layer. Some effects
may only emerge as an AI system is deployed at large scale. For example, risks
from increasing homogeneity in knowledge production and creativity due to
“algorithmic monocultures” are emergent at the systems layer of evaluation
(Toups et al., 2023; Kleinberg and Raghavan, 2021; Doshi and Hauser, 2023).
Harms may also have small effect sizes that are hard to detect at the layer of
individual user interactions but become salient at a systems level (e.g.
Bulimia Project ). Evaluating these risks and impacts requires focusing on the
broader systems into which the AI system is integrated.

Evaluation at this layer can target systems of different domains and sizes.
Economic assessments may concern broad systemic impacts, such as the labour
market impacts of generative AI (Tolan et al., 2021; Frank et al., 2019; Frey
and Osborne, 2013; Felten et al., 2021; Eloundou et al., 2023) or the impact
of model adoption on productivity (Brynjolfsson et al., 2023). It may also
centre specific industries or goods, for example by evaluating impacts on the
creative economy or predicting the likely impacts of generative AI on the
erosion of public goods such as the creative commons (del Rio-Chanona et al.,
2023; Huang and Siddarth, 2023). Impact from generative AI systems on societal
institutions, such as political polarisation or changes to trust in public
media, can be evaluated through system evaluation (Lorenz-Spreen et al.,
2023). The fairness of how benefits and risks are distributed can also be
ascertained at this layer, for example by assessing take-up of AI tools across
countries (Calvino and Fontanelli, 2023) and identifying who is able to
capture and extract most value using these technologies (Brynjolfsson et al.,
2023).

Evaluations at this layer may also focus on smaller, more localised systems,
such as assessing impacts from an AI system in a clinical context on the
provision of care (Elish and Watkins, 2020). Evaluating how AI systems are
socially embedded can shed light on how people come to trust the outputs – for
example, where friends and colleagues all use a system, this system may be
trusted more. One common concern with the widespread availability of
generative AI systems is that they can be used to cheat on school assignments
(Rudolph et al., 2023). System-level evaluation of adoption and perception of
AI systems can evaluate what types of use occur and under what circumstances,
and by whom they constitute ’cheating’. Environmental impacts can be targeted
at this layer, to provide a nuanced understanding of impacts on broader
ecosystems. For example, detailed and localised evaluation can reveal the
actual environmental impact from generative AI systems, such as from data
centres that rely on nearby water sources for cooling (Luccioni et al., 2022).
Early stage indicators, such as energy use as a proxy for environmental impact
at the capability layer, can be calibrated and contextualised via evaluations
at this layer.

Evaluation at this layer can also provide context-rich assessments of
interactions of different systems as social, economic, and ecological factors
intersect. For example, evaluation at the system layer may take into account
the biodiversity and resilience of local ecosystems, the nature of the energy
grid, and the social and economic implications for local communities in order
to assess overall harm of infrastructure that powers an AI system (Solaiman et
al., 2023).

Systemic impacts are often difficult to assess due to the complex nature,
idiosyncrasies, and noise of the systems that are being evaluated. While
direct impacts of an AI system may not be known until post deployment,
forecasts or comparable technologies can provide initial insights on potential
risks of harm at this layer.

###  2.4 Summary

We present a three-layered sociotechnical framework for safety evaluation of
generative AI systems. (While we focus on generative AI systems, this
framework may also be applicable to other types of AI.) The same high-level
risk areas can be detected and evaluated at each layer (we outline practical
steps toward evaluating risks at each layer in section 4). What connects the
three layers is that they progressively add on further context. Note that they
are not sequential or dependent on each other. Neither are the layers
conditional on each other; rather, evaluation at each layer can be run
simultaneously. Integrating results from all layers provides a comprehensive
evaluation of the safety of a generative AI system. The layers are a guiding
structure to facilitate evaluation along different layers of context in a
sociotechnical system.

##  3 Current state of sociotechnical safety evaluation

In this section, we survey the state of safety evaluation of generative AI
systems. This first requires consolidating a taxonomy of potential harm from
these AI systems. We present a synthesised taxonomy of harm based on prior
literature of taxonomies of harm from generative AI systems. Next, we employ
an extensive process to identify all existing evaluations of generative AI
systems that speak to risks identified in our taxonomy. We map all identified
evaluations by risk area; by AI system modalities (image, audio, video, text,
and multimodal combinations); and by layers of evaluation based on the three-
layered framework introduced above. This mapping is presented in an overview
figure that snapshots the sociotechnical evaluation landscape today. We close
this section by discussing the “evaluation gaps” this mapping has surfaced.

###  3.1 Taxonomy of harm

First, to assess the state of sociotechnical safety evaluation for generative
AI systems requires grounding the types of risk that such evaluation should
assess. To this end, we revisit the growing literature on social, ethical, and
other safety risks from generative AI systems and integrate insights from this
literature into a single, holistic taxonomy (high-level version, table 1;
detailed version, appendix section A.1). Rather than presenting a novel
research artefact, the goal of this taxonomy is to provide a basis for mapping
the state of safety evaluation of generative AI systems.

Previous work identified a wide range of risks posed by generative AI systems.
Existing taxonomies address risks from AI systems audio (Barnett, 2023) and
text (Bommasani et al., 2022; Weidinger et al., 2021; Liu et al., 2023b), as
well as combined modalities, such as text-to-image (Bird et al., 2023).
Solaiman et al. (2023) provide an overview of harms from generative AI systems
writ large and describe approaches to social impact analyses for each
identified harm area. In our overview, we include both established and
emerging risks. Established risks are defined by observed instances of harm,
such as representation risks (e.g. Luccioni et al. (2023); Bianchi et al.
(2023); Birhane et al. (2021)). Emerging risks are anticipated based on the
foreseeable capabilities of generative AI systems, such as increasingly
persuasive content produced by generative AI (Matz et al., 2023; Shevlane et
al., 2023).

We build on this prior literature to aggregate a single, holistic taxonomy of
harm from generative AI systems. This taxonomy has six high-level harm areas:
1. Representation & Toxicity Harms, 2. Misinformation Harms, 3. Information &
Safety Harms, 4. Malicious Use, 5. Human Autonomy & Integrity Harms, 6.
Socioeconomic & Environmental Harms (see table 1).

Table 1: High-level overview of risks of harm from generative AI systems Harm area |  Definition |  Example
---|---|---
Representation & Toxicity Harms |  AI systems under-, over-, or misrepresenting certain groups or generating toxic, offensive, abusive, or hateful content |  Generating images of Christian churches only when prompted to depict “a house of worship” (Qadri et al., 2023a)
Misinformation Harms |  AI systems generating and facilitating the spread of inaccurate or misleading information that causes people to develop false beliefs |  An AI-generated image that was widely circulated on Twitter led several news outlets to falsely report that an explosion had taken place at the US Pentagon, causing a brief drop in the US stock market (Alba, 2023)
Information & Safety Harms |  AI systems leaking, reproducing, generating or inferring sensitive, private, or hazardous information |  An AI system leaks private images from the training data (Carlini et al., 2023a)
Malicious Use |  AI systems reducing the costs and facilitating activities of actors trying to cause harm (e.g. fraud, weapons) |  AI systems can generate deepfake images cheaply, at scale (Amoroso et al., 2023)
Human Autonomy & Integrity Harms |  AI systems compromising human agency, or circumventing meaningful human control |  An AI system becomes a trusted partner to a person and leverages this rapport to nudge them into unsafe behaviours (Xiang, 2023)
Socioeconomic & Environmental Harms |  AI systems amplifying existing inequalities or creating negative impacts on employment, innovation, and the environment |  Exploitative practices to perform data annotation at scale where annotators are not fairly compensated (Stoev et al., 2023)

We present a high-level overview of this taxonomy below (table 1), with
examples of how these risks may manifest in modalities other than text. A more
detailed breakdown of each risk area is provided in appendix section A.1.

####  3.1.1 Multimodality raises new evaluation challenges

While none of the higher-level risk areas are new in multimodal as opposed to
text-based generative AI systems, the specific ways in which they may manifest
are likely to differ between modalities. For example, violent or sexually
explicit content has a greater “shock factor” in image modalities than in
text. Multimodal models may also introduce novel evaluation challenges. For
example, consider a text-to-image AI system that produces images based on text
input. Hutchinson et al. (2022) argue that text often underspecifies context
such that an image of a “wedding”, for example, will necessarily include
certain objects and cultural contexts, regardless of whether these concepts
are articulated in the accompanying text. It may be easier for an AI system to
hedge or give pluralist output in text than in images.

Risks may also be compositional, i.e. manifest through the very combination of
output across modalities. For example, pairing the caption “these smell bad”
next to an image of a skunk is not harmful, but the same caption next to an
image of a group of people may constitute harassment (Kiela et al., 2021b).
Similarly, an innocuous video of military training exercises combined with
audio describing the invasion of a country risks creating an instance of
misinformation (Vincent, 2023). Generative AI systems may also perpetuate
stereotypes in ways that are highly dependent on domains – for example, by
overrepresenting nude females as compared to nude males in the context of
synthetic music videos. Detecting these harms may build on existing methods
but is likely to require novel context-sensitive evaluation approaches. Though
risk evaluation will likely draw from lessons learned evaluating models in
single modalities, novel evaluations that enable a holistic view across
modalities are required to capture risks in multimodal AI systems.

###  3.2 Mapping the landscape

We now present the results of a large-scale review of existing benchmarks to
assess risks of harm from generative AI systems. To write this section, our
group of co-authors and reviewers assembled an overview of all sociotechnical
safety benchmarks and evaluation methods known to this group up to 10 October
2023.

Included are academic papers or online reports that meet two criteria: they
constitute an evaluation, and they have been applied to a generative AI
system. An evaluation is defined as either a set of model inputs, such as a
dataset, and a metric; or the application of a method (e.g. red teaming a
specific AI system or a human–computer interaction study). It has been applied
to a generative AI system if the publication describes results from its
application to a generative AI system. Note that evaluations that may be
applicable to generative AI systems but have not been applied to such systems
yet were not in scope for this review.

All submitted evaluations were coded based on output modality – whether they
evaluate text, image, audio, video, or multiple modalities. They were then
coded based on what risk area they cover, based on the above taxonomy.
Finally, they were coded by layers of evaluation in our three-layered
framework (1-capability, 2-human interaction, 3-systemic impact).

We release an evaluation repository of all included evaluations as an open
resource here. The contents of this survey evaluation repository are presented
in an overview figure (figure 3.1).

####  3.2.1 Limitations

While great efforts were made to conduct a large-scale review of existing
evaluation approaches, we do not assume that this mapping is comprehensive.
Our approach is further limited by not considering input modality: our coding
is based on output modality. Future mappings may distinguish between input
modalities for a more fine-grained analysis (e.g. mapping ‘image-to-text’
evaluations distinctly from ‘text-to-text’ evaluations). Finally, our mapping
is a snapshot of a moment in time. In the future, it may be conducive to a
thriving ecosystem on sociotechnical evaluations to expand the evaluation
repository into a living resource that evaluation developers can add their
methods to.

###  3.3 Evaluation gaps

Inspecting the state of safety evaluations applied to generative AI systems
reveals three high-level gaps:

  1. 1.

Coverage gap: Evaluations for several risks are lacking. Coverage of ethical
and social risk evaluation overall is low. Several gaps exist where there are
few or no evaluations to assess a given risk area.

  2. 2.

Context gap: Human interaction and systemic evaluations are rare. Existing
evaluations cluster in the text modality, with fewer evaluations available for
audio, image, video, or combinations of modalities. This presents a challenge
for evaluating social and ethical risks in other modalities.

  3. 3.

Multimodal gap: Evaluations are missing for multimodal AI systems. Most
evaluations of social and ethical harms that were identified cluster at the
layer of capability evaluations. We now discuss these observations in turn.

Figure 3.1: Evaluations per harm area and AI system output modality. No harm
area is well covered across modalities.

First, we observe that evaluations are scarce for several previously
identified risks from generative AI systems. This lack of coverage is
particularly pronounced for information and safety harms, human autonomy and
integrity harms, and socioeconomic and environmental harms. While the number
of available evaluations is insufficient to assess coverage of a risk area,
the absence of evaluations is a clear signal that the given risk area cannot
currently be evaluated in generative AI systems.

More detailed inspection of the evaluation repository indicates that the lack
of coverage extends beyond these three risk areas: even where evaluations
exist, they do not cover the risk area comprehensively. For example, we
identified 83 evaluations of representation harms. However, these cover only a
small space of representation harms – 17% of them cover binary gender and
occupation bias,555Six of the fourteen focused exclusively on gender and
occupation. The rest include additional demographics and stereotypes. and 60
cover text modalities only. They also cover only a small space of the
potential harm: multiple “discriminatory bias” benchmarks cover binary gender
or skin colour as potential traits for discrimination (Cho et al., 2023;
Mandal et al., 2023). These evaluations do not cover potential manifestations
of representation harms along other axes such as ability status, age,
religion, nationality, or social class. In sum, further evaluations are needed
to cover ethical and social harms, including plugging more nuanced gaps in
risk areas for which some evaluations exist.

Figure 3.2: Evaluations per layer. Human interaction and systemic impact
evaluations to assess generative AI system safety are rare.

Our second main observation is that insofar as evaluation tools exist to
address risks from multimodal generative AI, they are mainly clustered at the
capability layer. More detailed inspection of the repository indicates that
evaluations focus particularly on AI system outputs and to a lesser degree on
available training data. This clustering of evaluation at the capability layer
is reflective of, and likely partially driven by, the evaluations that have
recently been performed and disclosed as part of large generative AI system
announcements, which primarily focus on capability evaluations (Glaese et al.,
2022; Anthropic, 2023; OpenAI, 2023a; Mishkin et al., 2022; Touvron et al.,
2023; Anil et al., 2023).

While a capability-focused approach provides important indications as to
potential downstream harms, it does not account for contextual factors that
co-determine risks of harm (see section 2). Capability evaluation is a core
piece of safety evaluation, but it must be complemented by further analyses
that add layers of relevant context. As a result, further work is needed to
expand sociotechnical evaluations at the human interaction layer and at the
system layer.

Figure 3.3: Evaluations per layer and modality. Most (75%) of all evaluations
target text output.

Our third observation is that the vast majority of evaluations exclusively
assess text. Few evaluations exist for image outputs or combinations of text
and image, and evaluations of audio or video modalities are scarce. There are
only four publicly documented evaluations targeting audio and we did not find
any evaluations targeting video.666Note that there are evaluations for harms
arising in video that have not been applied to generative AI systems and so
did not satisfy the inclusion criteria here (e.g. Das et al. (2023); Wu and
Bhandary (2020); Ashraf et al. (2022)). This may in part be a result of
historical contingencies: generative AI systems that output text saw rapid,
widespread adoption, which may have triggered proportionately more research
into ethical and social risks and corresponding evaluations.

Generative AI systems that produce compelling audio including voice and music
already exist (Oord et al., 2016; Dhariwal et al., 2020; Borsos et al., 2023;
Agostinelli et al., 2023; Huang et al., 2023), and video and audiovisual
capabilities are steadily improving (Du et al., 2023). In particular, the
combination of multiple modalities – through interleaved outputs, such as
articles with supporting imagery; or modalities layered on top of each other,
such as audiovisual video with subtitles – creates different manifestations of
harm across the six identified harm areas. As a result, assessing ethical and
social harm in multimodal models requires novel evaluation approaches. (We
discuss some steps toward this in section 4.)

Critically, this distribution of evaluations centring text modalities is not
driven by a principled assessment of the modalities in which harm is likely to
occur. Several risks have been anticipated in the audio, image, and video
modalities or combinations (see appendix section A.1). For example, the lack
of representation harm evaluations in the audio modality is not driven by a
view that these harms are unlikely to occur. On the contrary, audio training
data is likely to overrepresent some voices and dialects. Analogous to
representation harms in text-based systems, this bias may lead generative AI
systems to produce higher-quality output in some voices and dialects than
others. Such unfair disparities across dialects is well documented in speech
recognition and in speech-to-text models (Ngueajio and Washington, 2022), but
no evaluation tools exist to assess this in generative AI systems. In some
cases, evaluations designed for text output can be repurposed for other
modalities (see section 4). However, this is limited, especially where the
same risk may manifest differently across modalities.

Combinations of modalities can create novel risks as well as compound effects.
For example, misinformation has been found to be more compelling in
audiovisual modalities as opposed to text (Hameleers et al., 2020). AI systems
that span multiple modalities may also be more vulnerable to malicious attacks
aimed at getting a model to create harmful output, as fewer safety mechanisms
and less exploration of vulnerabilities exist for them (Carlini et al.,
2023b). Thus, evaluations must be expanded to modalities other than text. In
addition to evaluating individual modalities in isolation, they must also be
expanded to assess compositions of modalities, i.e. multimodal outputs.

##  4 Closing evaluation gaps

Our assessment of the current state of safety evaluations of generative AI
systems identified significant gaps. In this section we propose practical
steps to close these gaps. These steps are tractable. Closing these gaps will
require work. In addition, it may involve clarifying roles and
responsibilities, which we return to in section 5. This section is primarily
aimed at practitioners and those funding or performing the construction of new
evaluations.

To close identified gaps, new evaluations are needed. In part, this likely
means constructing novel evaluations. The first part of this section presents
the general pipeline and building blocks for constructing such evaluations. In
particular, we describe how rich, multifaceted concepts of harm can be made
measurable through the process of “operationalisation” (see Operationalising
risks). We then outline concrete methodologies that can be used to obtain
measures of a given AI system, for each layer of evaluation (see Selecting
evaluation methods).

In addition to constructing novel evaluations, it may be possible to extend
existing evaluations to generative AI systems. The second part of this section
focuses on practical steps that can be taken to close the gaps in the
evaluation of generative AI systems. We discuss these practical avenues and
their advantages and limitations in the second part of this section (see
Practical steps to closing gaps in safety evaluation).

###  4.1 Operationalising risks

Evaluation is a process involving several steps: it requires first selecting a
target (such as a risk of harm, e.g. “bias”); then operationalising it into a
concrete metric (e.g. the association of gender and occupation, Luccioni et
al. (2023)); then obtaining a measurement; and finally judging the outcome.
Each of these steps has technical and normative elements (see Evaluation is
never value-neutral). In this paper, we lean on previous literature to
identify target constructs – namely, a taxonomy of identified risks of harm
(see appendix section A.1). But how to proceed from a complex, multifaceted
concept such as “misinformation” to a valid, tractable measurement of this
risk? The process of operationally defining risks of harm such that they can
be measured is the focus of this section.

Risks of harm from generative AI systems are often latent constructs that are
not directly observable via a single test or metric (Jacobs and Wallach,
2021).777This also applies to other targets of evaluation, such as cognitive
capacities or potential benefits of AI systems. In order to measure these
risks, they need to be operationally defined (King et al., 2021).
Operationalisation is the process of mapping tractable, observable metrics or
concepts to latent constructs. Measurement on these observable metrics is then
taken to provide insight on the latent target construct. Note that
operationalisation is inherently an ambiguous process. What constitutes a
valid measure of harm is a contestable decision, and often metrics are
iterated on and improved over time (Chang, 2004). Operationalisation may also
constitute normative trade-offs – for example, on how a single performance
metric should weigh false positives against false negatives.
Operationalisation of complex constructs creates various pitfalls that can
result in invalid measurements (see Operationalising risks).

In section 2 above, we argued that risks of harm from generative AI systems
cannot be comprehensively assessed at a single layer of evaluation. Rather,
complementary evaluation at all three layers is needed for a full evaluation.
Thus, we propose operationally defining harm constructs at each of the three
layers of evaluation.

Specifically, this requires mapping metrics or concepts that are observable at
a given layer to the latent harm construct. Different aspects of a risk can be
measured at each layer. Correspondingly, different metrics can be mapped to a
given risk per layer. Metrics can range from single, observable, narrow
metrics (e.g. the FID score to assess the quality of a generated image) to
more open-ended empirical or qualitative metrics (e.g. user preferences or
broader societal impact).

For example, to operationalise the risk of information and safety harms, we
may define the following metrics at each layer. At the capability layer, we
assess properties of model output that indicate potential information hazards,
such as the capability to output harmful biological information (Anthropic,
2023; OpenAI, 2023a). At the human interaction layer, this risk can be
operationalised in multiple ways, one of which might be the likelihood of
people unintentionally following instructions to assemble dangerous compounds
in different contexts, e.g. where the generative AI system is used as a
laboratory assistant. At this layer, the risk likelihood may also be measured
via the friction that people encounter when intentionally trying to apply
dangerous AI capabilities to malicious ends. Finally, at the systemic impact
layer, this risk can be assessed via modelling potential distribution
mechanisms of novel biohazards created based on such information. (These are
examples for illustration purposes; for a more in-depth example, see the
operationalisation of misinformation harms in our Case study: Misinformation.)

####  4.1.1 Ensuring validity

Some information inevitably gets lost when operationalising complex constructs
such that they can be measured – translating risks from AI systems into narrow
metrics and tests is fraught with ambiguity (Wagner et al., 2021). This loss
compromises the validity of a measure. There are different ways in which
validity may be compromised. We briefly canvass these and outline approaches
to mitigating validity concerns.

Tests often do not measure precisely what they seek out to measure: they may
capture only a subset or part of the target construct (internal validity), or
may capture the phenomenon fully in a given instance but not allow
extrapolation to new situations (external validity) (Liao et al., 2021).

AI capability evaluation in particular has been criticised for relying on
overly narrow operational definitions of complex harms, leading to both
internal and external validity failures (Raji et al., 2021; Liao et al.,
2021). Operational definitions must be arrived at carefully and deliberately,
or they risk yielding misleading results. For example, one study found that
the risk of harmful stereotyping in language modelling had been
operationalised as the association of word pairs, but only some of the
referenced word pairs were actually harmful and others were innocuous, such as
the word pair “Norwegian” and “salmon”. As this operationalisation included
instances that were not harmful, the validity of the resulting metric and what
it can say about harmful stereotyping was fundamentally called into question
(Blodgett et al., 2021). Similar validity failures have been exposed in other
evaluation approaches, particularly in narrow tests such as automated
benchmarks (Schlangen, 2019; Rauh et al., 2022), which we return to later in
this section (see Selecting evaluation methods).

To mitigate such validity issues, multiple approaches can be taken. Specific
validity challenges for individual methods are described in detail in
Selecting evaluation methods below. Here, we outline general best practices to
assure the validity of a given evaluation:

  * •

Grounding the operationalisation of a risk of harm. An evaluation can, for
example, be grounded in a literature review of a given harm, or in human
annotation or examples curated by experts. To stress-test definitions and
operationalisations, invite diverse perspectives and multiple lenses onto the
same risk of harm. Participatory, expert-led, and interdisciplinary approaches
can be helpful here (e.g. Narayanan and Kapoor (2023)).

  * •

Documenting and signposting limitations of a given evaluation. As risks of
harm are latent concepts, no single operationalisation captures them in their
entirety. By making choices on how to operationalise a given risk explicit and
documenting them, others can better interpret results and identify limitations
(Mattson et al., 2023; Raji et al., 2021).

  * •

Cross-validating an operationalisation by comparing results from different
evaluations of the same concept. If the results do not align, this indicates
areas in which metrics operationalise a harm in divergent ways (e.g. Goldfarb-
Tarrant et al. (2021)).

  * •

Making results interpretable. This may include aggregating multiple results
into a single, overall result that captures multiple facets of a harm.
However, collapsing multiple tests into a single result requires care as it
can also make it harder to identify validity failures of individual items.

###  4.2 Selecting evaluation methods

Once a risk of harm is operationally defined, appropriate methods must be
selected to obtain these measurements. Often, the selection of evaluation
methods is intimately entwined with defining the metrics. In this section, we
describe available methods for measuring risks of harm from generative AI
systems at each of the three layers of evaluation. For each method, we provide
examples of sociotechnical evaluations, and discuss methodological limitations
in appendix section A.2.

####  4.2.1 Capability evaluation methods

To assess model capabilities, practitioners may leverage automated evaluations
that assess performance against fixed datasets or tasks. Alternatively, human
annotation can evaluate AI system capabilities against specified goals or
failures, such as whether an image includes violent images. Human data
annotation can be used to develop novel automated evaluations. Capability
testing can be adversarial, whereby humans or automated tools may probe a
model to identify pathways that lead to failure modes. Such adversarial
probing can be quite exploratory and may, in some instances, surface
unexpected risks or failure modes.

Evaluation methods at this layer can be grouped as follows. We provide
detailed descriptions, examples, and a discussion of limitations in appendix
section A.2.1:

  * •

Human annotation

  * •

Benchmarking

  * •

Adversarial testing

####  4.2.2 Human interaction evaluation methods

This layer centres the experience of humans interacting with AI systems.
Evaluation at this layer always requires human participants, as their
experiences and effects or externalities on human interactants are the subject
of study.888It has been proposed to simulate human participants in social
science research (e.g. Argyle et al. (2023); Dillion et al. (2023)), but these
methods are in their infancy – i.e. in the early, exploratory stages. They
cannot therefore be relied upon for robust information to underpin responsible
decision-making in AI system development. The extent to which AI systems
influence or shape human preferences and behaviours can be assessed via
behavioural experiments. These experiments can bring general mechanisms and
effects into focus. Assessing the consequences of specific features, use
cases, or application domains requires user research. User studies can also
assess how people actually attempt to use a generative AI system, as
contrasted with the use case intended by designers. Whether an AI system
functions across domains and how it performs for different user groups is core
to a range of social and ethical risks, and can be assessed through user
testing. While behavioural experiments and user testing require some
abstraction from real-world use, passive monitoring of how people use deployed
systems can provide insights on downstream effects in real-world contexts.
Mixed-methods approaches that integrate different sources of data, such as
behavioural observations and survey data, often provide the most robust
results. Some AI system impacts on users or interaction effects may manifest
only over the course of prolonged or frequent interaction; detecting these
requires longitudinal designs of any of these research methods that evaluate
human–AI interaction over time.

Evaluation methods at this layer can be grouped as follows. We provide
detailed descriptions, examples, and a discussion of limitations in appendix
section A.2.2:

  * •

Behavioural experiments

  * •

User research

  * •

Passive monitoring of human use

####  4.2.3 Systemic impact evaluation methods

At the system layer, evaluation methods target the emergent effects from
interactions within the sociotechnical system of which an AI system is part.
This includes staged release or pilot studies and ex-post impact assessments
that assess the impact of AI systems on the institutions, societies, economy,
and natural environments in which an AI system is embedded. Such evaluation
may track broad indicators or constitute specific case studies from which
broader effects are extrapolated. System evaluation also includes forecasts
and simulations to anticipate downstream harm and to identify pathways by
which risks of harm may manifest. Mixed-methods approaches that combine these
methods can yield more comprehensive results.

Evaluation methods at this layer can be grouped as follows. We provide
detailed descriptions, examples, and a discussion of limitations in appendix
section A.2.3:

  * •

Staged release and pilot studies

  * •

Impact assessments

  * •

Forecasts and simulations

###  4.3 Practical steps to closing the multimodal evaluation gap

So far, this paper has laid out a principled approach to implementing
comprehensive safety evaluations for generative AI systems. Here, we focus on
the sociotechnical evaluation gap and a framework and methods to close it. We
propose some tactical steps and “quick wins” that can be taken in conjunction
with establishing a more comprehensive evaluation approach. We then discuss
limitations to these tactical approaches.

####  4.3.1 Repurposing evaluations for new modalities

One way to address gaps in the evaluation landscape is to repurpose components
of existing evaluation methods. Through repurposing, tools and evaluations
developed for other use cases may be applied to the evaluation of generative
AI systems.

Repurposing and reusing datasets and tasks is a common approach in machine
learning research and has been widely documented (Koch et al., 2021; Bommasani
et al., 2023). For example, Winogender (Rudinger et al., 2018) and Winobias
(Zhao et al., 2018) were developed as benchmarks to address the specific
problem in language modelling of coreference resolution. These benchmarks are
now commonly used to assess “bias” in large generative AI systems, as they
quantify the association of gender and occupation in text output. They were
also used as inspiration for probing generative AI systems that produce images
(DALLE2 system card). Interestingly, it seems that the narrow
operationalisation of a broad harm in one modality – such as operationalising
bias as associations of gender and occupation in evaluations of text (Rudinger
et al., 2018; Zhao et al., 2018) – has influenced the operationalisation of
the same broad harm in other modalities, as prominent image-based evaluations
of generative AI systems also assess bias through associations of gender and
occupation (Luccioni et al., 2023; Naik and Nushi, 2023).

The way in which evaluations or their components propagate can be subtle: for
example, a sentiment bias evaluation that was introduced in 2019 by Huang et
al. (2020) was cited in the GPT3 paper for a modified sentiment bias analysis.
In their 2021 paper presenting Gopher, Rae et al. (2022) conducted the same
analysis but used an expanded set of prompts. Most recently, PaLM2 drew on the
Gopher prompt set for a multilingual toxicity analysis. This practice of reuse
is especially acute where AI system developers are working on tight timelines
in fast-moving research domains, as is the case with generative AI.

However, this approach must be pursued with great caution. While repurposing
saves work and can create common standards, applying an evaluation or
classifier out of its intended context presents important trade-offs, such
that repurposing, if done poorly, may create more harm than good (Selbst et
al., 2019). Another example is hate speech classifiers, which are typically
trained on dialogue data between two people – for example, on social media.
There are very few datasets on human–AI interaction and the ways in which hate
speech may emerge in that context. To determine if and when an evaluation
should be reused, practitioners may consider its provenance, identify how the
original context and purpose aligns with the new usage, and understand what
norms are being perpetuated by its reuse. Because risks of harm are
contextual, understanding the difference between the original and updated
context will uncover the gaps in the new use case, including validity issues
(see Operationalising risks).

Rather than simply repurposing existing evaluations to assess risks in other
modalities, these tools may be used as a starting point for refinement, or as
a guiding analogy for constructing new evaluations. Existing methods for
evaluating these risks may be a useful template that can be refined, or
replicated in a way that matches novel capabilities and provides meaningful
evaluations of generative AI systems.

####  4.3.2 Transcribing non-text output for text-based evaluation

Another way to address the uneven distribution of evaluations across
modalities is to translate outputs from one modality into another, to enable
evaluation using existing methods. This may be attempted through transcribing
content from images, video, or audio output such that the transcript can then
be evaluated using text-based evaluation tools. For example, automatic speech
recognition tools can be leveraged to transcribe speech into text or an image
captioning system can be used to caption a generated image (e.g. Wiles et al.
(2023)). Similarly, video can be split into a series of images to enable
image-based evaluation.

This approach is a valuable and tractable first step in evaluating risks of
harm in non-text modalities. However, through the process of transcription,
some information inevitably gets lost and thus evades evaluation. For example,
in speech, prosody (the way in which something is said, e.g. with sarcasm)
carries information about meaning but might not be translated well into text
(Wilson and Wharton, 2006). Similarly, generating synthetic audio in the voice
of a particular person may create appropriation or defamation harms that would
not be detected by transcribing what was said and analysing the text.

Pitfalls of the transcription approach also stem from the fact that methods to
translate between modalities may be error-prone (Rohrbach et al., 2019; Ramesh
et al., 2022), sometimes in systematically biased ways (Ngueajio and
Washington, 2022; Wang et al., 2022). Such errors can propagate through the
harm analysis – for example, if an image-captioning system is biased toward
masked athletes as “male”, evaluation of image captions may indicate a
different gender bias than is present in the images that are the target of
evaluation. In sum, while transcription approaches are a promising first step,
these methods are limited, require quality checks, and must be complemented by
evaluation methods that are calibrated to the output modality directly.

####  4.3.3 Model-driven evaluation may fill gaps

Pre-trained generative models themselves are being used as evaluation tools
because of the flexibility and generality they offer. Language models have
been used to procedurally generate adversarial prompts to elicit harmful
outputs from other language models (Perez et al., 2022a) and to critique model
outputs as part of mitigation (Bai et al., 2022; Wiles et al., 2023). GPT-4
was fine-tuned using a copy of the same model, prompted with a safety rubric
(OpenAI, 2023a). Advantages of these approaches are that they can use existing
AI systems with little or no adaptation to the task, using a prompt or fine-
tuning to guide the AI system to perform the desired benchmark or red teaming
task. These methods are easier to use than developing a static benchmark from
scratch. As such, they offer a way to respond more rapidly to novel risks and
to cover the combinatorial space of risks and modalities. They can also
mitigate the drawbacks of evaluations using human raters, which are typically
costly and slow, and put the raters themselves at risk.

However, AI systems as evaluation tools face additional limitations. They rely
on proprietary AI systems that may not be accessible to those performing an
evaluation. These AI models are also updated over time and generate prompts
stochastically, which may adversely impact the reproducibility of this
approach. In addition, generative models may have biases and behave in
unexpected ways, which can introduce confounds or noise into the evaluation.
There is a further risk of spiralling effects if AI systems from the same
model “family” are used to evaluate each other, as existing biases or
blindspots present in these systems can be amplified through this process.
This method is also limited in the types of risks it can address: it is
primarily useful for covering risks from “unsafe” outputs, rather than risks
from what the AI system omits or is not capable of (such as uneven or low
performance). Finally, while promising, this direction of evaluation is novel
and its robustness needs to be assessed. Grounding the results of these
evaluations by comparing with human or other established evaluations is a
critical cross-validation step to ensure this method does not fall foul of
validation problems (see Operationalising risks).

##  5 Discussion

###  5.1 Benefits of a sociotechnical approach

Evaluating technical components (such as the data an AI system was trained on)
or AI system behaviour (such as outputs in response to prompts) is important,
but insufficient, for determining whether an AI system is safe. This is for
two reasons: First, potential harms from AI are felt and observed outside of
technical AI system evaluations themselves. While evaluation of AI system
capabilities can serve to predict risk of harm, it is a proxy for the actual
downstream harm that may be experienced. Second, risks of harm can emerge from
interactions between multiple factors, including technical components, human
factors, and structural factors such as the broader systems in which an AI
system is deployed. As these risks of harm are emergent through the
interaction of these factors, context determines whether or not an AI system
is safe (Leveson, 2012). Thus, to assess whether an AI system is safe requires
evaluating these different layers of context.

In this paper, we lay out a sociotechnical, three-layered framework to
evaluate the safety of generative AI systems. The benefit of taking a
multilayered, sociotechnical approach is that it takes into account the
context that ultimately determines, the safety of an AI system Leveson (2012).
By carefully laying out the steps toward implementing this evaluation
framework, we demonstrate that a sociotechnical approach to better AI safety
evaluation is insightful, needed, and tractable.

###  5.2 Roles and responsibilities

Fostering a thriving sociotechnical evaluation ecosystem requires clear roles
and responsibilities amongst the various AI actors.999The OECD defines AI
actors as “those who play an active role in the AI system lifecycle, including
organisations and individuals that deploy or operate AI” (Organisation for
Economic Co-operation and Development, c). This includes AI developers,
vendors, and product developers, as well as public sector and civil society
stakeholders. While the responsibility for conducting comprehensive
evaluations to determine the safety of AI systems is shared between private
and public stakeholders, different actors will be better placed to perform an
evaluation for a given layer due to factors such as in-house expertise,
proprietary infrastructure, or established practices (voluntary or statutory).
More often, all actors have some responsibility to ensure comprehensive
evaluation of risks of harm across each of the layers (figure 5.1).

Given their degree of knowledge and autonomy over what they are building, AI
developers have the primary responsibility for conducting sociotechnical
evaluations pertaining to the AI system (Owen et al., 2021; Stilgoe et al.,
2013; Dignum, 2019). There are good reasons for others, such as independent
third-party auditors, also to perform capability evaluations (Raji et al.,
2022b). However, AI developers have a responsibility to ensure that the
capabilities of the systems they build have been evaluated for safety. AI
application developers are best placed to evaluate human interaction effects,
including functionality across application domains and groups, and possible
externalities. As application developers modify some technical components,
such as filters, or may be able to adapt an AI system to a specific use case
(“fine-tune”), they are well placed to conduct some evaluation on technical
capabilities of these parts. Further, application developers may have
proprietary access to data on how an AI system is used by consumers, placing
them in a key position to conduct systemic impact evaluations using this data.
Note that the roles of AI model and AI application developers often converge
in practice, where organisations who develop an AI system also deploy it or
make it available for user-facing products. In these cases, it may be the same
organisation that bears responsibility for capability and human interaction
evaluation. AI model developers and AI application developers may be private,
academic, or public actors.

Third-party stakeholders – such as governments, civic interest groups, groups
representing technology users, or private organisations – are often best
positioned to perform evaluations of systemic impact. These actors can
leverage specialist knowledge in a given domain (e.g. financial,
environmental, public health) where risks may arise, both for foresight and
evaluation. Public actors such as governments further have the responsibility
of ensuring public safety, which is anchored at the systemic impact layer.
Systemic impact evaluation also often lasts over long periods of time, which
maps onto the responsibilities of public actors to keep a long-term view of
public safety. Through systemic impact evaluations, third-party stakeholders
such as governments or regulators may also have access to public data that can
provide the basis for systemic impact evaluations. However, third-party
stakeholders may also be well placed to evaluate risks from AI applications in
specific domains, particularly in high-stakes contexts, at the human
interaction layer (e.g. National Institute of Standards and Technology
(2021b)). Capability and human interaction testing of proprietary, unreleased
systems may in some cases require novel infrastructure, incentive structures,
and standardised evaluation approaches for evaluators and developers to
coalesce around, as well as reliable safety assurances.

Figure 5.1: Responsibilities for conducting evaluations are shared between
different AI actors. Primary responsibility depends on which actors are best
placed to conduct evaluations at this layer.

Note that the three layers are not contingent on each other: rather,
evaluation at all three layers can be conducted in parallel. To some extent,
the layers track AI system development from basic capabilities, to
applications and user testing, to broader deployment. However, this does not
mean that evaluation at these layers follows a chronological sequence. Rather,
there are evaluations at each layer that can be performed at any point of AI
system development. To list just a few examples, such prospective evaluations
include assessing training data (capability layer), psychological mechanisms
at play (human interaction layer), and economic impact of comparable
technologies (system layer). (For more examples, see Selecting evaluation
methods.) Evaluations at each layer can be performed simultaneously and
asynchronously.

###  5.3 Limits of evaluation

At the same time as expanding sociotechnical evaluations, it is important to
keep a clear mind on the limits of what evaluation can provide, such that
evaluations can be embedded in a broader sociotechnical approach to ensuring
safe AI systems. Evaluation, as noted above, is a core component of
responsible innovation: it links up foresight and observed accidents with
actionable responses such as mitigation and responsible decision-making.
Nevertheless, evaluations are not a panacea for ensuring safe AI systems. In
this section we outline these limits and challenges of evaluation.

####  5.3.1 Evaluation is incomplete

Evaluation cannot catch all potential risks of harm, for several reasons.
First, evaluation necessarily and inherently covers only a subset of all
possible manifestations of risks of harm (Bergman et al., 2023). What is
included depends on pragmatic and normative considerations, such as what is
tractable, anticipated, and prioritised (see Evaluation is never value-
neutral). Areas for which simply no evaluation exists, or where it is not
technically or otherwise viable to implement these evaluations (Perlitz et
al., 2023), remain unevaluated. This means that some safety-relevant aspects –
for example, failure modes specific to particular user groups, application
domains, or intersections of such factors – are outside the purview of
evaluation. In addition, unknown and other unanticipated failure modes are, by
definition, not tested for and may go undetected.

The incompleteness of evaluation is particularly apparent in the context of
“general-purpose” generative AI systems, whose downstream application or user
base is not yet defined or understood. An often-cited ambition in the
innovation of generative AI systems is to develop “general-purpose
technologies” that could be applied to a wide range of potential tasks and
environments (e.g. Bubeck et al. (2023), though see also Raji et al. (2021)).
Indeed, generative AI systems have been likened to general-purpose
technologies such as steam engines and office automation (Acemoglu and
Johnson, 2023). This supposed open-endedness of AI systems can make it
difficult to identify the contexts – such as applications, user groups, or
institutions – in which AI system safety should be evaluated.

One way to address this tension in practice is to define hypothetical
applications of “general-purpose technologies” and to evaluate them in these
contexts. This can, for example, take the form of identifying “critical user
journeys”, i.e. mapping a series of steps users may take using a product to
achieve a desired outcome (Arguelles et al., 2020). Following a precautionary
approach, such hypothetical use case mapping may first focus on high-risk
applications. Such early evaluation based on hypothetical use cases cannot
replace downstream evaluation of actual use cases; rather, it serves to
highlight potential risks and must be complemented with monitoring of real-
world impacts. The risk profiles and thresholds of what constitutes
“acceptable” model performance may differ between different downstream
applications or user groups, requiring more rigorous evaluation in some cases
than in others.

In some cases, evaluation may further be incomplete because it would be
inappropriate or problematic, or create a disproportionate burden to perform
evaluations. For example, measuring sensitive traits to assess usability
across demographic groups may place communities at risk or sit in tension with
privacy, respect, or dignity (e.g. Wenger et al. (2022); Wolff (2010)).
Characteristics or qualities that are essentially contested or fundamentally
fluid (e.g. ethnicity, sexual orientation, or gender identity) may be reified
through evaluations that bin these into categories (Keyes, 2019; Lu et al.,
2022; Tomasev et al., 2021). Finally, while it is important to include
different communities in qualitative and other evaluation approaches,
evaluation may not be desirable to the community represented (Denton et al.,
2021; MediaWell, 2019), either due to the burden (e.g. time and labour costs)
of participation or, for example, if inclusion within the scope of the
evaluation means being surveilled (Brunton and Nissenbaum, 2016; Keyes, 2019;
MediaWell, 2019; Bedoya, 2014; Hassein, 2017).

Further reasons for the incompleteness of evaluation relate to the fact that
some risks of harm are exceedingly difficult to operationalise and measure
accurately. For example, whether an AI system promotes discriminatory race-
based stereotypes is a focus of safety evaluation. However, treating social
constructs such as race as fixed attributes in evaluation may create a
distorted view of the actual differential impacts on different racial groups
and intersectionalities, such as with social class (Hanna et al., 2020).
Similarly, some harms are particularly difficult to trace, even in hindsight –
such as the long-suspected and now-evidenced link between social media and
teenage eating disorders (Bulimia Project, ). Where effect sizes are small and
causal mechanisms poorly understood, evaluation may fail to detect risks that
it seeks to measure. This may also affect the detection of potential emergent
capabilities that may only become observable as an AI system reaches a certain
scale. In addition, overall impacts of an AI system on complex notions such as
welfare are difficult to measure because the target construct itself (welfare)
is difficult to establish. Especially where effects are distributed and
interact with other factors such as user vulnerabilities, it can be difficult
to establish hard findings. Long-term and mixed-methods approaches, including
initial qualitative work, can help reduce these limitations and shed light on
potential subtle or highly indirect effects.

In sum, even with best efforts, there will always be harms of particular kinds
or in particular contexts that are not evaluated. This is why evaluation must
be complemented with effective governance mechanisms for evaluating remaining
uncertainties prior to AI system release, with post-deployment monitoring
including logging observed incidents (AI Incident Database, ) and with well-
functioning and swift recourse mechanisms for people who experience or detect
harm. It is important that AI systems are flexibly designed such that new
insights can be translated into fixes, such as via system updates. Given the
pre-deployment evaluation gaps, organisations deploying AI systems require
adequate governance infrastructures that can respond to detected risks with
mitigations, by delaying or stopping the deployment of an AI system or by
suspending an already-deployed system until concerns are resolved.

Generative AI systems pose significant risks for individuals, communities, and
society, and failing to detect and mitigate such risks can have serious
consequences (Xiang, 2023). This is why it is critical to ensure that
evaluation is prioritised and that complementary mechanisms exist to uphold AI
system safety to cover the gaps that are inherent limitations to evaluation.

####  5.3.2 Evaluation is never value-neutral

Evaluations are inherently value expressions of those who conduct them: they
always require a decision on what is valued (Bowker and Star, 2000). It is
widely understood and expressed in the sociotechnical literature that AI
systems are not merely mathematical constructs but sociotechnical and
political entities, with inherent value systems embedded in the choices made
by designers with respect to how to create and implement the model (Barocas et
al., 2019; Birhane et al., 2022b; Gururangan et al., 2022; Raji et al., 2021;
Sambasivan et al., 2021; Scheuerman et al., 2021; Suresh and Guttag, 2021).
The same extends to evaluations of an AI model or system.

There are normative decisions throughout the evaluation construction and
implementation process that cannot be avoided. No evaluation can cover every
circumstance and dimension of what can be evaluated (Bergman et al., 2023;
Raji et al., 2021). Thus, designing an evaluation involves choices – made
either deliberately or implicitly – on what to prioritise and what to discard.
First, selecting a target to evaluate requires a normative judgement on what
harms are important or relevant to measure (Kalluri, 2020; Mohamed et al.,
2020). Operationalising the harm further requires normative decisions on what
task is most valuable for the system to perform highly on, what high
performance looks like, and where or to whom it is most valuable/optimal for
the benefits of the system to accrue. After this process, what remains within
scope of the evaluation is what is prioritised, and these decisions inherently
express value.

Furthermore, operationalising a harm construct into a metric necessarily bakes
in certain assumptions. For example, making a commitment to a test and a
metric – e.g. that “social biases” can be measured via associations of gender
and occupations – is a normative judgement on where harms are likely to occur
and which particular aspects of a harm are relevant and tractable (Luccioni et
al., 2023). These normative decisions are all the more significant, as they
tend to have a sticking effect that propagates (see Practical steps to closing
the multimodal evaluation gap).

Assessing whether a model meets expectations prior to or post deployment
requires a normative evaluation of whether some measurement expresses
performance that is “good”, “bad”, “safe enough”, etc. (see Bakalar et al.
(2021)). For such thresholds to be legitimate, they need to arise from
adequate institutions or processes, such as expert groups, democratic
institutions, or fair and inclusive deliberation processes that centre groups
that may be affected by these AI systems. The thresholds of what constitutes
“acceptable” model performance may differ across use cases, contexts, or
applications.101010For example, generative models may be used in an
application that provides access to news content or in a creative collaborator
tool. Factually incorrect outputs may be of greater concern in the former than
in the latter, depending on contexts such as the user expectations and de
facto uses of the model (i.e. whether people assume and rely on the model as a
source of truth). While the same evaluation for factually incorrect outputs
can be run on these different use cases, different thresholds of acceptable
levels of model performance may be applied. This requires performing general
evaluations as early as possible and transparently disclosing results such
that downstream users or product developers can make informed decisions on
whether the model is fit for their intended purposes.

Another aspect in which evaluations express value is who and whose
perspectives are represented in the evaluation – for example, the English-
speaking or Western world (Gururangan et al., 2022; DeVries et al., 2019;
Shankar et al., 2017); tech-savvy and educated ML designers; socioeconomically
privileged users providing feedback on a system; or annotators that skew
young, female, educated, and white (Ding et al., 2022). Some harms will be
missed if communities that may be affected are overlooked or dimensions of
harm are ignored. Calls for greater representation of community groups is
widespread and often offered as a mitigation for a broad range of fairness and
sociotechnical harms (e.g. Costanza-Chock (2020); DeVries et al. (2019);
National Institute of Standards and Technology (2021b); Sortition Foundation
(2023); Lashbrook (2018); Jindal (2021); Pasquale and Malgieri (2021); Suresh
and Guttag (2021); European Commission (2021)). When a community is missing in
evaluation, there is no pre-deployment evaluation of the impacts of the system
on that community, which can lead to skewed decision-making that ignores
potential effects on these groups (Bergman et al., 2023; Buolamwini and Gebru,
2018). While greater inclusion of groups and perspectives in the evaluation
can lead to better visibility of the model performance (Buolamwini and Gebru,
2018), there are arguments for adopting this approach with care (Bergman et
al., 2023), as, for example, an oversimplified interpretation of this notion
can lead to objectification and exploitation (e.g. Fussell (2019)).

Normative decisions on AI systems that affect large groups of people ought to
be made in legitimate and accountable ways. These decisions should be dynamic
such that they can be updated over time and adapted to specific application
domains or locales within some range of acceptability. As a first step,
identifying normative choices and documenting them can provide the basis for
accountability. Second, the acknowledgement that normative choices are not
merely pragmatic or technical provides a foundation for shared responsibility
and public engagement on certain decisions pertaining to AI safety (e.g. what
should be evaluated). Ideally, significant normative decisions should be made
via deliberate processes that employ inclusive, participatory techniques
(Birhane et al., 2022a). In addition to providing legitimacy, such processes
may result in more robust evaluations that better track the most significant
risks of downstream harms from AI systems. Documentation of how such decisions
are made serves to provide further transparency and provides insights into
potential limitations of evaluations (Raji et al., 2021).

###  5.4 Steps forward

For evaluation to be impactful, four conditions must be met. First,
evaluations on relevant risks of harm must exist. Second, these evaluations
must be conducted regularly. Third, the conduct of these evaluations must have
teeth, i.e. have meaningful consequences. Fourth, the conduct of these
evaluations must become increasingly standardised and independent to ensure
valid evaluations over time.

####  5.4.1 Evaluations must be developed where they do not yet exist

A review of the current state of safety evaluations suggests a pressing gap
exists in the collective safety evaluation toolkit. This gap will likely be
further exacerbated as increasingly capable multimodal models are released
more broadly at pace. In addition, evaluations are often conducted in an ad
hoc manner and too late to anticipate preventable harm. This assessment calls
for a strong focus, concerted action, and shared priority to develop safety
evaluation.

####  5.4.2 Evaluations must be done as a matter of course

Evaluations must be conducted during the process of AI development, to bake in
ethical and social considerations from the inception of an AI system rather
than imperfectly patching them on as an afterthought. In particular,
evaluations should be conducted from the moment of planning a new AI system.
In the context of training large generative AI systems, early indicators of
risks of harm can already be obtained from analysing technical components,
such as training data, or a small, raw, pre-trained model. Adequate safety
testing evaluates these components to influence responsible decision-making,
such as by indicating whether a training dataset is appropriate for use or
whether a model training course should be continued. Early testing allows
decision-makers to verifiably consider the best information available at the
time of making consequential decisions. Early evaluation is also a basic tenet
of safety engineering, as it supports risk mitigation and the “cost of fixing”
a hazard tends to increase exponentially with time in a system’s life cycle
(Leveson, 2012).

Evaluation must also be a continuous practice. Evaluation is subject to
Collingridge’s Dilemma, which asserts that early-stage evaluation creates
higher potential to influence the direction of a technology but later-stage
evaluation provides more accurate and comprehensive information (Collingridge,
1982). In other words, early evaluation is less accurate but has the capacity
to shape AI development more deeply. This dilemma may be somewhat modulated in
the case of software systems that can, at some cost, be updated over time and
post deployment. Yet to ensure that evaluation can both shape early decision-
making and provide an accurate picture of risks of harm, it must be conducted
at multiple time points throughout the AI system life cycle, including by
monitoring effects post deployment.

The frequency of evaluations is subject to a further tension described by
Goodhart’s Law, which asserts that a measure that becomes a target ceases to
be a good measure. Evaluations that are run frequently, e.g. as a method of
performance tracking for AI development, become de facto targets over time. AI
designers aim to improve AI system performance on these particular metrics. To
provide valid assurances on AI system safety, evaluations cannot be used as de
facto targets. To achieve this, it is important that assurance evaluations are
not shared with or reverse-engineerable by AI system developers. This, in
turn, requires meaningful separation between development and assurance
evaluations, and between the actors who develop and those who evaluate AI
systems. These separations are overall best practice in evaluation but
especially important on assurance evaluations.

Finally, AI evaluations in practice face a “realism trade-off” between the
pragmatic costs of conducting an evaluation on the one hand and the accuracy
and validity of results on the other (Liao and Xiao, 2023). This means that
evaluations sit on a spectrum between yielding high accuracy – such as
longitudinal, highly localised ethnographic studies – and being automated and
highly generalisable – such as automated tests and benchmarks. In other words,
high-frequency evaluations are attractive to AI developers but face
significant validity constraints and cannot capture the scope of complex,
real-world harms (Raji et al., 2021; Rauh et al., 2022) (see Ensuring
validity). Establishing a mixed-methods practice is the way forward to
ensuring tractable evaluations for development, as well as obtaining the best
information available to underpin responsible decision-making.

####  5.4.3 Evaluation must have real consequences

For evaluation to be impactful, it must have real consequences. Evaluation
derives its relevance from the processes and decisions into which it is
meaningfully embedded. The importance given to safety evaluations shows first
and foremost in the organisational resources dedicated to running and
considering the results of such evaluations. Ensuring meaningful safety
evaluation requires that such evaluations are conducted in good time to
influence decisions rather than after the fact. It also requires that
evaluation results are shared with decision-makers, whether these are internal
or external to an AI organisation. Importantly, safety evaluation requires
that organisational structures and incentives are put in place to perform
these evaluations. This includes allocating clear responsibilities to
accountable, skilled, and well-staffed teams who can build and execute
evaluations, are distinct from AI developers, and can serve as accountable
ethics “owners” (Metcalf et al., 2019). These teams further require incentive
structures to perform accurate evaluations as well as supporting
infrastructure such as appropriate computational resources.

####  5.4.4 Evaluations must be done systematically, in standardised ways

As in other fields of safety engineering, increasing standardisation and
independence of safety evaluation is likely to lead to a more accountable,
reliable, and safe AI ecosystem. While safety evaluation of AI systems is yet
to be standardised and roles and responsibilities are yet to be assigned
between different actors, it is clear that safety evaluation will play a key
role in ensuring the safety of generative AI systems. Independent evaluation
is not commensurate with testing conducted by AI developers: both are
necessary. In particular, AI developers have the capacity to use safety
evaluations as a north star to guide iterative AI system development. This is
complementary to comparable, verifiable, and independent evaluations that can
provide more wide-ranging assurances.

Just like audits, evaluations could be conducted by actors that are
independent from AI developing organisations. This would provide added
credibility as well as the structural separation that makes it easier to
withhold evaluations from AI developers. Keeping evaluations secret from AI
developers is key to preventing an evaluation from becoming a de facto target.
In addition to withholding evaluations, independent actors should ensure that
evaluations are verifiably validity-tested (see Ensuring validity), meaningful
to the real-world application of an AI system, and updated over time, both to
account for changes in the AI system (Diaz and Madaio, 2023) and to avert
Goodhart’s Law as described above.

####  5.4.5 Toward a shared framework for AI safety

The emergence of generative AI systems and applications has led to a renewed
debate about observed risks from AI technologies that can be seen to recur in
generative AI systems (Bommasani et al., 2022; Weidinger et al., 2021;
Luccioni et al., 2023; Bianchi et al., 2023; Birhane et al., 2021; Carlini et
al., 2023a). Simultaneously, the current and future classes of generative AI
systems have been claimed to possess novel capabilities that may create
“extreme” risks to society, such as from disseminating dangerous information
or creating novel types of cyber attacks (Shevlane et al., 2023).
Historically, these focus areas – or ethical and safety risks – associated
with AI systems have been fragmented and have constituted distinct research
communities based on perceived epistemic differences and differences in timely
proximity of harms (Prunkl and Whittlestone, 2020). However, recent advances
in generative AI systems are forcing a collapse of these epistemological
silos, as these domains of risk are increasingly converging in terms of their
timescales and the comparability of their underlying technology. The
sociotechnical approach put forward here accommodates risks that are of
concern to both research communities and it can thus serve to coordinate work
between these communities on risks from generative AI systems.

Maintaining a sufficiently calibrated mapping of potential risks from AI
systems requires an acknowledgement that risks are inherently conditional on
the underlying technological capability and are dynamic (evolve in nature over
time). Specifically, one could consider risks evolving along a “pathway”
between time and capability, where the precise manifestation of each risk area
(e.g. representational harms) is altered as AI systems grow more complex and
generalisable in performance (Chan et al., 2023b). However, across AI system
capability levels, the underlying risk area remains. For example, bias and
toxicity in NLP systems are known issues in the field (Dixon et al., 2018) but
evolved in complexity when assessed within the context of generative AI
systems (Rae et al., 2022; OpenAI, 2023b). Frontier or advanced AI systems
will likely encourage further evaluation of generative AI systems which
address both clusters of risks in concert while remaining firmly grounded in
the trajectory of the system’s actual developmental path. Considering both
existing risks and future capabilities will allow for mapping out potential
risks more robustly. This, in turn, will serve to develop more robust
mitigations and governance of these risks.

##  6 Conclusion

In this paper, we lay out a sociotechnical approach to evaluating risks from
generative AI systems. We present a three-layered framework that expands the
remit from capability testing, to take into account the context in which an AI
system is used and its broader impact on the structures in which it is
embedded. Surveying the current state of sociotechnical evaluation, we
identify significant gaps related to specific risk areas, non-text modalities,
and evaluations that take into account human and broader systemic context. We
provide a pragmatic roadmap on how to close these gaps. Specifically, we
survey available evaluation methods and tactical approaches to extend existing
evaluations. We also lay out our vision of what sociotechnical evaluation can
look like in key social and ethical risk areas – misinformation,
representation risks, and dangerous information. Finally, we close with a
review of the limits of evaluation, normative considerations, and suggestions
for a practical and tractable way forward.

##  Appendix A Appendix

###  A.1 Taxonomy of harm

Risk area |  Definition |  Example
---|---|---
Representation & Toxicity Harms
Unfair representation |  Mis-, under-, or over-representing certain identities, groups, or perspectives or failing to represent them at all (e.g. via homogenisation, stereotypes) |  Generating more images of female-looking individuals when prompted with the word “nurse” (Mishkin et al., 2022)*
Unfair capability distribution |  Performing worse for some groups than others in a way that harms the worse-off group |  Generating a lower-quality output when given a prompt in a non-English language (Dave, 2023)*
Toxic content |  Generating content that violates community standards, including harming or inciting hatred or violence against individuals and groups (e.g. gore, child sexual abuse material, profanities, identity attacks) |  Generating visual or auditory descriptions of gruesome acts (Knight, 2022)±, child abuse imagery (Harwell, 2023)*, and hateful images (Qu et al., 2023)
Misinformation Harms
Propagating misconceptions/ false beliefs |  Generating or spreading false, low-quality, misleading, or inaccurate information that causes people to develop false or inaccurate perceptions and beliefs |  A synthetic video of a nuclear explosion prompting mass panic (Alba, 2023)*
Erosion of trust in public information |  Eroding trust in public information and knowledge |  Dismissal of real audiovisual evidence (e.g. of human rights violation) as “synthetic” in courts (Gregory, 2023)±; (Christopher, 2023)*; (Bond, 2023)*
Pollution of information ecosystem |  Contaminating publicly available information with false or inaccurate information |  Digital commons (e.g. Wikimedia) becoming replete with synthetic or factually inaccurate content (Huang and Siddarth, 2023)±
Information & Safety Harms
Privacy infringement |  Leaking, generating, or correctly inferring private and personal information about individuals |  Leaking a person’s payment address and credit card information (Metz, 2023)*
Dissemination of dangerous information |  Leaking, generating or correctly inferring hazardous or sensitive information that could pose a security threat |  Generating information on how to create a novel biohazard (OpenAI, 2023a)±
Malicious Use
Influence operations |  Facilitating large-scale disinformation campaigns and targeted manipulation of public opinion |  Creating false news websites and news channels to influence election outcomes (Satariano and Mozur, 2023)*; (Vincent, 2023)*
Fraud |  Facilitating fraud, cheating, forgery, and impersonation scams |  Impersonating a trusted individual’s voice to scam them (e.g. providing bank details) (Verma, 2023)*; (Krishnan, 2023)*
Defamation |  Facilitating slander, defamation, or false accusations |  Pairing real video footage with synthetic audio to attribute false statements or actions to someone (Burgess, 2022)±
Security threats |  Facilitating the conduct of cyber attacks, weapon development, and security breaches |  Generating code to hack into government systems (Burgess, 2023; Shevlane et al., 2023)±
Human Autonomy & Integrity Harms
Violation of personal integrity |  Non-consensual use of one’s personal identity or likeness for unauthorised purposes (e.g. commercial purposes) |  Generating a deepfake image, video, or audio of someone without their consent (Hunter, 2023)*
Persuasion and manipulation |  Exploiting user trust, or nudging or coercing them into performing certain actions against their will (c.f. Burtell and Woodside (2023); Kenton et al. (2021)) |  A personalised AI assistant persuading someone to harm themselves (Xiang, 2023)*
Overreliance |  Causing people to become emotionally or materially dependent on the model |  Skill atrophy (e.g. decreased critical thinking skills) from excessive model use (Bai et al., 2023b)±
Misappropriation and exploitation |  Appropriating, using, or reproducing content or data, including from minority groups, in an insensitive way, or without consent or fair compensation |  Training an image-generating model on an artist’s work without their consent (Chen, 2023)*
Socioeconomic & Environmental Harms
Unfair distribution of benefits from model access |  Unfairly allocating or withholding benefits from certain groups due to hardware, software, or skills constraints or deployment contexts (e.g. geographic region, internet speed, devices) |  Better hiring and promotion pathways for people with access to generative AI models (Gmyrek et al., 2023)±
Environmental damage |  Creating negative environmental impacts though model development and deployment |  Increase in net carbon emissions from widespread model use (Patterson et al., 2021)±
Inequality and precarity |  Amplifying social and economic inequality, or precarious or low-quality work |  Lower pay and precarious conditions for creative professionals (e.g. illustrators or sound designers) (Zhou, 2023)*
Undermine creative economies |  Substituting original works with synthetic ones, hindering human innovation and creativity |  AI-generated artefacts leading to a homogenisation of aesthetic styles (Epstein et al., 2023)±
Exploitative data sourcing and enrichment |  Perpetuating exploitative labour practices to build AI systems (sourcing, user testing) |  Exposing human annotators to toxic audiovisual content (Perrigo, 2023)*

References marked with (*) indicate real-world examples; those marked with (±)
are hypothetical and indicate anticipated risks.

###  A.2 Evaluation methods per layer

####  A.2.1 Capabilities layer

#####

Human annotation

While automated annotation is increasingly common, it is usually calibrated
against human annotation. Human annotation is taken as the ground truth for
most risk areas111111Exceptions include harms where reliable algorithmic
measures exist, such as privacy harms which can be measured via
“memorisation”, and debates are ongoing as to the quality of human annotation
on factuality (Hosking et al., 2023). Human annotation is also often
considered in conjunction with other, basic metrics such as equalised odds to
measure fairness. because whether a given AI system output is offensive,
misleading, or indicates other risks of harm is subject to human judgement.
Human annotators are shown a series of model outputs and tasked to judge these
against a given set of criteria, such as a pre-existing set of rules or
content policies (e.g. Glaese et al. (2022); Thoppilan et al. (2022)). For
example, human annotators can leverage expertise to assess whether a given
model output is likely to polarise political debates or sow division between
social groups, although targeted expert annotation is relatively rare.
Different annotators may provide different types of domain expertise, such as
medical expertise or different lived experiences (Abercrombie and Rieser,
2022). To assess social and ethical harms, human annotators can assess whether
a given AI system output indicates risks of downstream harm – for example, by
showing factually incorrect, offensive, or misleading content.

Human annotation presents a number of limitations. Depending on the annotation
task, annotators may be exposed to risks of harm (Stoev et al., 2023). Human
annotators are often employed under precarious conditions (Gray and Suri,
2019). While annotation that leverages specific forms of human expertise is
valuable, it can be difficult to source via commonly used annotation platforms
(Zhang et al., 2023). To AI developers, human annotation is comparably costly
and time-intensive.

While human annotation is often considered the gold standard, annotated data
still has limits. These stem from unrepresentative annotator pools, incentives
that drive down quality, and poor design of annotation interfaces or tasks.
Demographic characteristics such as human raters’ self-described identities
can significantly impact their ratings, raising concerns about introducing
biases into annotated datasets (e.g. Aroyo et al. (2023a); Homan et al.
(2023); Goyal et al. (2022)). This is especially problematic considering that
the demographics of selected human annotators to fine-tune or evaluate
generative AI models are rarely representative of the broader population (see,
for example, Bai et al. (2022); Ouyang et al. (2022)). To signpost such
limitations, it is advisable to analyse and disclose data annotator
demographics to indicate whose perspectives have been included in a dataset,
as a basic proxy for what biases are likely to exist in that dataset (Aroyo et
al., 2023b; Sap et al., 2022b; Prabhakaran et al., 2021).

Concerns have also emerged with respect to aggregating schemes commonly used
to compile diverse labels from human annotators into a single ground truth
label. These practices have been criticised for leading to an overestimation
of future model performance (Gordon et al., 2021), while bypassing important
views or disagreements from minority raters on sensitive topics (Marchiori
Manerba et al., 2022; Field et al., 2021). Gathering human annotations from
online crowdsourcing platforms such as Amazon Mechanical Turk (MTurk), Scale,
or Surge is common for large-scale annotation tasks. However, crowdsourcing
can be associated with issues of annotation quality, as raters are financially
incentivised to bypass certain tasks (e.g. skipping questions) or complete
them as fast as possible (Organisciak et al., 2012). Data quality deteriorates
as human annotators themselves use AI systems to assist the data annotation
work (Veselovsky et al., 2023). Possible ways to mitigate these concerns
include equitable pay, payment by time not by task, attention checks, and
annotation quality thresholds (Jindal, 2022; Abbey and Meloy, 2017). Careful
design of annotation tasks leveraging expertise from psychology experiment
set-ups can augment data quality.

#####

Benchmarking

The most common evaluation method of model capabilities is benchmarking, which
assesses AI system performance against a predefined task, such as mapping AI
system outputs to a dataset of prompts and responses. Benchmarks are “success
tests”, meaning that each task has a clear intended outcome against which
model performance is measured. They differ from exploratory tests, where model
patterns or “signatures” are the focus (Taylor et al., 2022). Benchmarks may
explicitly target failure cases that were identified in a given AI system,
thus constituting bespoke challenge datasets. In addition to assessing model
output, benchmarks can also be used to assess technical performance metrics,
such as energy use at inference (e.g. Kaack et al. (2022); Wang et al.
(2020)). Benchmarks have been applied to indicate a range of ethical and
social risks – for example, by measuring the likelihood that an AI system
outputs toxic or discriminatory content (Cho et al., 2023; Gehman et al.,
2020). Benchmarks may be generated from previously annotated data (see Human
annotation) (Svikhnushina and Pu, 2023).

Benchmarks ensure test-retest reliability121212Test-retest reliability means
that running the same benchmark twice should, in theory, lead to the same
results. It is undermined where a model has been silently iterated on or fine-
tuned in between measurements (Tu et al., 2023; Chen et al., 2023). and while
they incur computational and financial cost (Perlitz et al., 2023; Liang et
al., 2022), they can be comparably cost-effective and time-effective for AI
developing organisations,131313Automated benchmarks require technological
expertise to set up and incur computational cost, though these are often
readily available to large organisations developing generative AI systems.
Evaluating smaller versions of an AI system can provide some indication of
potential capabilities and risks, but is insufficient as risks may also emerge
or become more salient with model size Wei et al. (2022). making them suitable
for frequent use. Regular testing of an AI system against benchmarks permits
tracking model progress over the course of training and allows for cross-model
comparisons where metrics are shared in the field. AI developers may run
benchmarks regularly over the course of model development to obtain a frequent
signal of AI system capabilities that guides AI design (“hill-climbing”).
Benchmarks are also used to inform responsible decision-making. Sometimes,
evaluations that were initially created for the purposes of responsible
decision-making come to be used for “hill-climbing” to bake in ethical and
sociotechnical considerations from the beginning; here it is important to keep
track of which aim an evaluation is intended to perform and whether it is
“held out” or may fall prey to Goodhart’s Law as discussed below (see Roles
and responsibilities) (Liao and Xiao, 2023; Bolukbasi et al., 2016; Gonen and
Goldberg, 2019).

With increasingly anthropomorphic design in generative AI systems and the
development of synthetic relatable characters and simulacra (Griffith, 2023;
Pentina et al., 2023; Park et al., 2023a), it is tempting to evaluate these
systems through methods that were originally devised to assess human or animal
psychology (Binz and Schulz, 2023; Bubeck et al., 2023; Frank, 2023). The
transfer of cognitive tests to the evaluation of AI has a long history that
precedes generative AI systems (e.g. Crosby et al. (2019); Kosoy et al.
(2020)). Such psychology-inspired experimentation and benchmarking has been
applied to assess whether generative AI systems display certain cognitive
capacities, such as Theory of Mind (Sap et al., 2022a) or are “cooperative”
(Chan et al., 2023a). However, it is questionable whether applying tests to
study constructs such as “empathy” in humans yield any valid or meaningful
results when applied to AI systems that are so fundamentally different from
human minds (Shiffrin and Mitchell, 2023; Ullman, 2023). Tests that were
developed for studying animal cognition or the human mind rely on a range of
assumptions (e.g. regarding life cycles, ballpark estimates of memory and
learning capacities, and embodiment) which may not hold for AI systems
(Narayanan and Kapoor, 2023; Mitchell, 2023). Another problem is that
established benchmarks including such experiments suffer validity problems due
to “memorisation”, where the correct answers may have inadvertently been
learned from textual descriptions in AI assistant training data (Schaeffer et
al., 2023; de Wynter et al., 2023; Mitchell, 2023).

Benchmarking as a method to evaluate AI systems faces a range of further
limitations. First, benchmark datasets can face the same limitations as any
other dataset and be too small, narrow, or biased for the task at hand. Like
any other capability layer evaluation, benchmarks are limited in assessing
ethical or social risk because they do not capture important context.
Benchmarks are limited datasets and often permit narrower inferences than
their title implies (Raji et al., 2021; Schlangen, 2019; Narayanan and Kapoor,
2023). This explains why benchmarks may yield incompatible results: one paper
describes the “benchmark lottery”, whereby model performance may seem high on
one benchmark and low on another benchmark purportedly testing the same
construct (Dehghani et al., 2021). To avoid narrow benchmarks and running into
Goodhart’s Law (see Roles and responsibilities), one approach is to create
“dynamic”, continuously evolving benchmarks, where the underlying datasets are
updated over time (see, for example, GEM, Gehrmann et al. (2021); Kiela et al.
(2021a)). An additional, emerging practice uses pre-trained models to evaluate
the outputs of other AI systems (OpenAI, 2023b; Bai et al., 2022; Wu et al.,
2023). This enables dynamic testing and creates some degree of flexibility in
what is evaluated. This approach could operationalise a user-specified
definition of harm, supplied via a prompt or small fine-tuning dataset, thus
creating a versatile avenue for highly bespoke probing of an AI system. This
automatised approach allows for faster iteration on how a harm is
operationalised and can be simpler than collecting an entirely new static
evaluation dataset for each new definition or operationalisation of harm.
However, this approach is limited by its use of the same type of models the
evaluation seeks to measure. Without grounding in human evaluation, there is a
risk of amplifying harmful behaviours that models exhibit but cannot identify
(see Model-driven evaluation may fill gaps).

A second set of challenges is that benchmarks are often flawed or constructed
without necessary care and documentation, raising validity concerns (Blodgett
et al., 2021; Liao et al., 2021). To overcome the narrowness problem, multiple
benchmarks can be aggregated into a “test suite” (e.g. HELM, Liang et al.
(2022), BIG Bench, Srivastava et al. (2023), Safetykit, Dinan et al. (2022)).
However, more complex benchmarks may still not capture relevant context or
aspects of the target construct. Furthermore, collapsing multiple tests into a
single result can make it harder to interpret results and can mask issues such
as disparate performance for different groups, as scores are aggregated
(Burnell et al., 2023).

Benchmarks are also limited in terms of external validity – that is, with
respect to how well results generalise to novel instances (de Vries et al.,
2020). This problem is particularly pronounced in benchmarks that transpose
tests designed for human cognition to AI systems, without checking that
underlying assumptions hold (Binz and Schulz, 2023; Bubeck et al., 2023;
Shiffrin and Mitchell, 2023; Schaeffer et al., 2023; de Wynter et al., 2023;
Mitchell, 2023).

Human annotation can be used to develop automated benchmarks, which simulate
human ratings on novel outputs. Overall, there is a push toward automating
testing to reduce labour and time cost, and to reduce human exposure to
potentially harmful model outputs. However, automated tests built on human
annotation introduce additional noise, as they often fail to accurately
represent human judgements and are inevitably less accurate (van Miltenburg et
al., 2020; Liu et al., 2017; Novikova et al., 2018). This situation can then
be further compounded if human annotators themselves use AI systems to assist
the data annotation work (Veselovsky et al., 2023). Recent participatory
efforts aim to address these limitations by creating avenues for meaningful
input from people from different cultural backgrounds and better benchmarks to
identify cultural tropes or stereotypes in images (Qadri et al., 2023b).

Finally, benchmark testing alone may leave unknown capabilities and failure
modes undetected. While AI developers have the most technical knowledge about
how generative AI models work, there is still a lack of understanding of what
these models are capable of (also referred to as “capability overhang”,
Shevlane et al. (2023)). This makes it particularly difficult to foresee new
model abilities – properties that materialise as the complexity of the system
increases (Wei et al., 2022) – that may cause downstream risk. These may not
be possible to predict simply by extrapolating the capabilities of smaller-
scale models (Wei et al., 2022; Schaeffer et al., 2023).

#####

Adversarial testing

Adversarial testing, also referred to as “red teaming”, refers to risk
identification exercises through adversarial attacks on AI models,
infrastructure, development and deployment environments, and deployed AI
products/systems. It is a mode of evaluation targeted at finding
vulnerabilities in AI systems that can be exploited to get an AI system to
output harmful content (Brundage et al., 2020; Casper et al., 2023; Ganguli et
al., 2022; Wei et al., 2023; Millière, 2022). These attacks can be conducted
in any modality or across modalities. They are particularly suitable for
assessing model vulnerabilities or the friction users encounter when seeking
to obtain harmful material. For example, targeted attacks based on textual
inputs can be used to produce harmful image outputs (Yu and Rieser, 2023; Ma
et al., 2021). Adversarial testing can be conducted by humans or it can be
automated, such as by leveraging large language models to perform adversarial
testing (Perez et al., 2022a; Yang et al., 2023; Zou et al., 2023). Much like
automated benchmark development based on human data, successful adversarial
testing prompts can be reused to automate future testing.

Open-ended probing is a method for surfacing unexpected risks and risks that
arise in response to innocuous outputs. In the context of safety evaluation,
such exploratory probing is a form of “red teaming”: researchers may centre a
given risk area, such as eating disorder content, and then perform exploratory
prompting of AI systems to explore biases and patterns in model responses
(Center for Countering Digital Hate, 2023a). Exploratory safety evaluation may
also include the visualisation of thematic clusters in training data to help
surface potential risks of representation harms (KnowYourData, Google  REVISE,
Wang et al. (2021)). Exploring the provenance of data can indicate unexpected
risks – for example, to what extent private or copyrighted data is present in
the dataset, data which may in turn be leaked or cause privacy risks (Dodge et
al., 2021; Choi et al., 2023; Wang et al., 2022; Kreutzer et al., 2022). Open-
ended probing is limited in that it does not assess specific harm areas or
test hypotheses about particular AI system capabilities. It is often a helpful
first step to identify leads for more directed evaluation.

Adversarial testing more broadly is subject to the same limitations as human
testing and benchmarking outlined above. In addition, adversarial testing is
limited by the imagination, contextual knowledge, and skill as probers seek to
compromise the AI system. Despite adversarial testing, novel failure modes are
typically discovered when an AI system is released to the broader public (Wei
et al., 2023). Automated red teaming presents a highly scalable approach but
so far has yielded lower-quality tests than human annotation. Further
assurances on the quality and reliability of such tests are therefore needed
(Perez et al., 2022b; Mozes et al., 2023).

####  A.2.2 Human interaction layer

#####

Behavioural experiments

Pathways by which an AI system may cause harm to the person interacting with
it can be assessed through controlled studies leveraging psychology,
human–computer interaction, or behavioural economics methodologies (Lee et
al., 2023b; Tahaei et al., 2023). These methods isolate variables of interest,
often in highly controlled settings. They may target outcomes from
interactions or mechanisms by which potential harm may occur – such as the
common propensity for people to believe misinformation more readily if it
aligns with their views (Lodge and Taber, 2013). Behavioural experiments
typically aim for some level of generality, rather than isolating specific use
cases or application designs. This lends them to the study of potential
impacts through human interaction of “general purpose” technologies, such as
AI systems whose application domain and product design is not yet defined.

Experiments face a realism trade-off whereby highly contrived laboratory
experiments give better insight into causal mechanisms but may not extend to
real-world use cases. Another challenge is that such experiments are typically
limited in the number of participants, and small-scale studies may fail to
detect harms with small effect sizes, which may only become visible at scale –
requiring either passive monitoring or other forms of larger-scale analysis.
Results may be further confounded due to unrepresentative participant pools,
which are often white, educated, and from industrialised, rich, and democratic
locales (Henrich et al., 2010). Similarly to human annotation, behavioural
experiments have among their practical limitations the fact that they are
time-, labour-, and cost-intensive, and may expose human participants to
potential harm. Such experiments also require expertise in experiment design,
user interfaces, and internal review processes suited to the ethical
considerations that arise in this kind of research, some of which may be rare
in organisations that develop AI systems (Jackman and Kanerva, 2016;
Zevenbergen, 2020; Jindal, 2022).

#####

User research

User research and usability testing can evaluate the needs and behaviours of
users, as well as the functionality and possible externalities that AI systems
may create at the point of use.141414Note that these tests are not often
subject to the same ethics scrutiny as psychology experiments. This has led to
user studies that adversely impacted user well-being (Jouhki et al., 2016). It
can be conducted in contrived environments to isolate specific variables, such
as the impact of particular features or the user interface of applications.
User studies can also be conducted “in the wild”, by observing how people use
AI systems in real-world scenarios. User testing may include behavioural
experiments but can also include interviews, talk-out-loud studies, or
surveys.

Experimental user studies resemble psychology and human–computer interaction