.

# Towards a multi-stakeholder value-based assessment framework for algorithmic
systems

Mireia Yurrita<EMAIL_ADDRESS>0000-0002-9685-4873 Delft
University of TechnologyDelftThe Netherlands2628 CE ,  Dave Murray-Rust
0000-0001-6098-7861 Delft University of TechnologyDelftThe Netherlands
<EMAIL_ADDRESS>,  Agathe Balayn  0000-0003-2725-5305 Delft
University of TechnologyDelftThe Netherlands<EMAIL_ADDRESS>and
Alessandro Bozzon  0000-0002-3300-2913 Delft University of TechnologyDelftThe
Netherlands<EMAIL_ADDRESS>
(2018; 2022)

###### Abstract.

In an effort to regulate Machine Learning-driven (ML) systems, current
auditing processes mostly focus on detecting harmful algorithmic biases. While
these strategies have proven to be impactful, some values outlined in
documents dealing with ethics in ML-driven systems are still underrepresented
in auditing processes. Such unaddressed values mainly deal with contextual
factors that cannot be easily quantified. In this paper, we develop a value-
based assessment framework that is not limited to bias auditing and that
covers prominent ethical principles for algorithmic systems. Our framework
presents a circular arrangement of values with two bipolar dimensions that
make common motivations and potential tensions explicit. In order to
operationalize these high-level principles, values are then broken down into
specific criteria and their manifestations. However, some of these value-
specific criteria are mutually exclusive and require negotiation. As opposed
to some other auditing frameworks that merely rely on ML researchers’ and
practitioners’ input, we argue that it is necessary to include stakeholders
that present diverse standpoints to systematically negotiate and consolidate
value and criteria tensions. To that end, we map stakeholders with different
insight needs, and assign tailored means for communicating value
manifestations to them. We, therefore, contribute to current ML auditing
practices with an assessment framework that visualizes closeness and tensions
between values and we give guidelines on how to operationalize them, while
opening up the evaluation and deliberation process to a wide range of
stakeholders.

values, ML development and deployment pipeline, algorithm assessment, multi-
stakeholder

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††journalyear: 2022††copyright:
rightsretained††conference: 2022 ACM Conference on Fairness, Accountability,
and Transparency; June 21–24, 2022; Seoul, Republic of Korea††booktitle: 2022
ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22), June
21–24, 2022, Seoul, Republic of Korea††doi: 10.1145/3531146.3533118††isbn:
978-1-4503-9352-2/22/06††ccs: General and reference Evaluation††ccs: Human-
centered computing Human computer interaction (HCI)††ccs: Social and
professional topics User characteristics

##  1\. Introduction

In recent years, it has become clear that algorithmic systems might encode
harmful biases and might lead to unfair outcomes (Morley et al., 2020; Shen et
al., 2021). The dangers of using Machine Learning (ML) in Computer Vision (CV)
(Buolamwini and Gebru, 2018) or Natural Language Processing (NLP) (Davidson et
al., 2019; Sap et al., 2019; Bender et al., 2021; Larsonneur, 2021; Alshaabi
et al., 2021), for assessing recidivism (Tolan et al., 2019), for candidate
screening (Raghavan et al., 2020) and for recommending content on social media
platforms (Zubiaga et al., 2019; Parisi and Comunello, 2020; Kaiser and
Rauchfleisch, 2020; Ribeiro et al., 2020) have been pinpointed. The origins of
harmful algorithmic bias111Following the approach adopted by Shen et al. (Shen
et al., 2021), we will distinguish between harmful algorithmic biases and
harmful algorithmic behaviors, since not all harmful algorithmic behaviors
originate from biases and not all algorithmic biases are necessarily harmful
(Blodgett et al., 2020). might be diverse (Shen et al., 2021; Suresh and
Guttag, 2021). Just to mention a few, representativeness issues, play a key
role in disparate algorithmic performance (Chasalow and Levy, 2021; AI Ethics
Impact Group (AIEIG), 2020). The way in which data is collected (Bender et
al., 2021; Paullada et al., 2020) and labelled (Crawford and Paglen, 2019;
Paullada et al., 2020; Draws et al., 2021) is a major menace to data
soundness. Beyond the data generation process, aggregation, learning,
evaluation and deployment biases have been identified throughout the ML
pipeline (Suresh and Guttag, 2021). In response to harmful algorithmic bias,
current auditing processes 222We will use the term auditing processes to refer
to external audits, where third parties only have access to model outputs
(Sandvig et al., 2014). We will use the term assessment processes to refer to
an evaluation process that is applied “throughout the development process and
that enables proactive ethical intervention methods” (Raji et al., 2020). We
will not use the term Internal Audit defined by Raji and Smart (Raji et al.,
2020) to avoid erroneous inferences that would limit the stakeholders of our
framework to the employees of an organization. (Saleiro et al., 2018; Wilson
et al., 2021; Adler et al., 2018) have provided numerous useful bias detection
techniques (Hube et al., 2019; Ghai et al., 2020; Xu et al., 2018; Amini et
al., 2019; Edizel et al., 2020; Ball-Burack et al., 2021; Bellamy et al.,
2019; Zhang et al., 2018; Yan and Howe, 2020).

However, harmful algorithmic behavior is not limited to biases encoded in the
ML life cycle (Shen et al., 2021). The lack of social and cultural context in
the mathematical representation of socio-technical systems (Martin et al.,
2020; Shen et al., 2021) or the omission of changing practices and long-term
effects of the deployed systems (Kuijer and Giaccardi, 2018; D’Amour et al.,
2020; Bountouridis et al., 2019; Ionescu et al., 2021) are also some
problematic aspects that are hardly considered in current auditing processes.
Such processes mostly consist of quantitative analysis for assessing the
conformance of those systems to applicable standards (IEEE, 2008), rather than
additionally gaining insights into their contextual implications (Shen et al.,
2021; Raji et al., 2020). Furthermore, these auditing approaches solely rely
on ML researchers, and practitioners, who can fail to detect issues that arise
from context-dependent unanticipated circumstances during usage time (Shen et
al., 2021).

In this paper, we argue that: firstly, assessment processes for algorithmic
systems should go beyond bias auditing and take into account additional high-
level values 333We will adopt the definition of values used in philosophy of
science, following Birhane et al. (Birhane et al., 2021). Values of an entity
are, thus, defined as properties that are desirable for that kind of entity.
that are outlined in Artificial Intelligence (AI) ethics documents (European
Commission, 2019; OECD, 2019; China Electronics Standardization Institute,
2018; Japanese Cabinet Office and Innovation, 2019; United States Executive
Office of the President and on Technology, 2016; Amnesty International, 2018;
The Royal Society, 2019; Google, 2018; IBM, 2019; Microsoft, 2018; Fjeld et
al., 2020). Contestability, for example, has been identified as a key value of
algorithmic systems, but there is still little guidance on what contestability
requires (Lyons et al., 2021). In order to provide a good coverage of values
that deal with principled algorithmic behavior, we develop a value-based
assessment framework, where contextual conditions are considered along with
quantifiable measurements. We organize such values in a circular layout with
two bipolar dimensions. As claimed by Friedman et al. (Friedman et al., 2017),
values do not exist in isolation. They are situated in a delicate balance and
touching one value might have implications in another value (Friedman et al.,
2017). This means that value interactions need to be taken into account when
making choices about value prioritization and situating algorithmic systems in
a space of trade-offs (Bakalar et al., 2021). The circularity of our framework
makes such interactions explicit and facilitates the identification of common
motivations and tensions among values.

Secondly, an assessment process should give tangible guidelines for the
operationalization 444Our strategy follows the definition by Shahin et al.
(Shahin et al., 2021), where “operationalizing values” is defined as the
process of identifying values and translating them into concrete system
specifications that can be implemented. of values, so as to eventually put
ethics into practice following a context-aware approach (Shklovski and
Némethy, 2022). To this end, each value in our framework is broken down into
criteria manifested through quantifiable indicators, process-oriented
practices or signifiers555We adopt the definition given by Don Norman in his
2013 edition of “The Design of Everyday Things”. Signifiers are perceivable
cues of an affordance, affordances being “the relationship between the
properties of an object and the capabilities of the agent that determine how
the object could be possibly used”. In this paper, the “object” in question is
the ML-driven system.. These value-specific criteria and their manifestations
can be used either as a checklist if our framework is applied for evaluating a
system that is already developed, or for promoting such values if it is being
used during design time.

Thirdly, assessment processes should allow critical reflection on algorithmic
systems and engage in conflictual plurality666We understand conflictuality as
a solution for dealing with the “figure of alterity”. Unlike conflict, it
represents a method for linking opposing views and opening out onto the
exercise of thinking (Gaillard, 2016). Inevitable value tensions inherent in
the nature of socio-technical systems (Groves, 2015) require spaces for
ethical discussions (Shklovski and Némethy, 2022), that can benefit from the
insights of multiple stakeholders beyond ML practitioners (Shen et al., 2021;
Bakalar et al., 2021). To enable fruitful multi-stakeholder discussions (Lee
and Singh, 2021), we map and match value-specific communication means with
different stakeholders. We, therefore, contribute with:

  * •

A review of prominent high-level values in AI ethics and translation into
specific criteria through the:

    * –

Design of an assessment framework that facilitates the identification of
common motivations and tensions among values encoded in ML-driven systems.

    * –

Definition of guidelines to deal with the complex middle ground between
abstract values and concrete system specifications.

  * •

Translation of value-specific criteria into manifestations that are
understandable for diverse stakeholders through the:

    * –

Review of available means to communicate value manifestations to different
stakeholders based on their insight needs and nature of knowledge.

    * –

Definition of steps to introduce those communication means into multi-
stakeholder deliberation dynamics.

The remainder of the paper is organized as follows: in section 2, we analyze
related work for documenting and auditing ML systems. We also introduce the
theoretical basis of our framework. Section 3 describes and justifies the
selected values, criteria and manifestations and their arrangement in our
framework. Section 4 maps the stakeholders involved in the algorithm
evaluation process and reviews the available means for communicating system-
specific information to them. In sections 3 and 4, we illustrate the necessary
steps for navigating our framework through an example in the area of life
insurance application. We discuss our approach, its implications, and future
lines of work in section 5, and we conclude this paper in section 6.

##  2\. Background and related work

In this section, we survey current practices for documenting and auditing
technical specifications of algorithmic systems. We also provide the
theoretical basis of our framework.

###  2.1. Background

####  2.1.1. Standardized documentation.

In order to facilitate the audit of ML-driven systems, it is important that
technical specifications are collected and documented in a standardized way.
So far, ML system documentation practices are limited to datasets and models.

Documenting datasets. Recent studies in documentation practices for ML
datasets claim the need for greater data transparency (Hutchinson et al.,
2021). Since the quality of the prediction made by the ML system highly
depends on the way the data has been collected, the need for setting rigorous
practices (as it is the case in other areas of knowledge, such as social
sciences or humanities (Geiger et al., 2020)) has been highlighted (Paullada
et al., 2020). Likewise, the choice of what data to collect and how to collect
this data is in itself a value-laden decision (Denton et al., 2020; Scheuerman
et al., 2021). To standardize documentation for ML datasets and make data-
related decisions more transparent for other practitioners, various
methodologies have been suggested in the last years, “Datasheets” (Gebru et
al., 2020) and “Dataset Nutrition Labels” (Holland et al., 2018), for
instance. For NLP techniques, “Data Statements” are regarded as a dataset
characterization approach that helps developers anticipate biases in language
technology and understand how these can be better deployed (Bender and
Friedman, 2018).

Documenting models. In addition to documenting datasets, the importance of
disclosing the technical characteristics of ML models has also been
emphasized. A good example of model documentation practices are the “Model
Cards” (Mitchell et al., 2018).

####  2.1.2. Auditing techniques.

Various methodologies and tools for incorporating auditing tasks into the
Machine Learning workflow have been suggested. Aequitas (Saleiro et al., 2018)
is an open source toolkit to detect traces of bias in models. The toolkit
designed by Saleiro et al. (Saleiro et al., 2018) facilitates the creation of
equitable algorithmic decision-making systems where data scientists and
policymakers can easily use Aequitas for model selection, evaluation and
approval. Wilson et al. (Wilson et al., 2021) described a framework that helps
ensure fairness in socio-technical systems, and used it for auditing the model
of the startup pymetrics. Adler et al. (Adler et al., 2018) studied auditing
techniques for black-box models to discover whether proxy variables linked to
sensitive attributes indirectly influence the predictions of the system. The
end-to-end “Internal Audit Framework” suggested by Raji and Smart (Raji et
al., 2020) is of special interest for justifying the need of setting specific
guidelines to enable multi-stakeholder deliberation in assessment processes.
It consists of five main stages where the need for stakeholder diversity is
highlighted, e.g. the scoping stage calls for covering a “critical range of
viewpoints” to review the ethical implications of the system use case.

####  2.1.3. Motivation.

While standardized documentation practices (Gebru et al., 2020; Mitchell et
al., 2018; Holland et al., 2018; Bender and Friedman, 2018) and audits
(Saleiro et al., 2018; Adler et al., 2018; Wilson et al., 2021) have been
influential methodologies for dealing with harmful algorithmic bias, their
scope is limited to performing quantitative analysis over data and model
outputs so as to ensure compliance with applicable standards (IEEE, 2008).
Such an approach does not deal with additional ethical values which cannot be
easily quantified (Lee and Singh, 2021) and that are essential for ensuring
desirable algorithmic behavior. One could argue that “Datasheets” (Gebru et
al., 2020) and “ModelCards” (Mitchell et al., 2018) already devote a section
to the description of ethical considerations of datasets and models. Yet,
there are no specific guidelines on how to identify ethical issues. As
Shklovski et al. (Shklovski and Némethy, 2022) discovered, technical people
both in industry and academia struggle to identify what an ethical issue
entails. To address this caveat, as part of our value-based framework, we give
tangible guidelines for putting ethics into practice (Shklovski and Némethy,
2022; Morley et al., 2020). We operationalize each high-level value into
actionable value criteria and their manifestations. One could also argue that
Raji and Smart (Raji et al., 2020) already included an Ethics Review as part
of their end-to-end internal audit framework. Indeed, they exemplified such a
review by describing ethical considerations and potential mitigation
strategies against bias and privacy threats for a smile detection system.
However, this review does not address most of the values that are referred in
AI ethics documents. We fill in this gap by offering a good coverage of values
to examine, including those that normally go unnoticed in current documenting
and auditing practices.

###  2.2. Accounting for human values in the assessment of algorithmic systems

Our ML assessment framework identifies and arranges values encoded in
algorithmic systems by covering prominent principles in AI ethics and
organizing them in a circular structure.

####  2.2.1. Addressing human values in technology.

For the definition of our value-based framework, we followed other
theoretically grounded approaches, such as Value Sensitive Design (VSD)
(Friedman et al., 2017). VSD represents a pioneering endeavour where human
values are proactively considered throughout the process of technology design
(Davis and Nathan, 2015). Just as VSD does with interactive systems, we
address the need to account for human values during the design,
implementation, use, and evaluation (Davis and Nathan, 2015) of algorithmic
systems. To this end, we select and define values involved in ML-driven
systems, and we identify stakeholders that will be in contact with such
systems and whose standpoints need to be considered. Our approach resonates
with conceptual investigations described in VSD literature (Davis and Nathan,
2015).

The circular nature of our framework is inspired by Schwartz’s Theory of Basic
Human Values (Schwartz, 2012). This theory identifies individual value
priorities based on ten basic personal values. Values are arranged in a
circular form and categorized in four quadrants. These quadrants are located
in two bipolar dimensions, which visualize “oppositions between competing
values”. In addition, adjacency between values denotes a common motivation,
which results in these values forming a circular continuum. The advantage of
adopting a circular arrangement, like the one suggested by Schwartz, for ML-
driven systems is that value commonalities and trade-offs can be easily
identified thanks to their positioning. Considering the struggles of technical
people when addressing ethical issues (Shklovski and Némethy, 2022), an
explicit representation of value interactions will facilitate the analysis of
trade-offs and decision-making about value prioritization.

####  2.2.2. Ethical principles for ML-driven systems

The values considered in our assessment framework cover prominent principles
outlined in AI ethics. In the last five years, many institutions have studied
and defined high-level principles that AI systems should follow (Fjeld et al.,
2020). As a matter of fact, documents that aim at guiding the “ethical
development, deployment and governance of AI” are converging into a common set
of principles (Mittelstadt, 2019; Morley et al., 2020). However, high-level
principles are far from being actionable (Morley et al., 2020) and it is
necessary to provide answers on how to proceed (Aizenberg and van den Hoven,
2020). Efforts for going from “what” to “how” 777Expression used by Morley et
al. (Morley et al., 2020) to refer to the operationalization of ethical
principles in AI. The ’what’ refers to the ethical principles themselves,
whereas the ’how’ refers to the act of putting such principles into practice.
include the review carried out by Morley et al. (Morley et al., 2020), where
available tools for operationalizing ethical principles were examined.
Similarly, the AI Ethics Impact (AIEI) Group designed a framework for rating
the presence of ethical principles in AI systems, getting inspiration from
energy efficiency labels (AI Ethics Impact Group (AIEIG), 2020).

Our value-based framework differs from previous applied ethics frameworks
(Morley et al., 2020; AI Ethics Impact Group (AIEIG), 2020) in various ways.
Firstly, we arrange values in a circular form, which makes it easier to
navigate common motivations and trade-offs between values. Although such
common motivations and trade-offs can be inferred from current AI ethics
documents, we make them explicit by arranging values in a geometrically
meaningful way. This is especially useful for identifying overlaps between
values that are adjacent to each other and for detecting potential value
tensions that need to be negotiated and consolidated. Secondly, we do not
limit our ethics framework to a mere checklist. We follow Shklovski et al.
(Shklovski and Némethy, 2022) and combine the enumeration of tangible and
actionable value manifestations with the generation of an open space for
ethical debate. As opposed to the deterministic approach adopted by the AIEI
group (AI Ethics Impact Group (AIEIG), 2020), we map communication means for
facilitating ethical reflections of algorithmic systems and for addressing
ethical issues in practice (Shklovski and Némethy, 2022). Thirdly, as opposed
to previous applied ethics frameworks (Morley et al., 2020; AI Ethics Impact
Group (AIEIG), 2020), we embrace diversity in ethical reflections and deal
with the complexities that arise from plurality. In order to facilitate multi-
stakeholder discussions, we match available communication means for addressing
different value manifestations with stakeholders that present different
insight needs.

##  3\. Design of our value-based framework

In this section, we describe the composition of our value-based framework and
justify its arrangement. We provide the definition of each of the selected
values and the derived criteria and manifestations.

###  3.1. Methodology for reviewing values, criteria and manifestations in ML-
driven systems

To design our framework, we analysed documents outlining high-level ethical
principles that ML systems should follow. Our starting point was the review
performed by Fjeld et al. (Fjeld et al., 2020), where principles coming from
governments, inter-governmental organizations, multiple stakeholders, the
private sector, and the civil society were examined. In their review, Fjeld et
al. identify nine key themes, some of which overlap with the values outlined
in our framework. The identification of prominent high-level values was also
complemented with other reviews (AI Ethics Impact Group (AIEIG), 2020; Birhane
et al., 2021; Morley et al., 2020; Hidalgo et al., 2021; Dasha Simons, 2019).
To identify the criteria that define the fulfilment of prominent high-level
values, we navigated the visual representation provided by Fjeld et al. (Fjeld
et al., 2020) and accessed the documents that offer a higher coverage of the
value in question. For instance, for the value of privacy, one of our main
references has been the GDPR (European Commission, 2018).

We went from criteria to value manifestations through an extensive exploration
of available value-specific reviews that identify such manifestations. For
instance, for the value of security Xiong et al. (Xiong et al., 2021)
presented a thorough study of mechanisms used for securing the ML pipeline
against external threats. For explainability, Barredo-Arrieta et al. (Barredo
Arrieta et al., 2020) put together more than four hundred references and
mapped strategies in the field of Explainable Artificial Intelligence (Barredo
Arrieta et al., 2020). We partly rely on such reviews for identifying value
manifestations because our contribution lies in covering and putting together
a set of values and their manifestations in ML-driven systems to end up with a
“health-check” for assessing algorithmic systems, rather than rediscovering
such value manifestations ourselves. Similarly, for the values of performance
and fairness, we only included the main value manifestations that represent
the basis for any other derived metrics. That is to say, just as Verma et al.
(Verma and Rubin, 2018) did, we outline the main quantifiable indicators
(false positives, false negatives etc) used for measuring performance and
fairness, but we are aware that many other metrics that derive from these ones
can be insightful for specific contexts. Dealing with such compound metrics is
out of the scope of this work.

###  3.2. Assessment of algorithmic systems through a circular value-based
framework

Our resulting ML assessment framework arranges values in a circular form
(figure 1). Adjacency between values denotes a common motivation and
oppositions between competing values are represented through two bipolar
dimensions. For instance, adjacency between privacy and security denotes a
common objective towards the protection of sensitive information (Fjeld et
al., 2020; Russell et al., 2015) and resilience to external threats
(Microsoft, 2018). The trade-off between privacy and explainability, on the
other hand, is made explicit by their opposing positioning in our circular
framework. High-level values are then broken down into specific criteria and
their manifestations, as indicated in figure 2. Criteria defining a specific
value ultimately represent a set of questions to be asked as part of the
assessment process to ensure the fulfillment of the value in question —if the
framework is being applied before deployment— or the promotion of a specific
value —if the framework is being applied during design time—. These sets of
criteria are not unique and exclusive to one value. For instance, when
defining the criteria for privacy we refer to “data protection”, which is also
involved in security in the form of “resilience to attacks”. These overlaps
are precisely what we want to highlight and make explicit thanks to the
circularity of our framework and adjacency between values.

Figure 1. Graphic representation of our circular value-based assessment
framework. Oppositions between competing values are illustrated through the
arrangement of those values in bipolar dimensions and common motivations
through adjacency between values, which form a circular continuum.

Manifestations are classified in three groups depending on their nature: (1)
Quantifiable indicators are specific measurable parameters that numerically
manifest the (lack of) adequacy in the standards set for a criterion
(magenta). (2) Process-oriented practices are actions and mechanisms
implemented during the ML development or deployment process that advocate for
a certain value (olive). (3) Signifiers 888Check footnote 5 are files and
reports that describe the relationship between the properties of the
algorithmic system and humans that determine how that system can be used
(orange). There is a many-to-many relationship between criteria and
manifestations.

In the next subsections, we present opposing value categories in pairs. We
explain in detail values, the criteria that define them and their
manifestations.

Figure 2. Workflow for operationalizing high-level values and for enabling
multi-stakeholder assessment of algorithmic systems. This workflow represents
the methodology that we followed for structuring our framework and the steps
that researchers and practitioners should take to make use of it. (1) Select
and discuss project-specific values (V), (2) Decide on criteria (C) for
embodying those values, (3) Select the manifestations (M) that enact value-
specific criteria, (4) Map relevant stakeholders (S) to enable ethical
reflection of value and criteria tensions, (5) Match adequate communication
means (CM) to stakeholders.

##### Motivating example.

For illustrative purposes, we guide the reader through each stage of our
framework with a hypothetical yet plausible use case. Consider a team of
researchers is developing an ML-driven system for automating life insurance
application processes. The system shall accept or refuse the request of a life
insurance based on the following data: physiological information of the
candidate, details about their employment, insurance history, and individual
and family medical history. As part of a new wave of ethical finance
companies, the team would like to ensure that their work is ethically
grounded. However, it is not clear what that means in practice. From looking
at prominent literature, they can develop a sense that the model should be
fair and unbiased, and potentially that there should be some level of human
control or intervention possible. Yet, they cannot be sure if they have a good
set of representative values covered, and they do not know how to go about
communicating the way that their model embodies those values. They are now
reading through our multi-stakeholder value-based assessment framework.

###  3.3. Conservation vs Openness

The first dimension of our value-based framework captures the conflict between
conservation and openness. Values included within the conservation category
emphasize the necessity of ML-driven systems to preserve confidentiality with
regards to information, as well as, the need for the system to preserve
adequate robustness when it comes to performance. On the contrary, the
category of openness encompasses values that advocate for making system
components and specifications more accessible to the public.

####  3.3.1. Conservation.

Privacy, security and performance uphold confidentiality and robustness within
ML systems (Fjeld et al., 2020).

##### Privacy

The defining goal of privacy is the need for ML-driven systems to respect
individual’s informational confidentiality (Fjeld et al., 2020; AI Ethics
Impact Group (AIEIG), 2020) as part of their user rights (Birhane et al.,
2021). When applying this value to the ML development pipeline, data
processing itself should integrate privacy standards (AI Ethics Impact Group
(AIEIG), 2020; Fjeld et al., 2020; Morley et al., 2020), so that there is no
possibility of identifying sensitive information about individuals (Hidalgo et
al., 2021; Wachter and Mittelstadt, 2019). Furthermore, the need to provide
humans with agency over their data is emphasized (Fjeld et al., 2020). Based
on these definitions, we identified six main criteria for the fulfillment of
privacy within ML systems (table LABEL:tab:exampleprivacy). (1) Consent for
data usage (European Commission, 2018; Fjeld et al., 2020; AI Ethics Impact
Group (AIEIG), 2020): data subjects should be appropriately informed when
their data is being used and their explicit approval is needed. (2)
Implementation of data protection mechanisms (Floridi, 2019; Fjeld et al.,
2020; AI Ethics Impact Group (AIEIG), 2020): during the development of
algorithmic systems, resources should be devoted to making user data
management secure and confidential. (3) Users having control over their data
and ability to restrict its processing (European Commission, 2018; Fjeld et
al., 2020): users should be able to limit the way their personal data is being
used. (4) Users having the right to rectify (European Commission, 2018; Fjeld
et al., 2020; AI Ethics Impact Group (AIEIG), 2020): users should be able to
modify their data at any time. (5) Users having the right to erase their data
(European Commission, 2018; Fjeld et al., 2020; AI Ethics Impact Group
(AIEIG), 2020): this criterion refers to the right that users have to be
forgotten. (6) Users having right to access their data (European Commission,
2018; The IEEE Global Initiative on Ethics of Autonomous and Intelligent
Systems., 2019): this right empowers users to have agency over their data.
These criteria manifest in various ways. Signifiers include: a written
declaration of consent (European Commission, 2018), detailed descriptions of
the collected data, how data is handled, how long it will be kept and the
purpose of collecting that data (Mehldau, 2007). These signifiers are
necessary for users to fully understand what sharing their data entails.
Process-oriented practices include the obfuscation of data (AI Ethics Impact
Group (AIEIG), 2020) and forms and submission mechanisms to object data
collection and make complaints (Blazevic et al., 2021).

Table 1. Illustration of how to move from values, to criteria and their manifestations for privacy. |  Value |  Criteria |  Manifestations
---|---|---|---
Conservation |  Privacy |  (1) Consent for data usage (European Commission, 2018; Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020) (2) Data protection (Floridi, 2019; Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020) (3) Control over data / ability to restrict processing (European Commission, 2018; Fjeld et al., 2020) (4) Right to rectification (European Commission, 2018; Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020) (5) Right to erase the data (European Commission, 2018; Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020) (6) Right of access by data subject, data agency (European Commission, 2018; The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems., 2019) |  • Written declaration of consent (European Commission, 2018) • Description of what data is collected (Mehldau, 2007) • Description of how data is handled (Mehldau, 2007) • Purpose statement of data collection (Mehldau, 2007) • Statement of how long the data is kept (Mehldau, 2007) • Form and submission mechanisms to object data collection and to make complaints (Blazevic et al., 2021) • Obfuscation of data (AI Ethics Impact Group (AIEIG), 2020)

##### Security

Definitions characterizing security —table LABEL:tab:examplesecurity—
highlight the need for ML systems to be (1) resilient to potential maleficent
attacks (Morley et al., 2020; Fjeld et al., 2020) and to present a (2)
predictable (Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020;
European Commission, 2019) and (3) robust (AI Ethics Impact Group (AIEIG),
2020) behavior at any time. This includes implementing mechanisms to protect
user privacy, such as strategies that ensure that inferences about an
individual cannot be made by interrogating the model (Microsoft, 2018; Hidalgo
et al., 2021; Wachter and Mittelstadt, 2019). Following the survey performed
by Xiong et al. (Xiong et al., 2021), different methodologies that aim at
protecting algorithmic systems against external threats (process-oriented
practices) have been classified into two main groups. The first group consists
of defence methods against integrity threats at two different stages of the ML
pipeline: during training time (Biggio and Roli, 2018; Cretu et al., 2008;
Globerson and Roweis, 2006) and during prediction time (Biggio and Roli, 2018;
Lyu et al., 2015; Goodfellow et al., 2014; Papernot et al., 2016). The second
group aim at defending the ML system against privacy threats, namely
membership inference attacks (Shokri et al., 2016; Nasr et al., 2018; Ye et
al., 2021; Dwork et al., 2006; Jia et al., 2019).

Table 2. Criteria and their manifestations for security. |  Value |  Criteria |  Manifestations
---|---|---|---
Conservation |  Security |  • • • • • • • • • (1) Resilience to attacks: protection of privacy (Microsoft, 2018; Hidalgo et al., 2021; Wachter and Mittelstadt, 2019), vulnerabilities, fallback plans (Morley et al., 2020; Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020; Google, 2018) (2) Predictability (Fjeld et al., 2020; AI Ethics Impact Group (AIEIG), 2020; European Commission, 2019) (3) Robustness / reliability: prevent manipulation (AI Ethics Impact Group (AIEIG), 2020) |  AGAINST INTEGRITY THREATS (Xiong et al., 2021):  • Training time (Xiong et al., 2021) Ex.: • Data sanitization 999It ensures data soundness by identifying abnormal input samples and by removing them (Xiong et al., 2021). (Biggio and Roli, 2018; Cretu et al., 2008) • Robust learning 101010It ensures that algorithms are trained on statistically robust datasets, with little sensitivity to outliers (Xiong et al., 2021). (Biggio and Roli, 2018; Globerson and Roweis, 2006) • Prediction time (Xiong et al., 2021) • Model enhancement (Biggio and Roli, 2018; Lyu et al., 2015; Goodfellow et al., 2014; Papernot et al., 2016) Ex.: • Adversarial Learning 111111Adversarial samples are introduced to the training set (Xiong et al., 2021). • Gradient masking 121212Input gradients are modified to enhance model robustness (Xiong et al., 2021). • Defensive Distillation 131313The dimensionality of the network is reduced (Xiong et al., 2021). AGAINST PRIVACY THREATS (Xiong et al., 2021):  • Mitigation techniques (Nasr et al., 2018): • Restrict prediction vector to top k classes 141414Applicable when the number of classes is very large. Even if the model only outputs the most likely k classes, it will still be useful (Shokri et al., 2016). (Shokri et al., 2016) • Coarsen the precision of the prediction vector 151515It consists in rounding the classification probabilities down (Shokri et al., 2016). (Shokri et al., 2016) • Increase entropy of the prediction vector 161616Modification of the softmax layer (in neural networks) to increase its normalizing temperature (Shokri et al., 2016). (Shokri et al., 2016) • Use regularization 171717Technique to avoid overfitting in ML that penalizes large parameters by adding a regularization factor $\lambda$ to the loss function (Shokri et al., 2016). (Shokri et al., 2016; Kaya et al., 2020) • Differential privacy mechanisms (Nasr et al., 2018): • Differential privacy 181818It prevents any adversary from distinguishing the predictions of a model when its training dataset is used compared to when other dataset is used (Ye et al., 2021) (Ye et al., 2021; Dwork et al., 2006). Ex.: • Adversarial regularization 191919Membership privacy is modeled as a min-max optimization problem, where a model is trained to achieve minimum loss of accuracy and maximum robustness against the strongest inference attack (Nasr et al., 2018). (Nasr et al., 2018) • MemGuard 202020Noise is added to the confidence vector of the attacker so as to mislead the attacker’s classifier (Jia et al., 2019) (Jia et al., 2019)

##### Performance

The value of performance —table LABEL:tab:exampleperformance— is defined by
the (1) correctness of predictions (Fjeld et al., 2020; European Commission,
2019), along with the (2-5) resources necessary to reach such predictions
(Birhane et al., 2021; AI Ethics Impact Group (AIEIG), 2020; Kulynych et al.,
2018). The conditions under which systems are evaluated will have a direct
impact on the “appropriateness score” that these systems will obtain in the
form of a quantifiable indicator (Dotan and Milli, 2019). In other words, if
the level of performance is solely measured in terms of accuracy, regardless
of the needed data, prerequisites will be inherently favoring big “data-
hungry” (Lee, 1973) models. As far as the measurement of performance is
concerned, this is mainly done through quantifiable indicators, either
referring to the preciseness of the results (Mitchell et al., 2018; Wexler et
al., 2019) or to the estimated consumption of environmental resources (García-
Martín et al., 2019; Gao et al., 2020; Mahendran, 2021; Assran et al., 2019;
Dalton et al., 2019).

Table 3. Criteria and their manifestations for performance. |  Value |  Criteria |  Manifestations
---|---|---|---
Conservation |  Performance |  (1) Correctness of predictions (Fjeld et al., 2020; European Commission, 2019; Birhane et al., 2021) (2) Memory efficiency (Birhane et al., 2021; AI Ethics Impact Group (AIEIG), 2020) (3) Training efficiency (Birhane et al., 2021) (4) Energy efficiency (Birhane et al., 2021; AI Ethics Impact Group (AIEIG), 2020) (5) Data efficiency (Birhane et al., 2021) |  • Accuracy (for classification, sum of true positive and true negative rates) (Mitchell et al., 2018; Wexler et al., 2019) • False Positive and False Negative rates (Mitchell et al., 2018; Wexler et al., 2019) • False Discovery and Omission Rate (Mitchell et al., 2018) • Mean and median error (Wexler et al., 2019) • R2 score (Bird et al., 2020) • Precision and recall rates (Wexler et al., 2019) • Area under ROC curve (AUC) (Bird et al., 2020) • Estimation of energy consumption through (García-Martín et al., 2019): • performance counters • simulation • instruction- or architecture-level estimations • real-time estimation • Estimation of GPU memory consumption (Gao et al., 2020; Mahendran, 2021) • Wall-clock training time (Assran et al., 2019; Dalton et al., 2019)

####  3.3.2. Openness.

Transparency and explainability advocate for making system components and
specifications accessible.

##### Transparency

Documents providing high-level principles for AI define transparency as the
property that enables traceability and monitoring of algorithmic systems
(Morley et al., 2020; Fjeld et al., 2020) —table
LABEL:tab:exampletransparency—. Transparency relates to the right to
information (Fjeld et al., 2020) and requires that data or algorithms present
some level of accessibility (The Royal Society, 2019). That is to say, data
and models should present some level of (1) interpretability (The Royal
Society, 2019; Birhane et al., 2021), so as to (2) enable human oversight
(Fjeld et al., 2020; Morley et al., 2020). Those data and models should also
be (3) accessible (AI Ethics Impact Group (AIEIG), 2020; The Royal Society,
2019; Fjeld et al., 2020), as a step towards achieving (4) traceability
(Morley et al., 2020) and (5) reproducibility (Birhane et al., 2021).
Manifestations of such criteria emerge mostly in the form of documentation
detailing technical aspects of the algorithmic system (considered signifiers
in our framework) (Gebru et al., 2020; Bender and Friedman, 2018; Chasalow and
Levy, 2021; Geiger et al., 2020; AI Ethics Impact Group (AIEIG), 2020; Morley
et al., 2020; Mitchell et al., 2018; The Royal Society, 2019). Process-
oriented practices mostly focus on giving open access to data and algorithms
(Fjeld et al., 2020; Birhane et al., 2021; AI Ethics Impact Group (AIEIG),
2020; The Royal Society, 2019), regularly reporting key information about the
system (Fjeld et al., 2020) and notifying users whenever they are being
subject to or interacting with an algorithmic system (Fjeld et al., 2020).

Table 4. Criteria and their manifestations for transparency. |  Value |  Criteria |  Manifestations
---|---|---|---
Openness |  Transparency |  (1) Interpretability of data and models (The Royal Society, 2019; Birhane et al., 2021) (2) Enabling human oversight of operations (Fjeld et al., 2020; Morley et al., 2020) (3) Accessibility of data and algorithm (AI Ethics Impact Group (AIEIG), 2020; The Royal Society, 2019; Fjeld et al., 2020) (4) Traceability (Morley et al., 2020) (5) Reproducibility (Birhane et al., 2021) |  • Description of data generation process (Gebru et al., 2020; Bender and Friedman, 2018; Chasalow and Levy, 2021; Geiger et al., 2020; AI Ethics Impact Group (AIEIG), 2020; Morley et al., 2020) • Disclosure of origin and properties of models and data (Mitchell et al., 2018; AI Ethics Impact Group (AIEIG), 2020; The Royal Society, 2019) • Open access to data and algorithm (Fjeld et al., 2020; Birhane et al., 2021; AI Ethics Impact Group (AIEIG), 2020; The Royal Society, 2019) • Notification of usage/interaction (Fjeld et al., 2020) • Regular reporting (Fjeld et al., 2020)

##### Explainability

Explainable Artificial Intelligence (XAI) is formed by a set of techniques
that allow a wide range of stakeholders to understand why or how a decision
was reached by an algorithmic system (The Royal Society, 2019; Floridi, 2019).
Explainability is, thus, conceived as —table LABEL:tab:exampleexplainability—
an interface that translates reasoning mechanisms of the system into formats
that are (1) comprehensible (Barredo Arrieta et al., 2020; Fjeld et al., 2020;
Floridi et al., 2018; Floridi, 2019; European Commission, 2019; OECD, 2019;
Birhane et al., 2021; The Royal Society, 2019). In addition, strategies for
making black-box algorithms more interpretable facilitate their (2) monitoring
(Morley et al., 2020) and, therefore, make them (3) suitable for evaluation
(Fjeld et al., 2020; Morley et al., 2020). XAI techniques (process-oriented
practices) are very diverse in nature. As claimed by Vera Liao et al. (Liao et
al., 2020) and Barredo-Arrieta et al. (Barredo Arrieta et al., 2020),
explainability methodologies are usually classified by the scope of the
explanation, complexity of the model, model specificity and the stage of the
ML pipeline where such a strategy is to be used. For our framework, we will
consider that explainable models can be either (a) interpretable by design or
they can be (b) explained by additional post-hoc explanations (Barredo Arrieta
et al., 2020).

Table 5. Criteria and their manifestations for explainability. |  Value |  Criteria |  Manifestations
---|---|---|---
Openness |  Explainability |  (1) Ability to understand AI systems and the decision reached (Floridi et al., 2018; Floridi, 2019; European Commission, 2019; OECD, 2019; Birhane et al., 2021; The Royal Society, 2019) (2) Traceability (Morley et al., 2020) (3) Enable evaluation (Fjeld et al., 2020; Morley et al., 2020) |  • Interpretability by design (Barredo Arrieta et al., 2020) • Post-hoc explanations (Barredo Arrieta et al., 2020)

###  3.4. Universalism vs Individual Empowerment

The second dimension captures the conflict between universalism and individual
empowerment. Values included within the individual empowerment category
emphasize the defense of the decision subjects’ interests. These principles
advocate for giving decision subjects the means to oppose to the conclusion
reached and uphold the need for putting humans in the loop. Values within the
universalism category emphasize the need to equalize system behavior to all
and to ensure that such a system adheres to the interests of society as a
whole, beyond the interests of a few individuals.

####  3.4.1. Universalism.

Respect for public interest, fairness and non-discrimination uphold the need
to ensure equitable and socially acceptable system behavior for all.

Table 6. Criteria and their manifestations for respect for public interest. |  Value |  Criteria |  Manifestations
---|---|---|---
Universalism |  Respect for public interest |  (1) Desirability of technology (Chasalow and Levy, 2021; Abebe et al., 2020; Keyes et al., 2019) (2) Benefit to society (Fjeld et al., 2020; Floridi et al., 2018; Floridi, 2019; Morley et al., 2020) (3) Environmental impact (AI Ethics Impact Group (AIEIG), 2020; Bender et al., 2021) |  • Diverse and inclusive forum for discussion (Mission assigned by the French Prime Minister, 2019; Fjeld et al., 2020) • Measure of social and environmental impact (Morley et al., 2020; Raji et al., 2020; Bender et al., 2021)

##### Respect for public interest

The value of respect for public interest —table LABEL:tab:examplerespect—
deals with the (1) appropriateness of developing algorithmic systems for a
certain purpose within a specific context. As Keyes et al. (Keyes et al.,
2019) claimed, making ML-driven systems fairer, more transparent and more
accountable is insufficient if we ignore the purpose of developing and
implementing these systems in a certain context in the very first place (leins
et al., 2020; Tsarapatsanis and Aletras, 2021). Algorithmic systems should,
therefore, (2) be beneficial to society and humanity as a whole (Floridi et
al., 2018; Floridi, 2019; Morley et al., 2020; Fjeld et al., 2020), respect
law (Birhane et al., 2021) and be aligned with human norms (Fjeld et al.,
2020). This involves giving a clear justification of the purpose and benefits
of building such a system (Morley et al., 2020; Chasalow and Levy, 2021; Abebe
et al., 2020; Keyes et al., 2019), so that the deployment of the system in
question upholds public-spirited goals (Fjeld et al., 2020). Universalism aims
at protecting the welfare of all, both people and nature (Kulynych et al.,
2018). AI systems’ (3) negative impacts on environment should, therefore, be
considered and valued (AI Ethics Impact Group (AIEIG), 2020; Bender et al.,
2021). To this end, process-oriented practices include the creation of diverse
and inclusive forums for discussion (Mission assigned by the French Prime
Minister, 2019; Fjeld et al., 2020), whereas signifiers include the
qualitative measurement of social and environmental impact (Raji et al., 2020;
Bender et al., 2021; Morley et al., 2020).

##### Fairness

The value of fairness represents a complex concept that accepts multiple
definitions (Kearns and Roth, 2019; Bakalar et al., 2021), some of which
cannot be satisfied simultaneously (Kearns and Roth, 2019; Hidalgo et al.,
2021; Harrison et al., 2020). Overall, we will understand fairness in terms of
parity in output (Dodge et al., 2019) and equal treatment (AI Ethics Impact
Group (AIEIG), 2020) among individuals. When addressing more specific
definitions of fairness (1-8) —table LABEL:tab:examplefairness—, we will adopt
the approach followed by Verma et al. (Verma and Rubin, 2018), which was also
echoed by Mehrabi et al. (Mehrabi et al., 2021). ML techniques generally
conceive fairness in terms of statistical metrics (Green and Hu, 2018) and
observe whether specific quantifiable indicators are above or below the
thresholds set for a certain application. Even if error rates were equal
across groups for a certain application, if those rates are too high, the
system could still be considered unfair (Harrison et al., 2020). This means
that for our value-based framework we outline the quantifiable indicators that
are normally used for manifesting fairness-related criteria, but we do not
determine the threshold for these indicators to be considered good enough for
a specific application. Similarly, the quantifiable indicators relate to the
output of the system, rather than the outcome that these outputs lead to.

Table 7. Criteria and their manifestations for fairness. |  Value |  Criteria |  Manifestations
---|---|---|---
Universalism |  Fairness |  (1) Individual fairness 212121Similar individuals should be treated in a similar way. Diverging definitions state that: two individuals that are similar with respect to a common metric should receive the same outcome (fairness through awareness); or any protected attribute should not be used when making a decision (fairness through unawareness); or the outcome obtained by an individual should be the same if this individual belonged to a counterfactual world or group (counterfactual fairness) (Mehrabi et al., 2021). (Mehrabi et al., 2021; Dwork et al., 2011; Barredo Arrieta et al., 2020; Kusner et al., 2017) (2) Demographic parity 222222The probability of getting a positive outcome should be the same whether the individual belongs to a protected group or not (Mehrabi et al., 2021). (Harrison et al., 2020; Hidalgo et al., 2021; Srivastava et al., 2019; Mehrabi et al., 2021; Dwork et al., 2011; Kusner et al., 2017; Barredo Arrieta et al., 2020; Kearns et al., 2018; Verma and Rubin, 2018) (3) Conditional Statistical parity 232323Given a set of factors L, individuals belonging to the protected or unprotected group should have the same probability of getting a positive outcome (Mehrabi et al., 2021). (Mehrabi et al., 2021; Verma and Rubin, 2018) (4) Equality of opportunity 242424The probability for a person from class A (positive class) of getting a positive outcome, which should be the same regardless of the group (protected group or not) that the individual belongs to (Mehrabi et al., 2021). (Mehrabi et al., 2021; Hardt et al., 2016; van Berkel et al., 2021) (5) Equalized odds 252525The probability for a person from class A (positive class) of getting a positive outcome and the probability for a person from class B (negative class) of getting a negative outcome should be the same (Mehrabi et al., 2021). (Mehrabi et al., 2021) (6) Treatment equality 262626The ratio of false positives and negatives has to be the same for both groups (Mehrabi et al., 2021). (Mehrabi et al., 2021; Berk et al., 2017) (7) Test fairness 272727For any probability score S, the probability of correctly belonging to the positive class should be the same for both the protected and unprotected group (Mehrabi et al., 2021).(Mehrabi et al., 2021; Verma and Rubin, 2018; Chouldechova, 2016) (8) Procedural fairness 282828It deals with the fairness of the decision-making process that leads to the outcome in question (Grgic-Hlaca et al., 2018). (Mehrabi et al., 2021; Grgic-Hlaca et al., 2018; Kusner et al., 2017) |  • Accuracy across groups (for classification, sum of true positive and true negative rates) (Kleinberg et al., 2016; Chouldechova, 2016; Harrison et al., 2020; Morley et al., 2020) • False positive and negative rates across groups (Kleinberg et al., 2016; Chouldechova, 2016; Mehrabi et al., 2021; Wang et al., 2021; Saleiro et al., 2018) • False discovery and omission rates across groups (Mitchell et al., 2018; Saleiro et al., 2018) • Pinned AUC (Mitchell et al., 2018; Dixon et al., 2018) • Debiasing algorithms (Bellamy et al., 2019) • Election of protected classes based on user considerations (Grgic-Hlaca et al., 2018)

##### Non-discrimination

The value of non-discrimination, as defined in our framework —table
LABEL:tab:examplenondiscrim—, deals with algorithmic systems not being
socially biased (Birhane et al., 2021) and ensuring that equal accessibility
is provided to all individuals (Morley et al., 2020). This means that (1)
quality and integrity of data should be evaluated and ensured (Hidalgo et al.,
2021; Fjeld et al., 2020; Geiger et al., 2020; Paullada et al., 2020; Morley
et al., 2020) in order to prevent “socially constructed biases, inaccuracies,
errors, and mistakes” (Morley et al., 2020) from being present in the data.
Processes that safeguard inclusive data generation (Chasalow and Levy, 2021;
Geiger et al., 2020; AI Ethics Impact Group (AIEIG), 2020; Morley et al.,
2020) and analysis procedures for identifying potential biases in data and for
assessing its quality (Hidalgo et al., 2021; Fjeld et al., 2020; Geiger et
al., 2020; Paullada et al., 2020; Morley et al., 2020) are strategies that
avoid social stereotypes being codified, maintained and amplified (Hidalgo et
al., 2021). Furthermore, non-discriminatory systems should (2) ensure
diversity and inclusiveness in the design process (Fjeld et al., 2020;
European Commission, 2019; Morley et al., 2020). From a process-oriented
perspective, participants involved in the development process should, thus,
present diverse profiles (Fjeld et al., 2020; Zhou et al., [n. d.]; Lee et
al., 2019; AI Ethics Impact Group (AIEIG), 2020). Finally, giving (3) equal
access to the technology (Birhane et al., 2021; AI Ethics Impact Group
(AIEIG), 2020; Morley et al., 2020; Fjeld et al., 2020) avoids the growth of
inequalities as a consequence of deploying AI systems (Fjeld et al., 2020).

Table 8. Criteria and their manifestations for non-discrimination. |  Value |  Criteria |  Manifestations
---|---|---|---
Universalism |  Non-discrimination |  (1) Quality and integrity of data (Hidalgo et al., 2021; Fjeld et al., 2020; Geiger et al., 2020; Paullada et al., 2020; Morley et al., 2020) (2) Inclusiveness in design (Fjeld et al., 2020; European Commission, 2019; Morley et al., 2020) (3) Accessibility (Birhane et al., 2021; AI Ethics Impact Group (AIEIG), 2020; Morley et al., 2020; Fjeld et al., 2020) |  • Inclusive data generation process (Chasalow and Levy, 2021; Geiger et al., 2020; AI Ethics Impact Group (AIEIG), 2020; Morley et al., 2020) • Analysis of data for potential biases, data quality assessment (AI Ethics Impact Group (AIEIG), 2020; Mehrabi et al., 2021; Fjeld et al., 2020; Hidalgo et al., 2021; Gebru et al., 2020) • Diversity of participant in development process (Fjeld et al., 2020; Zhou et al., [n. d.]; Lee et al., 2019; AI Ethics Impact Group (AIEIG), 2020) • Access to code and technology to all (Fjeld et al., 2020; Birhane et al., 2021; AI Ethics Impact Group (AIEIG), 2020; Morley et al., 2020)

####  3.4.2. Individual empowerment.

Contestability, human control and human agency address the politics behind
algorithmic systems (Arnstein, 2019; Winner, 1980) and deal with the issues
caused by power imbalances (Chasalow and Levy, 2021; Kalluri, 2020; Birhane et
al., 2021; Lyons et al., 2021; Vaccaro et al., 2020).

##### Contestability

The value of contestability —table LABEL:tab:examplecontestab— is defined as
the value that ensures that users have the necessary information to (1) enable
argumentation against conclusions reached by algorithmic systems (Balayn and
Gürses, 2021; Lee and Baykal, 2017; Alfrink et al., 2020; Fjeld et al., 2020;
Kalluri, 2020; The Royal Society, 2019; Lyons et al., 2021; European
Commission, 2019). This involves (2) empowering citizens (Balayn and Gürses,
2021; Kalluri, 2020; European Commission, 2019) to investigate and influence
AI (Kalluri, 2020), as part of a broader regulatory approach (Lyons et al.,
2021). As a matter of fact, contestability has been identified as a “critical
aspect of future public decision-making systems” (Alfrink et al., 2020). This
implies that, from a documentation perspective (signifiers), users should be
made aware of who determines what constitutes a contestable decision, who is
accountable for it and who can contest a decision. This last point is
particularly necessary to determine whether (legal) representatives of
decision subjects can act on their behalf. The review mechanism in place and
the workflow of contestations (Lyons et al., 2021) are policy-related details
that users should also be informed about. From a process-oriented standpoint,
mechanisms for users to ask questions and to record disagreements should also
be put in place (Hirsch et al., 2017; Mitra, 2021).

Table 9. Criteria and their manifestations for contestability. |  Value |  Criteria |  Manifestations
---|---|---|---
Individual empowerment |  Contestability |  • • • • (1) Enable argumentation / negotiation against a decision (Balayn and Gürses, 2021; Lee and Baykal, 2017; Alfrink et al., 2020; Fjeld et al., 2020; Kalluri, 2020; The Royal Society, 2019; Lyons et al., 2021; European Commission, 2019) (2) Citizen empowerment (Balayn and Gürses, 2021; Kalluri, 2020; European Commission, 2019) |  • Information of who determines and what constitutes a contestable decision and who is accountable (Lyons et al., 2021) • Determination of who can contest the decision (subject or representative) (Lyons et al., 2021) • Indication of type of review in place (Lyons et al., 2021) • Information regarding the contestability workflow (Lyons et al., 2021) • Mechanisms for users to ask questions and record disagreements with system behavior (Hirsch et al., 2017; Mitra, 2021)

##### Human Control

The value of human control —table LABEL:tab:examplecontrol— addresses the
influence that data-driven technologies have over humans and that leads to a
reduction of human agency, power and control (Patel, 2021). Algorithmic
systems should be controllable (Birhane et al., 2021) and (1) subject to user
and collective influence (Birhane et al., 2021; Lee and Baykal, 2017). They
should also be (2) subject to human review (Fjeld et al., 2020). Governance
mechanisms that ensure human oversight of automated decisions are, thus,
necessary to maintain control and influence over such systems (Morley et al.,
2020). It should be possible to (3) choose how and even whether (in the very
first place) to delegate a decision to an automated system (Fjeld et al.,
2020). From a development perspective, levels of human discretion should be
established (European Commission, 2019; Microsoft, 2018) and the ability to
override decisions made by a system (European Commission, 2019) ought to be
set up by design. Once the system is deployed, it should be continuously
monitored to enable adequate intervention when necessary (Fjeld et al., 2020;
Telia Company, 2019; European Commission, 2019).

Table 10. Criteria and their manifestations for human control. |  |  Value |  Criteria |  Manifestations
---|---|---|---|---
Individual |  empowerment |  Human Control |  (1) User/collective influence (Birhane et al., 2021; Lee and Baykal, 2017) (2) Human review of automated decision (Fjeld et al., 2020) (3) Choice of how and whether to delegate (Fjeld et al., 2020) |  • Continuous monitoring of system to intervene (Fjeld et al., 2020; Telia Company, 2019; European Commission, 2019) • Establishment levels of human discretion during the use of the system (European Commission, 2019; Microsoft, 2018) • Ability to override the decision made by a system (European Commission, 2019)

##### Human Agency

The value of human agency —table LABEL:tab:exampleagency— deals with the risks
of algorithmic systems displacing human autonomy (European Commission, 2019;
Fjeld et al., 2020). As claimed by Cila et al. (Cila et al., 2020),
algorithmic systems may displace human agency in governance processes and may
undermine human autonomy. ML-driven systems advocating for human agency
should, therefore, (1) respect human autonomy (European Commission, 2019;
Fjeld et al., 2020; Morley et al., 2020) and (2) citizens’ power to decide
(Birhane et al., 2021; European Commission, 2019). In addition, (3) decision
subjects should be able to opt out of an automated decision (Fjeld et al.,
2020; European Commission, 2019). The manifestations of such criteria involve
giving knowledge and tools to users to comprehend and interact with AI systems
(European Commission, 2019) (signifier) and, from a process-oriented
perspective, providing strategies for users to self-assess the systems
(European Commission, 2019).

Table 11. Criteria and their manifestations for human agency. |  |  Value |  Criteria |  Manifestations
---|---|---|---|---
Individual |  empowerment |  Human agency |  (1) Respect for human autonomy (European Commission, 2019; Fjeld et al., 2020; Morley et al., 2020) (2) Power to decide. Ability to make informed autonomous decision (Birhane et al., 2021; European Commission, 2019) (3) Ability to opt out of an automated decision (Fjeld et al., 2020; European Commission, 2019) |  • Give knowledge and tools to comprehend and interact with AI system (European Commission, 2019) • Opportunity to self-assess the system (European Commission, 2019)

##### Selecting values, criteria and manifestations for our example use case.

Returning to the hypothetical insurance modelling team from our motivating
example (section 3.2), they decided to apply our value-based framework before
launching their system. They quickly realised that they need to consider more
values than those outlined in current auditing processes. For example
transparency, non-discrimination, supporting human agency and the public good.
They also discovered a range of methods for enacting those values: from data
handling processes that ensure anonymity and meaningful consent around the
model, to models of fairness appropriate to their case.

Although we cover prominent ethical principles in AI and the assessment of the
algorithmic system might include all of them, here we focus on a subset of
those values for illustrative purposes. We imagine that the researchers
developing the algorithmic life insurance application system want to focus on
explainability and privacy (fig 2). We assume that they are dealing with a
blackbox algorithm that is not interpretable by design. The team needs to
examine whether the algorithmic system and the decision reached are
understandable. Additionally, the deployed XAI methods should enable
traceability and evaluation of the system. As far as the explainability
manifestations are concerned, since they are dealing with a blackbox
algorithm, they need to deploy adequate post-hoc explanations. When it comes
to privacy, the data used for training and testing the algorithmic model
should have been obtained through the explicit approval of the decision
subjects. These subjects should have been informed about the nature and
purpose of the data that is collected, the way this data is handled and
stored. Decision subjects should also have agency and control over their data.
Additionally, data protection mechanisms should have been implemented to make
sure that there is no possibility of identifying sensitive (in this case
medical) data about the subjects. These two values that the team needs to
advocate for, represent some trade-offs: XAI methods uphold interpretability
of algorithmic systems and some of them even rely on comparing data instances
at inference time with those used for training the system. This would directly
violate the subjects’ right to have their data protected and confidentiality
ensured.

##  4\. Towards a multi-stakeholder critical reflection of algorithmic systems

Since there are value trade-offs, like the one outlined in our example use
case, and certain value-specific criteria are mutually exclusive, we follow
the claim made by Raji and Smart (Raji et al., 2020), and advocate for
standpoint diversity. This implies involving a wide range of stakeholders in
the negotiation process (Shklovski and Némethy, 2022) to discuss and
critically reflect on the degree to which each of the values should be
promoted in detriment of the other one and how the prioritization process
should take place. These stakeholders will possess different types of
knowledge and will present different insight needs. In this section we map
those stakeholders and match them with the most suitable communication means.

###  4.1. Methodology for identifying relevant stakeholders and communication
means

To identify relevant stakeholders, we follow the stakeholder characterization
of Suresh et al. (Suresh et al., 2021). They classified stakeholders in a two
dimensional matrix, where one dimension captured the nature of the knowledge
of the stakeholders (formal, instrumental or personal) and the second one
identified the context in which that knowledge manifests (Machine Learning,
data domain, and the general milieu). Formal knowledge entails a deep
understanding of the theories of a certain domain. Instrumental knowledge
refers to the capability of applying formal knowledge in one of the three
contexts. Personal knowledge is acquired by the participation of the subject
in a specific context. The two dimensional-matrix classification results in
nine different stakeholder profiles. To facilitate the process of mapping the
stakeholders to tailored communication means, we narrow those stakeholders
down into four categories 292929This reduced classification is backed up by
the framework employed by Barredo-Arrieta et al. (Barredo Arrieta et al.,
2020) when identifying the explainability needs of various stakeholders..

We then proceed to identify the means to communicate system-related
information to different stakeholders. We searched such means using arXiv and
Google search, so as to cover the state of the art in terms of research papers
and open source toolkits. Each search referred to specific value criteria and
manifestations, although many of the found means address more than one value.
This review does not intend to be exhaustive. We expect novel research to
address value manifestations that still present scarce resources in our
framework. Hence our review is just a snapshot of some of the available
communication means until January 2022, but we host the latest version on an
online repository 303030https://github.com/mireiayurrita/valuebasedframework
and is open to anyone’s contribution. We aim at creating a living document
that will keep growing and that will address current research gaps as time
goes by.

###  4.2. Mapping stakeholders

We characterize four main stakeholders in our framework —table
LABEL:tab:mappingstakeholders—: (1) The development team: they have the
formal, instrumental and personal knowledge in the domain of ML (Suresh et
al., 2021). They want to ensure and improve product efficiency and research
new functionalities (Barredo Arrieta et al., 2020). (2) Auditing team: they
have the formal and instrumental knowledge of the general milieu, meaning that
they are aware of the social theories behind AI, and are able to evaluate
technical specifications of ML systems. They aim at verifying model compliance
with legislation (Barredo Arrieta et al., 2020) (3) Data domain experts: they
have the theoretical (formal) and instrumental knowledge of the application
context (healthcare, economics etc.). They look forward to gaining scientific
or domain-specific knowledge (Barredo Arrieta et al., 2020; Suresh et al.,
2021), trust the model (Barredo Arrieta et al., 2020; Suresh et al., 2021) and
act based on the model output (Suresh et al., 2021). And (4) Data subjects:
they have the personal knowledge of the data domain in which the AI is being
applied and the general milieu. They aim at understanding their situation
(Barredo Arrieta et al., 2020), verifying that the decision is fair (Barredo
Arrieta et al., 2020), contesting the decision (if needed) (Suresh et al.,
2021) and understanding how their data is being used (Suresh et al., 2021).

Table 12. Description of potential stakeholders that can be brought together as part of our value-based framework. These stakeholders have been mapped following the two dimensional criteria (type of knowledge —formal, instrumental or personal— and contexts in which this knowledge manifests —ML, data domain, milieu—) outlined by Suresh et al. (Suresh et al., 2021). The nature of their knowledge and the purpose of gaining insight for each of them have also been defined. Stakeholder |  Mapping (Suresh et al., 2021) |  Nature of knowledge |  Purpose of insight
---|---|---|---
Development team |  ML, Formal + Instrumental + Personal |  • “Knowledge of the math behind the architecture” (Suresh et al., 2021) • “Stakeholder involved in an ex-ante impact assessment of the automatic decision system”(Henin and Le Métayer, 2021) |  • Ensure/improve product efficiency and debug (Barredo Arrieta et al., 2020) • Research new functionalities (Barredo Arrieta et al., 2020)
Auditing team |  Milieu, Formal + Instrumental |  • “Familiarity with broader ML-enabled systems” (Suresh et al., 2021) • “Experts who intervene wither upstream or downstream” (Henin and Le Métayer, 2021) |  • Verify model compliance with legislation (Barredo Arrieta et al., 2020)
Data domain experts |  Data domain, Formal + Instrumental |  • “Theories relevant to the data domain” (Suresh et al., 2021) • “Professional involved in the operational phase of the automatic decision system” (Henin and Le Métayer, 2021) |  • Gain scientific or domain-specific knowledge (Barredo Arrieta et al., 2020; Suresh et al., 2021) • Trust the model (Barredo Arrieta et al., 2020; Suresh et al., 2021) • Act based on the output (Suresh et al., 2021)
Decision subjects |  Data domain + Milieu, Personal |  • “Lived experience and cultural knowledge” (Suresh et al., 2021) • “Layperson affected by the outcomes of the automatic decision system” (Henin and Le Métayer, 2021) |  • Understand their situation (Barredo Arrieta et al., 2020) • Verify fair decision (Barredo Arrieta et al., 2020) • Contest decision (Suresh et al., 2021) • Understand how one’s data is being used (Suresh et al., 2021)

##### Mapping stakeholders in our example use case

Going back to our example, once explainability and privacy have been broken
down into specific criteria manifestations, the team needs to map the
stakeholders who will take part in the assessment process (fig 2). The
development team represents the stakeholders who have the knowledge of the
math behind the system. An external auditing team will join the discussion to
make sure that the model is aligned with current legislation. Since the
algorithmic life insurance application system deals with medical data, the
data domain experts will be represented by a medical team and a life insurance
expert. Decision subjects will be laypeople who seek to understand and verify
their situation with regards to data usage and the decision reached by the
system.

###  4.3. Mapping tailored communication means

We then examine each of the reviewed means and identify their typology, the
value manifestations that they cover and the stakeholders that can make use of
it, as illustrated in table LABEL:tab:exampleprivacymeans for privacy
dashboards. The objective of mapping value manifestations, stakeholder
profiles and communication means is that of enabling a fruitful and informed
discussion among stakeholders. We classify these means in three categories:
(1)  Descriptive documents (red), (2) Design strategies (blue), and (3) Ready-
to-use tools (green). Appendix A summarizes the rest of the communication
means we identified and maps them to value manifestations and stakeholders for
whom such methods are suitable.

The stakeholders assigned to a specific communication means are based on the
audience addressed by the original authors of such methodologies. In some
cases, the characterization of the intended audience was not as granular as
our stakeholder mapping and the authors merely differed experts in ML from
non-experts. Based on the nature of knowledge that we assigned to each of the
mentioned stakeholders in section 4.1, we considered that the development and
auditing teams are able to understand technically formulated system details
(experts) whereas data domain experts and decision subjects would require more
accessible communication means (non-experts). Similarly, some of the
communication means identified for explainability are suitable for any
stakeholder, but the original authors formulated the post-hoc explanations
with varying degrees of complexity, which should be taken into account when
trying to deploy such strategies. If the target audience are data subjects, we
echo van Berkel et al. (van Berkel et al., 2021) and Cheng et al. (Cheng et
al., 2019) and recommend to limit presentation complexity and to instruct
participants throughout the session.

Table 13. Illustration of how we mapped communication means with values, manifestations and stakeholders (DT = Development Team; AT = Auditing Team; DE = Data Domain Experts; DS = Decision Subjects). Privacy dashboards are tools (green) that allow users to interactively assess the collection and usage of their data. The rest of the reviewed communication means are characterized in appendix A. |  Means |  Value |  Manifestation(s) | Stakeholder |  Application (model) |  Approach |  Visual elements
---|---|---|---|---|---|---|---
|  DT |  AT |  DE |  DS
[B] |  Privacy dashboards (Zimmermann et al., 2014; Earp and Staddon, 2016; Fischer-Hübner et al., 2016; Herder and van Maaren, 2020; Farke et al., 2021) |  Privacy |  • Description of what and why data is collected • Description of how data is handled |  |  |  |  ✓ |  Agnostic |  |  • Timelines • Bar charts • Maps • Network graphs
Human agency |  • Self-assessment of the system |  |  |  |
Trans-parency |  • Disclosure of properties of data |  |  |  |

It should be noted that this mapping process represents a first step to making
a wide range of stakeholders with different backgrounds understand each other.
We are aware that communicating system-related information in a tailored way
does not directly lead to the resolution of value trade-offs, and that design
strategies are necessary for facilitating such conversations (Heijne and van
der Meer, 2019). In any case, the exercise of resolving value tensions should
be a communicative process, rather than a simple explanation (O’Hara, 2020).
However, the means used for communicating specifications of the system will
play a key role in the dynamics that will take place in those sessions.

##### Assigning communication means to each stakeholder in our example use
case.

The life insurance researchers are now looking into appropriate methods for
communicating values to different stakeholders (fig 2), so that they can
develop a comprehensive plan that ensures both compliance and communication of
values.

Based on the mapping presented in appendix A, tables
LABEL:tab:tailoredmeanscodes and LABEL:tab:detailedmeans, for the value of
explainability and its manifestations in the form of post-hoc explanations,
the team can use various design strategies and tools as part of their
assessment process. To facilitate the navigation of table
LABEL:tab:detailedmeans, they first examine table LABEL:tab:tailoredmeanscodes
to locate the type of means (tool, strategy, or documentation), values and
stakeholders they are interested in. Once they select the codes associated to
each communication means, they check table LABEL:tab:detailedmeans to see
whether the value manifestations in question are addressed and to explore the
selected communication means. If the team working on the life insurance case
prefers a ready-to-use tool over the description of design strategies for
assessing explainability, they can use InterpretML (Nori et al., 2019) and
especially the DiCE (Mothilal et al., 2019) functionality, (code [AC]) with
the development and auditing teams to evaluate counterfactual examples. These
counterfactual examples tell how input features should change in order for the
output of the system to be different. That is to say, how the individual
applying for life insurance should be different, physically, or when it comes
to insurance or medical history, for them to accept the application (if the
original output was a refusal). However, this tool might not be suitable for
non-experts who are not familiar with ML-related concepts. In the life
insurance use case, the medical and insurance team and the decision subjects
should receive a description of how the output changes if a feature is
perturbed, absent or present adapted to their insight needs. This can be done
by describing the answers to the questions “Why, Why not and How to be that”
for a certain output (Liao et al., 2020) (code [P]). As for privacy
manifestations, the development and auditing teams can examine data collection
and storage specifications through the Datasheet (Gebru et al., 2020)
associated to the dataset in question (code [K]). Special attention should be
paid to the “Collection” and “Preprocessing/cleaning/labelling” sections. For
decision subjects, iconsets (European Commission, 2018; Rossi and Palmirani,
2017; Holtz et al., 2011; Mehldau, 2007) (code [A]) and privacy dashboards
(Zimmermann et al., 2014; Earp and Staddon, 2016; Fischer-Hübner et al., 2016;
Herder and van Maaren, 2020; Farke et al., 2021) (code [B]) are means for them
to explore how their data is being used. It should be noted that the cell that
intersects between data domain experts and privacy in table
LABEL:tab:tailoredmeanscodes is blank. Based on the characterization of
stakeholders that we provided, data privacy-related matters are not directly
linked to the purpose that data domain experts show when willing to explore
algorithmic systems. This is translated into scarcity of methodologies related
to privacy manifestations that directly address data domain experts.

##  5\. Discussion and future work

We discuss important aspects of our framework below.

Design choices for creating a value-based framework. We aim at examining
values that characterize ML-driven systems rather than the organizations
responsible for these systems. Hence, we did not integrate accountability or
responsibility as a value per se in our framework. We are aware that
algorithms cannot be held responsible for the potential harm that they might
cause (Bryson et al., 2017; Hidalgo et al., 2021), and that in order to
effectively deploy such systems, there is an urgent call for accountability
(Yazdanpanah et al., 2021; Alfrink et al., 2020). Likewise, we are aware of
the need for rigorous frameworks that support accountability (Hutchinson et
al., 2021) and we consider that the act of conceiving an assessment framework
itself answers to the need to evaluate and audit algorithmic systems (Fjeld et
al., 2020). Nevertheless, we did not explicitly highlight the profiles of the
people accountable for the system. We decided to follow Zhu et al. (Zhu et
al., 2021) and considered accountability as a governance issue. We do,
however, believe that entities up the chain of command should be held
accountable for the potential harm caused by algorithmic systems (Lyons et
al., 2021; Hidalgo et al., 2021; Schaerer et al., 2009; AI Ethics Impact Group
(AIEIG), 2020; Fjeld et al., 2020). It should also be noted that values and
criteria presented in this paper might not be unique (Shklovski and Némethy,
2022). We acknowledge current discussions in VSD about the shortcomings of
pre-selecting values (Davis and Nathan, 2015) and, hence, do not claim
universality. Extension and modification of values is possible in our
framework, but are subject to respecting continuity and opposition between
values. Similarly, criteria and manifestations can be extended and subsets
could be included to create situationally-specific versions of the framework.
Since the aim of our framework is to encourage critical reflection
(Frauenberger et al., 2016; Ustek-Spilda et al., 2019) and we identified some
value manifestations that require additional communication means, we
particularly encourage those context-specific adaptations to happen. Under no
circumstances should the scarcity of communication means for certain values
identified in our framework represent an excuse to justify inaction or to
ignore such values.

Context dependence and consistency. As echoed by Liscio et al. (Liscio et al.,
2021), in order to translate values into system requirements (Pommeranz et
al., 2012; van de Poel, 2013), to reason about conflicting values (Ajmeri et
al., 2020; Murukannaiah and Singh, 2014) and to communicate them to different
stakeholders (Fred van Raaij and Verhallen, 1994), it is necessary to situate
these values within a context. The prioritization of values depends on the
application context of such systems (AI Ethics Impact Group (AIEIG), 2020). In
this paper, we showed an example of how the framework could be applied to a
particular use case. However, considering the differences between value
alignments and tensions that may arise due to context dependence, the validity
and consistency of our framework is still to be tested. Future work needs to
validate our framework across scenarios (van Berkel et al., 2021; Ghassemi et
al., 2021) through user studies or synthetic experiments (Sokol and Flach,
2020).

Need for standardization. To systematically review and revisit value
priorities and tensions among different stakeholders, our framework should be
part of a broader evaluation workflow (Lee and Singh, 2021), such as the one
suggested by Raji and Smart (Raji et al., 2020). Besides, practices from
software engineering such as the Values Dashboard (Nurwidyantoro et al., 2021)
could be adopted (Shneiderman, 2020; Thew and Sutcliffe, 2018). This dashboard
promotes awareness of values and aims at triggering discussions among
stakeholders. It claims to be beneficial in each phase of the software
development process, from inception to release, and establishes strategies,
such as Timelines or Issues, that are already common practice on software
development platforms like Github.

Implications of our work. Our multi-stakeholder value-based framework
facilitates the unveiling of assumptions that encode political and social
values made by developers (Raji et al., 2020). By bringing together a wide
range of stakeholders to evaluate and discuss value manifestations, one can
anticipate and remedy harmful algorithmic behaviors before deploying a system.
Besides, we provide researchers and industry practitioners with a good
coverage of values to evaluate their systems and the association of such
values to actionable value manifestations. This contributes greatly to the
adoption of ethical approaches by practically-minded people (Morley et al.,
2020). For researchers, we provide them with an easy-to-navigate mapping of
value manifestations, stakeholders and communication means. Our framework also
visually illustrates research gaps that need to be addressed. Blank spaces in
appendix A or values with a scarce number of associated communication means
directly refer to valuable research opportunities. For instance, for the value
of fairness, a great deal of effort has been devoted to designing ready-to-use
tools for stakeholders with a deep understanding of ML (developers and
auditing teams). However, means for addressing fairness manifestations and
communicating them to decision subjects have not received the same attention.
For industry practitioners, we gathered ready-to-use open source toolkits
(appendix A) that can be directly applied to their own use cases. Moreover,
since we host this mapping on an online repository 313131Check footnote 30
open to future contributions, we hope that the number of tools addressing each
of the identified value manifestations will grow and that the benefits of
designing such a framework will be even more tangible in the future.

##  6\. Conclusions

In this paper, we designed a value-based framework for assessing algorithmic
systems from a multi-stakeholder perspective. This provides investigators of
algorithmic systems with an actionable set of criteria manifestations to
operationalize high-level ethical principles. We arranged eleven prominent
values of ML-driven systems in a circular composition, so that common
motivations and trade-offs can be easily identified.

We then broke down each of these values into a set of criteria and their
correspondent manifestations in the form of quantifiable indicators, process-
oriented practices, and signifiers. In addition, we examined available tools
for communicating those value manifestations to different stakeholders based
on the nature of their knowledge and their insight needs. This should enable
to bring a wide range of stakeholders together to systematically assess values
encoded in a system and facilitate value- and ethics-related discussions among
them. This work completes previous studies that claim the need for
incorporating a diverse range of stakeholders and viewpoints in the ML
workflow, so that conflicting priorities and value tensions can be reviewed,
negotiated and consolidated.

###### Acknowledgements.

We thank Kars Alfrink for valuable feedback on an earlier version of this
paper and our colleagues at the DCODE Network for insightful discussions. This
project has received funding from the European Union’s Horizon 2020 research
and innovation programme under the Marie Skłodowska‐Curie grant agreement No
955990.

## References

  * (1)
  * Abebe et al. (2020) Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G. Robinson. 2020.  Roles for computing in social change. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. ACM, New York, NY, USA, 252–260.  https://doi.org/10.1145/3351095.3372871
  * Adler et al. (2018) Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian. 2018.  Auditing black-box models for indirect influence.  _Knowledge and Information Systems_ 54, 1 (2018), 95–122.  https://doi.org/10.1007/s10115-017-1116-3
  * AI Ethics Impact Group (AIEIG) (2020) AI Ethics Impact Group (AIEIG). 2020\.  From Principles to Practice An interdisciplinary framework to operationalise AI ethics.  https://www.ai-ethics-impact.org/resource/blob/1961130/c6db9894ee73aefa489d6249f5ee2b9f/aieig---report---download-hb-data.pdf
  * Aizenberg and van den Hoven (2020) Evgeni Aizenberg and Jeroen van den Hoven. 2020.  Designing for human rights in AI.  _Big Data & Society_ 7, 2 (7 2020), 2053951720949566\.  https://doi.org/10.1177/2053951720949566
  * Ajmeri et al. (2020) Nirav Ajmeri, Hui Guo, Pradeep K Murukannaiah, and Munindar P Singh. 2020.  Elessar: Ethics in Norm-Aware Agents. In _Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems_ _(AAMAS ’20)_. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 16–24.
  * Alfrink et al. (2020) Kars Alfrink, T. Turel, A. I. Keller, N. Doorn, and G. W. Kortuem. 2020.  Contestable City Algorithms. International Conference on Machine Learning Workshop.
  * Alqaraawi et al. (2020) Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. 2020\.  Evaluating saliency map explanations for convolutional neural networks. In _Proceedings of the 25th International Conference on Intelligent User Interfaces_. ACM, New York, NY, USA.  https://doi.org/10.1145/3377325.3377519
  * Alshaabi et al. (2021) Thayer Alshaabi, David Rushing Dewhurst, Joshua R Minot, Michael V Arnold, Jane L Adams, Christopher M Danforth, and Peter Sheridan Dodds. 2021.  The growing amplification of social media: measuring temporal and social contagion dynamics for over 150 languages on Twitter for 2009–2020.  _EPJ Data Science_ 10, 1 (2021), 15.  https://doi.org/10.1140/epjds/s13688-021-00271-0
  * Amershi et al. (2015) Saleema Amershi, Max Chickering, Steven M. Drucker, Bongshin Lee, Patrice Simard, and Jina Suh. 2015\.  ModelTracker. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA.  https://doi.org/10.1145/2702123.2702509
  * Amini et al. (2019) Alexander Amini, Ava P Soleimany, Wilko Schwarting, Sangeeta N Bhatia, and Daniela Rus. 2019\.  Uncovering and Mitigating Algorithmic Bias through Learned Latent Structure. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ _(AIES ’19)_. Association for Computing Machinery, New York, NY, USA, 289–295.  https://doi.org/10.1145/3306618.3314243
  * Amnesty International (2018) Access Now Amnesty International. 2018\.  Toronto Declaration: Protecting the Right to Equality and Non-Discrimination in Machine Learning Systems.  https://www.accessnow.org/cms/assets/uploads/2018/08/The-Toronto-Declaration_ENG_08-2018.pdf
  * Anik and Bunt (2021) Ariful Islam Anik and Andrea Bunt. 2021.  Data-Centric Explanations: Explaining Training Data of Machine Learning Systems to Promote Transparency. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA.  https://doi.org/10.1145/3411764.3445736
  * Arnstein (2019) Sherry R Arnstein. 2019\.  A Ladder of Citizen Participation.  _Journal of the American Planning Association_ 85, 1 (1 2019), 24–34.  https://doi.org/10.1080/01944363.2018.1559388
  * Assran et al. (2019) Mahmoud Assran, Joshua Romoff, Nicolas Ballas, Joelle Pineau, and Michael Rabbat. 2019.  Gossip-based Actor-Learner Architectures for Deep Reinforcement Learning.  (6 2019).
  * Bakalar et al. (2021) Chloé Bakalar, Renata Barreto, Stevie Bergman, Miranda Bogen, Bobbie Chern, Sam Corbett-Davies, Melissa Hall, Isabel Kloumann, Michelle Lam, Joaquin Quiñonero Candela, Manish Raghavan, Joshua Simons, Jonathan Tannen, Edmund Tong, Kate Vredenburgh, and Jiejing Zhao. 2021\.  Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems.  (3 2021).
  * Balayn and Gürses (2021) Agathe Balayn and Seda Gürses. 2021.  Beyond Debiasing: Regulating AI and its inequalities.  https://edri.org/our-work/if-ai-is-the-problem-is-debiasing-the-solution/
  * Ball-Burack et al. (2021) Ari Ball-Burack, Michelle Seng Ah Lee, Jennifer Cobbe, and Jatinder Singh. 2021.  Differential Tweetment: Mitigating Racial Dialect Bias in Harmful Tweet Detection. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 116–128.  https://doi.org/10.1145/3442188.3445875
  * Barredo Arrieta et al. (2020) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. 2020.  Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  _Information Fusion_ 58 (6 2020), 82–115.  https://doi.org/10.1016/J.INFFUS.2019.12.012
  * Bellamy et al. (2019) R K E Bellamy, K Dey, M Hind, S C Hoffman, S Houde, K Kannan, P Lohia, J Martino, S Mehta, A Mojsilović, S Nagar, K Natesan Ramamurthy, J Richards, D Saha, P Sattigeri, M Singh, K R Varshney, and Y Zhang. 2019\.  AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias.  _IBM Journal of Research and Development_ 63, 4/5 (2019), 1–4.  https://doi.org/10.1147/JRD.2019.2942287
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science.  _Transactions of the Association for Computational Linguistics_ 6 (12 2018).  https://doi.org/10.1162/tacl_a_00041
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 610–623.  https://doi.org/10.1145/3442188.3445922
  * Berk et al. (2017) Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. 2017.  Fairness in Criminal Justice Risk Assessments: The State of the Art.  (3 2017).
  * Biggio and Roli (2018) Battista Biggio and Fabio Roli. 2018.  Wild patterns: Ten years after the rise of adversarial machine learning.  _Pattern Recognition_ 84 (12 2018), 317–331.  https://doi.org/10.1016/j.patcog.2018.07.023
  * Binns et al. (2018) Reuben Binns, Max Van Kleek, Michael Veale, Ulrik Lyngs, Jun Zhao, and Nigel Shadbolt. 2018\.  ’It’s Reducing a Human Being to a Percentage’; Perceptions of Justice in Algorithmic Decisions.  (1 2018).  https://doi.org/10.1145/3173574.3173951
  * Bird et al. (2020) Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. 2020.  _Fairlearn: A toolkit for assessing and improving fairness in AI_.  Technical Report MSR-TR-2020-32. Microsoft.  https://www.microsoft.com/en-us/research/publication/fairlearn-a-toolkit-for-assessing-and-improving-fairness-in-ai/
  * Birhane et al. (2021) Abeba Birhane, Pratyusha Kalluri, Dallas Card, William Agnew, Ravit Dotan, and Michelle Bao. 2021\.  The Values Encoded in Machine Learning Research.  (6 2021).
  * Blazevic et al. (2021) Alice Namuli Blazevic, Patrick Mugalula, and Andrew Wandera. 2021\.  Towards Operationalizing the Data Protection and Privacy Act 2020: Understanding the Draft Data Protection and Privacy Regulations, 2020.  _SSRN Electronic Journal_ (2021).  https://doi.org/10.2139/ssrn.3776353
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé, and Hanna Wallach. 2020.  Language (Technology) is Power: A Critical Survey of ”Bias” in NLP.  (5 2020).
  * Bountouridis et al. (2019) Dimitrios Bountouridis, Jaron Harambam, Mykola Makhortykh, Mónica Marrero, Nava Tintarev, and Claudia Hauff. 2019.  SIREN: A Simulation Framework for Understanding the Effects of Recommender Systems in Online News Environments. In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ _(FAT* ’19)_. Association for Computing Machinery, New York, NY, USA, 150–159.  https://doi.org/10.1145/3287560.3287583
  * Bryson et al. (2017) Joanna J. Bryson, Mihailis E. Diamantis, and Thomas D. Grant. 2017.  Of, for, and by the people: the legal lacuna of synthetic persons.  _Artificial Intelligence and Law_ 25, 3 (9 2017), 273–291.  https://doi.org/10.1007/s10506-017-9214-9
  * Buolamwini and Gebru (2018) Joy Buolamwini and Timnit Gebru. 2018.  Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ _(Proceedings of Machine Learning Research, Vol. 81)_ , Sorelle A Friedler and Christo Wilson (Eds.). PMLR, 77–91.  https://proceedings.mlr.press/v81/buolamwini18a.html
  * Cai et al. (2019) Carrie J. Cai, Jonas Jongejan, and Jess Holbrook. 2019\.  The effects of example-based explanations in a machine learning interface. In _Proceedings of the 24th International Conference on Intelligent User Interfaces_. ACM, New York, NY, USA.  https://doi.org/10.1145/3301275.3302289
  * Calvert et al. (2019) Simeon C Calvert, Daniël D Heikoop, Giulio Mecacci, and Bart Van Arem. 2019.  A human centric framework for the analysis of automated driving systems based on meaningful human control.  _Theoretical Issues in Ergonomics Science_ 21, 4 (2019), 478–506.  https://doi.org/10.1080/1463922X.2019.1697390
  * Chasalow and Levy (2021) Kyla Chasalow and Karen Levy. 2021.  Representativeness in Statistics, Politics, and Machine Learning. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 77–89.  https://doi.org/10.1145/3442188.3445872
  * Cheng et al. (2019) Hao-Fei Cheng, Ruotong Wang, Zheng Zhang, Fiona O’Connell, Terrance Gray, F Maxwell Harper, and Haiyi Zhu. 2019.  Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi.org/10.1145/3290605.3300789
  * China Electronics Standardization Institute (2018) China Electronics Standardization Institute. 2018.  Original CSET Translation of ”Artificial Intelligence Standardization White Paper”.  https://cset.georgetown.edu/research/artificial-intelligence-standardization-white-paper/
  * Chouldechova (2016) Alexandra Chouldechova. 2016\.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  (10 2016).
  * Cila et al. (2020) Nazli Cila, Gabriele Ferri, Martijn de Waal, Inte Gloerich, and Tara Karpinski. 2020.  The Blockchain and the Commons: Dilemmas in the Design of Local Platforms. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–14.  https://doi-org.tudelft.idm.oclc.org/10.1145/3313831.3376660
  * Crawford and Paglen (2019) Kate Crawford and Trevor Paglen. 2019.  Excavating AI: The Politics of Training Sets for Machine Learning.
  * Cretu et al. (2008) Gabriela F. Cretu, Angelos Stavrou, Michael E. Locasto, Salvatore J. Stolfo, and Angelos D. Keromytis. 2008\.  Casting out Demons: Sanitizing Training Data for Anomaly Sensors. In _2008 IEEE Symposium on Security and Privacy (sp 2008)_. IEEE, 81–95.  https://doi.org/10.1109/SP.2008.11
  * Dalton et al. (2019) Steven Dalton, Iuri Frosio, and Michael Garland. 2019\.  Accelerating Reinforcement Learning through GPU Atari Emulation.  (7 2019).
  * D’Amour et al. (2020) Alexander D’Amour, Hansa Srinivasan, James Atwood, Pallavi Baljekar, D Sculley, and Yoni Halpern. 2020.  Fairness is Not Static: Deeper Understanding of Long Term Fairness via Simulation Studies. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 525–534.  https://doi.org/10.1145/3351095.3372878
  * Dasha Simons (2019) Dasha Simons. 2019\.  _Design for fairness in AI: Cooking a fair AI Dish_.  Technical Report. Delft University of Technology. Graduation project. MSc in Strategic Product Design.  http://resolver.tudelft.nl/uuid:5a116c17-ce0a-4236-b283-da6b8545628c
  * Davidson et al. (2019) Thomas Davidson, Debasmita Bhattacharya, and Ingmar Weber. 2019\.  Racial Bias in Hate Speech and Abusive Language Detection Datasets. In _Proceedings of the Third Workshop on Abusive Language Online_. Association for Computational Linguistics, Stroudsburg, PA, USA.  https://doi.org/10.18653/v1/W19-3504
  * Davis and Nathan (2015) Janet Davis and Lisa P. Nathan. 2015.  Value Sensitive Design: Applications, Adaptations, and Critiques.  In _Handbook of Ethics, Values, and Technological Design_. Springer Netherlands, Dordrecht, 11–40.  https://doi.org/10.1007/978-94-007-6970-0_3
  * Denton et al. (2020) Emily Denton, Alex Hanna, Razvan Amironesei, Andrew Smart, Hilary Nicole, and Morgan Klaus Scheuerman. 2020.  Bringing the People Back In: Contesting Benchmark Machine Learning Datasets.  (7 2020).  https://arxiv.org/abs/2007.07399
  * Dhurandhar et al. (2018) Amit Dhurandhar, Pin-Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. 2018.  Explanations based on the Missing: Towards Contrastive Explanations with Pertinent Negatives.  (2 2018).
  * Dixon et al. (2018) Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018.  Measuring and Mitigating Unintended Bias in Text Classification. In _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_. ACM, New York, NY, USA, 67–73.  https://doi.org/10.1145/3278721.3278729
  * Dodge et al. (2019) Jonathan Dodge, Q. Vera Liao, Yunfeng Zhang, Rachel K. E. Bellamy, and Casey Dugan. 2019\.  Explaining Models: An Empirical Study of How Explanations Impact Fairness Judgment.  (1 2019).  https://doi.org/10.1145/3301275.3302310
  * Dotan and Milli (2019) Ravit Dotan and Smitha Milli. 2019.  Value-laden Disciplinary Shifts in Machine Learning.  (12 2019).
  * Draws et al. (2021) Tim Draws, Alisa Rieger, Oana Inel, Ujwal Gadiraju, and Nava Tintarev. 2021.  A Checklist to Combat Cognitive Biases in Crowdsourcing.  _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ 9, 1 (10 2021), 48–59.  https://ojs.aaai.org/index.php/HCOMP/article/view/18939
  * Dwork et al. (2011) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Rich Zemel. 2011.  Fairness Through Awareness.  (4 2011).
  * Dwork et al. (2006) Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006.  Calibrating Noise to Sensitivity in Private Data Analysis.  265–284.  https://doi.org/10.1007/11681878_14
  * Earp and Staddon (2016) Julia Earp and Jessica Staddon. 2016.  ”I had no idea this was a thing”. In _Proceedings of the 6th Workshop on Socio-Technical Aspects in Security and Trust_. ACM, New York, NY, USA, 79–86.  https://doi.org/10.1145/3046055.3046062
  * Edizel et al. (2020) Bora Edizel, Francesco Bonchi, Sara Hajian, André Panisson, and Tamir Tassa. 2020.  FaiRecSys: mitigating algorithmic bias in recommender systems.  _International Journal of Data Science and Analytics_ 9, 2 (2020), 197–213.  https://doi.org/10.1007/s41060-019-00181-5
  * European Commission (2018) European Commission. 2018\.  2018 reform of EU data protection rules.  https://ec.europa.eu/commission/sites/beta-political/files/data-protection-factsheet-changes_en.pdf
  * European Commission (2019) European Commission. 2019\.  Ethics guidelines for trustworthy AI.  https://www.aepd.es/sites/default/files/2019-12/ai-ethics-guidelines.pdf
  * Farke et al. (2021) Florian M. Farke, David G. Balash, Maximilian Golla, Markus Dürmuth, and Adam J. Aviv. 2021\.  Are Privacy Dashboards Good for End Users? Evaluating User Perceptions and Reactions to Google’s My Activity (Extended Version).  (5 2021).
  * Fischer-Hübner et al. (2016) Simone Fischer-Hübner, Julio Angulo, Farzaneh Karegar, and Tobias Pulls. 2016.  Transparency, Privacy and Trust – Technology for Tracking and Controlling My Data Disclosures: Does This Work?  3–14.  https://doi.org/10.1007/978-3-319-41354-9_1
  * Fjeld et al. (2020) Jessica Fjeld, Nele Achten, Hannah Hilligoss, Adam Nagy, and Madhulika Srikumar. 2020.  Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-Based Approaches to Principles for AI.  _SSRN Electronic Journal_ (2020).  https://doi.org/10.2139/ssrn.3518482
  * Floridi (2019) Luciano Floridi. 2019\.  Translating Principles into Practices of Digital Ethics: Five Risks of Being Unethical.  _Philosophy & Technology_ 32, 2 (6 2019).  https://doi.org/10.1007/s13347-019-00354-x
  * Floridi et al. (2018) Luciano Floridi, Josh Cowls, Monica Beltrametti, Raja Chatila, Patrice Chazerand, Virginia Dignum, Christoph Luetge, Robert Madelin, Ugo Pagallo, Francesca Rossi, Burkhard Schafer, Peggy Valcke, and Effy Vayena. 2018.  AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations.  _Minds and Machines_ 28, 4 (12 2018).  https://doi.org/10.1007/s11023-018-9482-5
  * Frauenberger et al. (2016) Christopher Frauenberger, Marjo Rauhala, and Geraldine Fitzpatrick. 2016.  In-Action Ethics: Table 1.  _Interacting with Computers_ (6 2016).  https://doi.org/10.1093/iwc/iww024
  * Fred van Raaij and Verhallen (1994) W. Fred van Raaij and Theo M.M. Verhallen. 1994.  Domain‐specific Market Segmentation.  _European Journal of Marketing_ 28, 10 (10 1994), 49–66.  https://doi.org/10.1108/03090569410075786
  * Friedman et al. (2017) Batya Friedman, David G. Hendry, and Alan Borning. 2017\.  A Survey of Value Sensitive Design Methods.  _Foundations and Trends® in Human–Computer Interaction_ 11, 2 (2017), 63–125.  https://doi.org/10.1561/1100000015
  * Gaillard (2016) Georges Gaillard. 2016\.  La conflictualité : une modalité de lien où s’arrime la destructivité humaine.  _Connexions_ 106, 2 (2016), 71.  https://doi.org/10.3917/cnx.106.0071
  * Gao et al. (2020) Yanjie Gao, Yu Liu, Hongyu Zhang, Zhengxian Li, Yonghao Zhu, Haoxiang Lin, and Mao Yang. 2020.  Estimating GPU memory consumption of deep learning models. In _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_. ACM, New York, NY, USA, 1342–1352.  https://doi.org/10.1145/3368089.3417050
  * García-Martín et al. (2019) Eva García-Martín, Crefeda Faviola Rodrigues, Graham Riley, and Håkan Grahn. 2019.  Estimation of energy consumption in machine learning.  _J. Parallel and Distrib. Comput._ 134 (12 2019), 75–88.  https://doi.org/10.1016/j.jpdc.2019.07.007
  * Gebru et al. (2020) Timnit Gebru, Google Jamie Morgenstern, Briana Vecchione, and Jennifer Wortman Vaughan. 2020.  Datasheets for Datasets.
  * Geiger et al. (2020) R Stuart Geiger, Kevin Yu, Yanlai Yang, Mindy Dai, Jie Qiu, Rebekah Tang, and Jenny Huang. 2020.  Garbage in, Garbage out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 325–336.  https://doi.org/10.1145/3351095.3372862
  * Ghai et al. (2020) Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, and Klaus Mueller. 2020.  Measuring Social Biases of Crowd Workers using Counterfactual Queries.  (4 2020).
  * Ghassemi et al. (2021) Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L Beam. 2021\.  The false hope of current approaches to explainable artificial intelligence in health care.  _The Lancet Digital Health_ 3, 11 (11 2021), e745–e750.  https://doi.org/10.1016/S2589-7500(21)00208-9
  * Globerson and Roweis (2006) Amir Globerson and Sam Roweis. 2006.  Nightmare at test time. In _Proceedings of the 23rd international conference on Machine learning - ICML ’06_. ACM Press, New York, New York, USA, 353–360.  https://doi.org/10.1145/1143844.1143889
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014.  Explaining and Harnessing Adversarial Examples.  (12 2014).
  * Google (2018) Google. 2018.  AI at Google: Our Principles.  https://www.blog.google/technology/ai/ai-principles/
  * Green and Hu (2018) Ben Green and Lily Hu. 2018\.  The Myth in the Methodology: Towards a Recontextualization of Fairness in Machine Learning. In _Machine Learning: The Debates workshop at the 35th International Conference on Machine Learning (ICML)_. Stockholm, Sweden.
  * Grgic-Hlaca et al. (2018) Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2018.  Beyond Distributive Fairness in Algorithmic Decision Making: Feature Selection for Procedurally Fair Learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_. Vol. 32.
  * Groves (2015) Christopher Groves. 2015\.  Logic of Choice or Logic of Care? Uncertainty, Technological Mediation and Responsible Innovation.  _NanoEthics_ 9, 3 (12 2015), 321–333.  https://doi.org/10.1007/s11569-015-0238-x
  * Hardt et al. (2016) Moritz Hardt, Eric Price, and Nathan Srebro. 2016.  Equality of Opportunity in Supervised Learning. In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ _(NIPS’16)_. Curran Associates Inc., Red Hook, NY, USA, 3323–3331.
  * Harrison et al. (2020) Galen Harrison, Julia Hanson, Christine Jacinto, Julio Ramirez, and Blase Ur. 2020.  An empirical study on the perceived fairness of realistic, imperfect machine learning models. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. ACM, New York, NY, USA, 392–402.  https://doi.org/10.1145/3351095.3372831
  * Heijne and van der Meer (2019) Katrina Heijne and Han van der Meer. 2019.  _Road Map for Creative Problem Solving Techniques Organizing and facilitating group sessions_.  Boom Uitgevers Amsterdam.
  * Hemment et al. (2019) Drew Hemment, Ruth Aylett, Vaishak Belle, Dave Murray-Rust, Ewa Luger, Jane Hillston, Michael Rovatsos, and Frank Broz. 2019\.  Experiential AI.  _AI Matters_ 5, 1 (4 2019), 25–31.  https://doi.org/10.1145/3320254.3320264
  * Henderson et al. (2020) Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020\.  Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning.  (1 2020).
  * Henin and Le Métayer (2021) Clément Henin and Daniel Le Métayer. 2021.  Beyond explainability: justifiability and contestability of algorithmic decision systems.  _AI & SOCIETY_ (7 2021).  https://doi.org/10.1007/s00146-021-01251-8
  * Herder and van Maaren (2020) Eelco Herder and Olaf van Maaren. 2020.  Privacy Dashboards: The Impact of the Type of Personal Data and User Control on Trust and Perceived Risk. In _Adjunct Publication of the 28th ACM Conference on User Modeling, Adaptation and Personalization_. ACM, New York, NY, USA, 169–174.  https://doi.org/10.1145/3386392.3399557
  * Hidalgo et al. (2021) César Hidalgo, Diana Orghian, Jordi Albo-Canals, Filipa de Almeida, and Natalia Martin. 2021\.  _How Humans Judge Machines_.  MIT Press.  https://hal.archives-ouvertes.fr/hal-03058652
  * Hirsch et al. (2017) Tad Hirsch, Kritzia Merced, Shrikanth Narayanan, Zac E. Imel, and David C. Atkins. 2017\.  Designing Contestability. In _Proceedings of the 2017 Conference on Designing Interactive Systems_. ACM, New York, NY, USA.  https://doi.org/10.1145/3064663.3064703
  * Holland et al. (2018) Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018.  The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards.  (5 2018).
  * Holtz et al. (2011) Leif-Erik Holtz, Katharina Nocun, and Marit Hansen. 2011\.  Towards Displaying Privacy Information with Icons.  338–348.  https://doi.org/10.1007/978-3-642-20769-3_27
  * Hong et al. (2021) Matthew K. Hong, Adam Fourney, Derek DeBellis, and Saleema Amershi. 2021.  Planning for Natural Language Failures with the AI Playbook. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA, 1–11.  https://doi.org/10.1145/3411764.3445735
  * Hube et al. (2019) Christoph Hube, Besnik Fetahu, and Ujwal Gadiraju. 2019\.  Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–12.  https://doi-org.tudelft.idm.oclc.org/10.1145/3290605.3300637
  * Hutchinson et al. (2021) Ben Hutchinson, Andrew Smart, Alex Hanna, Emily Denton, Christina Greer, Oddur Kjartansson, Parker Barnes, and Margaret Mitchell. 2021\.  Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 560–575.  https://doi.org/10.1145/3442188.3445918
  * IBM (2019) IBM. 2019.  IBM Everyday Ethics for AI.  https://www.ibm.com/watson/assets/duo/pdf/everydayethics.pdf
  * IEEE (2008) IEEE. 2008.  IEEE Standard for Software Reviews and Audits.  _IEEE Std 1028-2008_ (2008), 1–53.  https://doi.org/10.1109/IEEESTD.2008.4601584
  * Ionescu et al. (2021) Stefania Ionescu, Anikó Hannák, and Kenneth Joseph. 2021\.  An Agent-Based Model to Evaluate Interventions on Online Dating Platforms to Decrease Racial Homogamy. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 412–423.  https://doi.org/10.1145/3442188.3445904
  * Japanese Cabinet Office and Innovation (2019) Technology Japanese Cabinet Office, Council for Science and Innovation. 2019\.  Social Principles of Human-Centric Artificial Intelligence.  https://www8.cao.go.jp/cstp/english/humancentricai.pdf
  * Jia et al. (2019) Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. 2019.  MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examples.  (9 2019).
  * Jin et al. (2021) Weina Jin, Jianyu Fan, Diane Gromala, Philippe Pasquier, and Ghassan Hamarneh. 2021.  EUCA: A Practical Prototyping Framework towards End-User-Centered Explainable Artificial Intelligence.  (2 2021).  https://arxiv.org/abs/2102.02437
  * Kaiser and Rauchfleisch (2020) Jonas Kaiser and Adrian Rauchfleisch. 2020.  Birds of a Feather Get Recommended Together: Algorithmic Homophily in YouTube’s Channel Recommendations in the United States and Germany.  _Social Media + Society_ 6, 4 (10 2020), 2056305120969914\.  https://doi.org/10.1177/2056305120969914
  * Kalluri (2020) Pratyusha Kalluri. 2020\.  Don’t ask if artificial intelligence is good or fair, ask how it shifts power.  _Nature_ 583, 7815 (2020).  https://doi.org/10.1038/d41586-020-02003-2
  * Kaya et al. (2020) Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2020\.  On the Effectiveness of Regularization Against Membership Inference Attacks.  (6 2020).
  * Kearns et al. (2018) Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018.  Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In _Proceedings of the 35th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer Dy and Andreas Krause (Eds.). PMLR, 2564–2572.  https://proceedings.mlr.press/v80/kearns18a.html
  * Kearns and Roth (2019) Michael Kearns and Aaron Roth. 2019.  _The Ethical Algorithm: The Science of Socially Aware Algorithm Design_.  Oxford University Press, Inc., USA.
  * Keyes et al. (2019) Os Keyes, Jevan Hutson, and Meredith Durbin. 2019.  A Mulching Proposal: Analysing and Improving an Algorithmic System for Turning the Elderly into High-Nutrient Slurry. In _Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems_ _(CHI EA ’19)_. Association for Computing Machinery, New York, NY, USA, 1–11.  https://doi.org/10.1145/3290607.3310433
  * Kleinberg et al. (2016) Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016\.  Inherent Trade-Offs in the Fair Determination of Risk Scores.  (9 2016).
  * Kluttz et al. (2018) Daniel Kluttz, Nitin Kohli, and Deirdre K. Mulligan. 2018\.  Contestability and Professionals: From Explanations to Engagement with Algorithmic Systems.  _SSRN Electronic Journal_ (2018).  https://doi.org/10.2139/ssrn.3311894
  * Krafft and Zweig (2019) TD Krafft and K Zweig. 2019\.  Transparenz und Nachvollziehbarkeit algorithmenbasierter Entscheidungsprozesse.  _Ein Regulierungsvorschlag_ (2019).
  * Kuijer and Giaccardi (2018) Lenneke Kuijer and Elisa Giaccardi. 2018.  Co-Performance: Conceptualizing the Role of Artificial Agency in the Design of Everyday Life. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi-org.tudelft.idm.oclc.org/10.1145/3173574.3173699
  * Kulynych et al. (2018) Bogdan Kulynych, Rebekah Overdorf, Carmela Troncoso, and Seda Gürses. 2018.  POTs: Protective Optimization Technologies.  (6 2018).  https://doi.org/10.1145/3351095.3372853
  * Kusner et al. (2017) Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017.  Counterfactual Fairness. In _Advances in Neural Information Processing Systems_ , I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett (Eds.), Vol. 30. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2017/file/a486cd07e4ac3d270571622f4f316ec5-Paper.pdf
  * Larsonneur (2021) Claire Larsonneur. 2021\.  Intelligence artificielle ET/OU diversité linguistique : les paradoxes du traitement automatique des langues.  http://www.hybrid.univ-paris8.fr/lodel/index.php?id=1542
  * Lee (1973) Douglass B. Lee. 1973\.  Requiem for Large-Scale Models.  _Journal of the American Institute of Planners_ 39, 3 (5 1973), 163–178.  https://doi.org/10.1080/01944367308977851
  * Lee and Baykal (2017) Min Kyung Lee and Su Baykal. 2017.  Algorithmic Mediation in Group Decisions: Fairness Perceptions of Algorithmically Mediated vs. Discussion-Based Social Division. In _Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing_ _(CSCW ’17)_. Association for Computing Machinery, New York, NY, USA, 1035–1048.  https://doi.org/10.1145/2998181.2998230
  * Lee et al. (2019) Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, and Ariel D Procaccia. 2019.  WeBuildAI: Participatory Framework for Algorithmic Governance.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW (11 2019).  https://doi.org/10.1145/3359283
  * Lee and Singh (2021) Michelle Seng Ah Lee and Jatinder Singh. 2021.  Risk Identification Questionnaire for Unintended Bias in Machine Learning Development Lifecycle.  _SSRN Electronic Journal_ (2021).  https://doi.org/10.2139/ssrn.3777093
  * leins et al. (2020) kobi leins, Jey Han Lau, and Timothy Baldwin. 2020.  Give Me Convenience and Give Her Death: Who Should Decide What Uses of NLP are Appropriate, and on What Basis?. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA.  https://doi.org/10.18653/v1/2020.acl-main.261
  * Liao et al. (2020) Q. Vera Liao, Daniel Gruen, and Sarah Miller. 2020\.  Questioning the AI: Informing Design Practices for Explainable AI User Experiences.  (1 2020).  https://doi.org/10.1145/3313831.3376590
  * Liao et al. (2021) Q. Vera Liao, Milena Pribić, Jaesik Han, Sarah Miller, and Daby Sow. 2021.  Question-Driven Design Process for Explainable AI User Experiences.  (4 2021).
  * Liscio et al. (2021) Enrico Liscio, Michiel van der Meer, Luciano C Siebert, Catholijn M Jonker, Niek Mouter, and Pradeep K Murukannaiah. 2021.  Axies: Identifying and Evaluating Context-Specific Values. In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 799–808.
  * Long and Magerko (2020) Duri Long and Brian Magerko. 2020.  What is AI Literacy? Competencies and Design Considerations. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_. Association for Computing Machinery, New York, NY, USA, 1–16.  https://doi-org.tudelft.idm.oclc.org/10.1145/3313831.3376727
  * Lyons et al. (2021) Henrietta Lyons, Eduardo Velloso, and Tim Miller. 2021\.  Conceptualising Contestability: Perspectives on Contesting Algorithmic Decisions.  (2 2021).  https://doi.org/10.1145/3449180
  * Lyu et al. (2015) Chunchuan Lyu, Kaizhu Huang, and Hai-Ning Liang. 2015\.  A Unified Gradient Regularization Family for Adversarial Examples. In _2015 IEEE International Conference on Data Mining_. IEEE, 301–309.  https://doi.org/10.1109/ICDM.2015.84
  * Mahendran (2021) N. Mahendran. 2021\.  Analysis of memory consumption by neural networks based on hyperparameters.  (10 2021).
  * Martin et al. (2020) Donald Martin, Jr Google Vinodkumar Prabhakaran Google Jill Kuhlberg, and Andrew S Smart Google William Isaac DeepMind. 2020.  Extending the Machine Learning Abstraction Boundary: A Complex Systems Approach to Incorporate Societal Context.  (2020).
  * Mehldau (2007) M. Mehldau. 2007\.  Iconset for data-privacy declarations v 0.1.  https://netzpolitik.org/wp-upload/data-privacy-icons-v01.pdf
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021\.  A Survey on Bias and Fairness in Machine Learning.  _ACM Comput. Surv._ 54, 6 (7 2021).  https://doi.org/10.1145/3457607
  * Microsoft (2018) Microsoft. 2018\.  AI Principles.  https://www.microsoft.com/en-us/ai/responsible-ai?activetab=pivot1%3aprimaryr6
  * Mishra and Rzeszotarski (2021) Swati Mishra and Jeffrey M Rzeszotarski. 2021.  Designing Interactive Transfer Learning Tools for ML Non-Experts. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. ACM, New York, NY, USA.  https://doi.org/10.1145/3411764.3445096
  * Mission assigned by the French Prime Minister (2019) Mission assigned by the French Prime Minister. 2019.  For a Meaningful Artificial Intelligence: Toward a French and European Strategy.  https://www.aiforhumanity.fr/pdfs/MissionVillani_Report_ENG-VF.pdf
  * Mitchell et al. (2018) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2018.  Model Cards for Model Reporting.  (10 2018).  https://doi.org/10.1145/3287560.3287596
  * Mitra (2021) Tanushree Mitra. 2021\.  Provocation: Contestability in Large-Scale Interactive {NLP} Systems. In _Proceedings of the First Workshop on Bridging Human{–}Computer Interaction and Natural Language Processing_. Association for Computational Linguistics, 96–100.
  * Mittelstadt (2019) Brent Mittelstadt. 2019\.  Principles alone cannot guarantee ethical AI.  _Nature Machine Intelligence_ 1, 11 (2019), 501–507.  https://doi.org/10.1038/s42256-019-0114-4
  * Morley et al. (2020) Jessica Morley, Luciano Floridi, Libby Kinsey, and Anat Elhalal. 2020.  From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices.  _Science and Engineering Ethics_ 26 (2020), 2141–2168.  https://doi.org/10.1007/s11948-019-00165-5
  * Mothilal et al. (2019) Ramaravind Kommiya Mothilal, Amit Sharma, and Chenhao Tan. 2019\.  Explaining Machine Learning Classifiers through Diverse Counterfactual Explanations.  (5 2019).  https://doi.org/10.1145/3351095.3372850
  * Murukannaiah and Singh (2014) Pradeep K Murukannaiah and Munindar P Singh. 2014.  Xipho: Extending Tropos to Engineer Context-Aware Personal Agents. In _Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems_ _(AAMAS ’14)_. International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 309–316.
  * Nasr et al. (2018) Milad Nasr, Reza Shokri, and Amir Houmansadr. 2018.  Machine Learning with Membership Privacy using Adversarial Regularization.  (7 2018).
  * Nori et al. (2019) Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. 2019\.  InterpretML: A Unified Framework for Machine Learning Interpretability.  (9 2019).
  * Nurwidyantoro et al. (2021) Arif Nurwidyantoro, Mojtaba Shahin, Michel Chaudron, Waqar Hussain, Harsha Perera, Rifat Ara Shams, and Jon Whittle. 2021\.  Towards a Human Values Dashboard for Software Development: An Exploratory Study.  (7 2021).
  * OECD (2019) OECD. 2019.  Recommendation of the Council on Artificial Intelligence.  https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0406
  * O’Hara (2020) Kieron O’Hara. 2020\.  Explainable AI and the philosophy and practice of explanation.  _Computer Law & Security Review_ 39 (11 2020), 105474\.  https://doi.org/10.1016/J.CLSR.2020.105474
  * Papernot et al. (2016) Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016.  Distillation as a Defense to Adversarial Perturbations Against Deep Neural Networks. In _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 582–597.  https://doi.org/10.1109/SP.2016.41
  * Parisi and Comunello (2020) Lorenza Parisi and Francesca Comunello. 2020.  Dating in the time of “relational filter bubbles”: exploring imaginaries, perceptions and tactics of Italian dating app users.  _The Communication Review_ 23, 1 (2020), 66–89.  https://doi.org/10.1080/10714421.2019.1704111
  * Patel (2021) Reema Patel. 2021\.  Reboot AI with human values.  _Nature_ 598, 7879 (10 2021).  https://doi.org/10.1038/d41586-021-02693-2
  * Paullada et al. (2020) Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton, and Alex Hanna. 2020\.  Data and its (dis)contents: A survey of dataset development and use in machine learning research.  (12 2020).  http://arxiv.org/abs/2012.05345
  * Pommeranz et al. (2012) Alina Pommeranz, Christian Detweiler, Pascal Wiggers, and Catholijn Jonker. 2012.  Elicitation of situated values: need for tools to help stakeholders and designers to reflect and communicate.  _Ethics and Information Technology_ 14, 4 (12 2012), 285–303.  https://doi.org/10.1007/s10676-011-9282-6
  * Raghavan et al. (2020) Manish Raghavan, Solon Barocas, Jon Kleinberg, and Karen Levy. 2020.  Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 469–481.  https://doi.org/10.1145/3351095.3372828
  * Raji et al. (2020) Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. 2020\.  Closing the AI accountability gap. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. ACM, New York, NY, USA, 33–44.  https://doi.org/10.1145/3351095.3372873