[1]Mirthe Dankloff

[1]Emma Beauxis-Aussalet

[1]Computer science, Vrije Universiteit Amsterdam, De Boelelaan 1105,
Amsterdam, 1081 HV Amsterdam, The Netherlands

2]Informatics Institute, University of Amsterdam, Science Park 900, Amsterdam,
1098XH, The Netherlands

# Analysing and Organising Human Communications for AI Fairness-Related
Decisions

Use Cases from the Public Sector

<EMAIL_ADDRESS>Vanja Skoric<EMAIL_ADDRESS>Giovanni Sileno
<EMAIL_ADDRESS>Sennay Ghebreab<EMAIL_ADDRESS>Jacco Van Ossenbruggen
<EMAIL_ADDRESS><EMAIL_ADDRESS>*  [

###### Abstract

AI algorithms used in the public sector, e.g., for allocating social benefits
or predicting fraud, often involve multiple public and private stakeholders at
various phases of the algorithm’s life-cycle. Communication issues between
these diverse stakeholders can lead to misinterpretation and misuse of
algorithms. We therefore investigated the communication processes for
fairness-related decisions by conducting semi-structured interviews with
practitioners working on algorithmic systems in the public sector. By applying
qualitative coding analysis, we identify key elements of communication
processes that underlie fairness-related human decisions. We analyze the
division of roles and tasks, the required skills, and the challenges perceived
by stakeholders. We formalize the underlying communication issues within a
network of stakeholders in a conceptual framework that: (i) represents the
communication patterns identified in the interviews, and (ii) outlines missing
elements, such as actors who miss skills or collaborators for their tasks. The
framework is used for describing and analyzing key organizational issues for
fairness-related decisions, for collecting evidence on the communication gaps,
and for drafting interventions on the patterns of collaboration and
communication. Three general patterns emerge from the resulting analysis: (1)
Policy-makers, civil servants, and domain experts are less involved compared
to developers throughout a system’s life-cycle. This leads to developers
taking on extra roles such as advisor, while they potentially miss the
required skills and guidance from domain experts. (2) End-users and policy-
makers often lack the technical skills to interpret a system’s limitations and
uncertainty, and rely on actors having a developer role for making decisions
concerning fairness issues. (3) Citizens are structurally absent throughout a
system’s life-cycle, which may lead to decisions that do not include relevant
considerations from impacted stakeholders.

###### keywords:

Communication Framework, Fairness, Transparency, Accountability, Public
Sector, Qualitative User Study

##  1 Introduction

Algorithms are increasingly being used for various forms of public sector
services such as allocating social benefits in the domains of education,
health, and detecting fraud in allowances and taxes [1, 2, 3, 4, 5]. These
applications can be beneficial, but can also have detrimental consequences for
citizens in high-stake scenarios. Notorious examples where incorrect
predictions led to wrongful accusations of citizen minorities are the COMPAS
case in the US111Correctional Offender Management Profiling for Alternative
Sanctions (COMPAS): the software used to predict the risk of a person
recommitting a crime was more inclined to falsely accuse African-American
offenders than Caucasian offenders [6, 7]., the SyRI-case222System Risk
Indication (SyRI) was a legal instrument used by the Dutch government to
detect various forms of fraud, including social benefits, allowances, and
taxes fraud. See for instance ’SyRI legislation in breach of European
Convention on Human Rights’ at https://edu.nl/xjubf, and the Childcare Benefit
Scandal333Thousands of families had to repay child welfare subsidies after
being wrongly accused of fraud by the tax authority. See for instance the
European parliamentary questions at https://edu.nl/y3h3j. in the Netherlands.
The latter eventually led to the resignation of the Dutch government in
2021444See e.g. “Dutch Government resigns over Child Benefit Scandal”, The
Guardian (1, 2021),https://www.theguardian.com/world/2021/jan/15/dutch-
government-resigns-over-child-benefits-scandal.

These examples highlight the problem of fairness in AI. Fairness in this
context refers to fair outcomes for decision-making, a principle that
prescribes that algorithmic decision-making must have an absence of prejudice
or favoritism toward an individual or group based on their inherent or
acquired characteristics [6]. Nowadays, fairness –and related issues in AI –
are widely recognized in well-established legal and ethical guidelines [8, 9,
10]. According to the European Commission’s Ethics guidelines on trustworthy
AI [9], an important step in supporting trustworthy AI includes involving and
educating all stakeholders about their roles and needs throughout the AI
system’s life-cycle. Indeed, algorithms are always part of a process driven by
many stakeholders’ design choices and socio-cultural norms [11]. All the
(design) decisions that are made throughout a system’s life-cycle codify the
underlying socio-cultural norms of the stakeholders [12, 13, 14]. For
instance, when allocating benefits in the public sector, it has to be decided
which data features are relevant for the ‘eligibility’ for social benefits
[15]. Furthermore, the punitive (e.g., detecting fraudsters) or assistive
(e.g., allocating social benefits) nature of policy interventions might
require balancing false positive and false negative rates [16, 2]. Therefore,
a solely technical approach to fairness is insufficient, and involving diverse
actors and stakeholders is important for ensuring that public interests are
prioritized and that potential harms are minimized [17, 18].

To address such issues, we investigate the communication and collaborations
between stakeholders throughout an algorithm’s life-cycle. In this paper, we
use a working definition of fairness-related decisions for all design
decisions and practices applied by stakeholders that can potentially lead to
bias, discrimination, and other forms of prejudice against different groups,
individuals, or communities [11, 15, 19]. We do not consider a predefined
scope of fairness-related decisions and focus on the non-exhaustive scope of
decisions that emerge along our investigations. We focus on internal
communications between the direct stakeholders who use or build a system,
rather than external communications with the general public [19, 20]. We do
this by identifying the roles, the divisions of tasks, the required skills,
and the potential communication challenges between diverse actors occurring
throughout the algorithm’s life-cycle. The research questions we address are
the following:

  * •

RQ1: Which actors, roles, and tasks can be identified in multi-stakeholder
interactions throughout the phases of an algorithm’s life-cycle when making
fairness-related decisions?

  * •

RQ2: Which communication patterns and challenges can be identified when
stakeholders make fairness-related decisions?

To answer these questions, we conducted 11 semi-structured in-depth interviews
with public practitioners working on algorithmic systems. For reasons of
better accessibility, we concentrated on experts from organizations in the
Netherlands, but the methodology applied in the study can be easily replicated
in other contexts to further extend our results. From the interviews, we
identified who makes decisions about what, and at which phase of the
algorithm’s life-cycle. We analyzed the interview transcripts to identify the
elements that constitute communication patterns and challenges, and we labeled
them through in-vivo, descriptive, and process coding [21].

We further structured our findings by building a conceptual framework that
draws the key relationships between the constitutive elements of communication
patterns that underlie fairness-related human decisions. First, we found that
it is crucial to differentiate stakeholders by the individual actors, the
roles that actors assume when contributing to a task that involves fairness-
related decisions, and the skills that a task requires or an actor has. For
example, simply describing a stakeholder as a developer can omit to indicate
that the same actor (i.e., the same person) also assumes the role of advisor
with domain expertise when they decide which features are to be used as
predictors for detecting fraud. Not only do such actors endorse more than a
developer role, but they can also miss the skills required for their extra
role. Second, we found that it is crucial to identify the elements that
stakeholders are missing, to describe the communication challenges that
stakeholders experience when making fairness-related decisions. Thus, the
conceptual framework we derived for analyzing the communication patterns in
fairness-related decisions has 3 main characteristics: (1) it differentiates
actors from their roles or skills; (2) it considers 6 key elements of
communication patterns: Actors, Roles, Skills, Tasks, Information exchange,
and Phases in the algorithm’s life-cycle; and (3) it can specify the elements
that are deemed missing in the communication patterns. After analyzing the
interview transcripts using this conceptual framework, we formalized 3 general
patterns that emerged from the participants: (1) Developers play the most
prominent role in most tasks and phases of the algorithm’s life-cycle even
though they miss guidance from stakeholders with advisor and policy-maker
roles and domain expertise skills; (2) end-users and policy-makers often lack
the technical skills to interpret a system’s limitations and uncertainty, and
the related fairness implications; and (3) inputs from citizens are
structurally absent in fairness-related decisions throughout a algorithm’s
life-cycle.

These communication challenges indicate inadequate model governance, and the
potential inability to recognize and address fairness issues throughout an
algorithm’s life-cycle. This can lead to misinterpretation and misuse of
algorithms, with critical implications for the impacted populations. The
communication challenges we identified, and the conceptual framework we
derived may help identify such issues before they arise in practice, after
algorithms are deployed.

##  2 Related Work

Several frameworks and theories from various domains have been proposed to
characterize the dynamics of interactions amongst a network of actors [22,
23]. Actor-Network Theory (ANT) and mediation theory, for example, describe
the relations and interactions within a network of (artificial and natural)
actors [24, 25, 23]. Following ANT, interaction with technology is never
neutral as it influences or mediates the way tasks and decisions are carried
out. On the other hand, technology is continuously mediated by human social
aspects, e.g. in formulating design goals. To describe the context of
reciprocal interactions between human actors and technology, we can broadly
refer to socio-technical systems (STS) approaches [22]. A view centered on STS
does not consider technology alone, it rather stresses the interactive nature
of social and technical structures within an organization or society as a
whole. This approach is increasingly used in the field of AI, to assess
fairness and ethics from a broader normative context in which actors interact
and operate, as opposed to focusing on individual actors alone [26, 27, 28].

Other frameworks have been proposed to investigate the power structures within
a network of actors. Following the tripartite model for ethics in technology,
three main roles are often identified through their responsibilities: (1) the
developer, who handles the technical aspects; (2) the user, who handles the
practical usage of the system, and (3) the regulator’s role, who is
responsible for making the value decisions [29]. Prior research on automated
systems for public decision-making has shown a shift of discretionary power
from the regulator roles to developer roles, often making the latter the main
decision-makers [30]. When developers become the main decision-makers for
design decisions, this can exclude stakeholders without technical knowledge
from important decisions about the system [31, 32]. These imbalanced power
dynamics can lead to a form of technocracy, where governance and (moral)
decision-making are based on technological insights and may only yield
technological solutions [29, 33, 18].

Beyond these theoretic considerations, empirical field research has been
conducted to investigate data practices at local governments [34, 20, 35]. For
instance, Siffels et al. (2022) argue that with the process of
decentralization in the Netherlands, many tasks from the central government
were delegated to municipalities without giving them more resources and
capacities. Municipalities invested in data practices to deal with additional
tasks and to distribute limited (social) resources. Due to a lack of data
literacy, however, public servants were unable to recognize ethical issues and
thus sought collaboration with external partners. Other research showed that
depending on their roles and tasks, stakeholders can be involved at different
phases in the algorithm’s life-cycle [36, 37, 5, 38]. Decision-makers from
public organizations are often involved in the procurement and deployment
phases. Developers, sometimes from third parties, tend to be more involved in
the development phase [36, 38]. This can sometimes lead to “The problem of
many hands”, which indicates a decreased ability to be transparent and
responsible, because parts of the management of the algorithm’s life-cycle are
outsourced to different stakeholders [34, 39]. Jonk and Iren (2021) performed
semi-structured interviews with practitioners at 8 municipalities, to
investigate the actual and intended use of algorithms [35]. They found a lack
of common terminology and algorithmic expertise, at a technical level and at a
governance and operational level. The authors argue that municipalities would
benefit from a governance framework to guide them in the use of tools,
methods, and good practices to handle potential risks. Lastly, Fest, Wieringa,
and Wagner (2022) investigated how higher-level ethical and legal frameworks
influence daily practices for data and algorithms used in the Dutch public
sector [20]. They found that applying existing frameworks remains challenging
for practitioners because they do not feel competent or miss the required
skills to make decisions for their practices to be responsible and
accountable. Data professionals, as a result, get too much autonomy and
discretion power for handling decisions that belong to the core of public
sector operations and mandates.

What is still missing in previous work is a framework to characterize the
communication processes that underlie fairness-related human decisions
throughout an algorithm’s life-cycle. The frameworks and theories in related
works indicate that such communication and decision processes arise within a
socio-technical interactive network, where algorithms are part of a governance
structure comprising actors with different roles and tasks. The literature
also shows that our research must consider the interactions between
stakeholders who have direct or indirect interactions with an algorithm, and
with the populations impacted by the algorithm. Thus, we aim at identifying
how fairness-related decisions are mediated by stakeholders who may or may not
have direct access to socio-technical information that is relevant for
addressing fairness issues.

##  3 Methodology

###  3.1 Semi-structured interviews

We conducted 11 semi-structured interviews. Each interview lasted for
approximately one hour. We formulated the interview questions in an open-ended
manner, where participants were able to share their information in their own
words whilst following a general structure of topics [40, 41]. Before
conducting the interviews, participants received some example questions and a
short description of the research. At the start of the interview, participants
gave their consent for their interview to be used in this research. Also, they
were asked to discuss one use case they were involved in. The questions used
for the interviews can be found in Table 9 in the appendix and are divided
into three main sections:

  1. 1

General: Investigation of the project and use case to which the participant
contributed, the other actors involved, and the participant’s team, roles, and
envisioned (end) users.

  2. 2

Development process: Investigation of the type of datasets, resources, tasks,
phases, and roles needed throughout the algorithm’s life-cycle to make
fairness-related decisions.

  3. 3

Considerations: Investigation of the perceived challenges for role and task
division, the potential improvements or failures of the system, and the
communication gaps. The questions also concerned the assessment of error and
bias, and the the potential negative impacts of the algorithm.

In the first two sections, participants were asked to describe the general
procedures and practices used in the AI system’s life-cycle. Participants had
the opportunity to mention internal communication and key elements of the
communication processes that underlie fairness-related human decisions. We
specifically asked about communication issues in the third section of the
interview. This division was made to provide the opportunity for spontaneous
answers beyond our specific questions.

We preliminary tested all interview questions with a pilot with 5 researchers
from different disciplines in our research lab. The questions were deemed
suitable for letting participants describe their communication process and
related issues. The suitability of the questions was checked in terms of
comprehensibility and relevance to our research questions. No questions were
altered afterwards.

###  3.2 Case Studies

We recruited participants who have been collaborating on multi-stakeholder
projects in the public sector. Participants working in the social domain, e.g.
social benefit allocation or fraud detection were of particular interest
because the impacts on citizens can be critical. We used a repository of use
cases that was made available to us by the Dutch Ministry of Interior
Affairs555Some examples of public domain use cases in the Netherlands can also
be found via the Artificial Intelligence Netherlands Coalition (NL AI
Coalitie) website (https://nlaic.com/use-cases) and in [4, 5].,666Dutch
Ministry of Interior Affairs and Kingdom Relations
https://www.rijksoverheid.nl/ministeries/ministerie-van-binnenlandse-zaken-en-
koninkrijksrelaties. Next to that, we used the snowball sampling technique to
recruit participants.

Interviewee | Role | Technical background
---|---|---
P1 | Developer & Researcher | yes
P2 | Manager & Researcher | no
P3 | Manager | no
P4 | Manager | yes
P5 | Advisor & Researcher | no
P6 | Developer & Researcher | yes
P7 | Manager | no
P8 | Advisor & Researcher | yes
P9 | Developer & Researcher | yes
P10 | Advisor & Researcher | no
P11 | Manager | yes
Table 1: Description of participants

Table 1 describes the participants, their roles at the time of involvement,
and if they have a technical background. We consider those who are not
educated or have no experience in technical science to not have a technical
background. 10 participants were involved in the social security domain, and 1
participant was in the education domain.

###  3.3 Qualitative Coding analysis

We performed a qualitative coding analysis by labeling key codes777Saldaña
(2013) describes that “A code in qualitative inquiry is most often a word or
short phrase that symbolically assigns a summative, salient, essence-
capturing, and/or evocative attribute for a portion of language-based or
visual data.” In addition, a code can be understood as a researcher-generated
construct that symbolizes the construct and assigns an interpreted meaning.
from the interview output. We used in vivo888“In Vivo” coding is also named
“literal coding” and refers to a word or short phrase from the actual language
found in the qualitative data record e.g., terms used by participants
themselves [42, 21]., descriptive999Descriptive coding refers to summarizing
the basic topic of a passage of qualitative coding in a word (noun) or short
phrase [21]. and process coding101010Process coding refers to “action coding”
which implies action from more simple observable activity (e.g., reading) to
more general conceptual action (such as adapting) [21]. to identify the
process of communication exchange between diverse actors, as well as the
practices and choices made at each stage of the algorithm’s life-cycle.

The coding analysis was performed in multiple cycles. At each round of coding,
pieces of text are annotated with codes that represent the concepts mentioned
by participants. The codes are refined, merged, or split into categories after
each round. This was repeated until no further refinement of the codes was
needed. Two of the authors perform a separate coding analysis to reduce the
impact of personal bias. We performed coding analysis by hand and using a
coding analysis tool111111Atlas.ti: The Qualitative Data Analysis & Research
Software https://atlasti.com/. We compared both coding analyses to identify
discrepancies or alignments. Beforehand, both analysts agreed that particular
attention should be drawn to identifying the roles, tasks, phases, and
challenges from the interview transcripts. For example, if a participant were
to mention that ”[person X] is a developer and performs bias analysis in the
development phase”, the actor, the role, the task, and the phase would be
labeled.

Figure 1: Example of interview Q&A (left) and the corresponding coding labels
(right) from the qualitative coding analysis

In Figure 1, an example is given for the interview output (left) and the
corresponding codes (right). The Figure shows the colors corresponding to the
groups of codes for challenge, roles, task, and phase. On the right, an
example of the corresponding descriptive codes can be found. For example,
“it’s hard to get a focused answer” was summarized as an information exchange
challenge of the type where “more input is needed”. We added the corresponding
role(s) to the codes in brackets “[]”. If the code concerned multiple roles,
we added “-” to indicate a relation for information exchange. In this example,
more input is needed between the end-user and the developer role.

After all interview transcripts were annotated with codes, we analyzed which
codes co-occurred within the answers to each question, e.g. we counted which
roles occurred together with a specific phase, task, or challenge.

###  3.4 Constructing a conceptual framework

The co-occurrence analysis alone did not capture the relations between codes,
i.e. “end-user is missing in the development phase” would still count as a co-
occurrence of the codes end-user (role) and development (phase), although the
role of end-user was actually missing. Therefore, we constructed a framework
that further analyses the codes we identified by describing their relations
and characteristics. The conceptual framework aims to describe the key codes
and relationships between high-level groups of codes (e.g. Actor, Role, Skill,
Task, Phases), and the key characteristic that underlie the challenges
mentioned in the interviews. We can then represent challenges such as citizens
are actors with the role of Data subject (an Actor-Role relationship), and
that actors with such roles are “missing” (a characteristic of Actors).

We constructed the conceptual framework iteratively, following a method
similar to those used for constructing ontologies [43, 44, 45, 46]. This means
that we continuously adjusted the framework until it would represent every
code we identified from the qualitative coding analysis. We added definitions,
characteristics, and properties to the identified concepts. We added
descriptions to each concept to agree on common definitions. The relationships
and characteristics we used to build the conceptual framework are based on the
interviews and were in accordance with some of the definitions we found from
documents provided by the European Commission on Trustworthy AI, and from
other sources in the literature [8, 9, 44, 45, 47, 29]. For example, by
describing the type of private or public affiliation (e.g., national
institute, ministry, or municipality) we can contextualize how tasks and roles
are divided within multi-stakeholder collaborations. Relations are added
between codes. For example, an actor always “has” a certain role whereas a
task “involves” a role “during” a phase.

##  4 Results

In the next section, we first describe the use cases discussed by the
participants in the interviews (section 4.1). Then the results of our
qualitative coding analysis are given in section 4.2. Finally, we further
analyze the communication challenges and apply them to document the
communication patterns and challenges we identified in Section 4.3.

###  4.1 Use cases

In all use cases, multiple stakeholders were involved with varying
expertise—from social workers to developers, researchers, program managers,
and advisors from third parties. For most use cases (10 out of 11), the
procurement for the algorithm came from government organizations and
municipalities. Furthermore, the envisioned end-users of the systems were in
10 out of 11 cases policy-makers or social workers at municipalities with
minimal or no technical expertise. End-users and policy-makers were mentioned
to be the same in most of our use cases. For the remaining use case in the
educational domain, teachers were the envisioned end-users.

###  4.2 Qualitative Coding Analysis

####  4.2.1 Identified Codes and Concepts

Our qualitative coding analysis first focused on identifying the main types of
Roles, Tasks, and Challenges. It resulted in identifying 7 codes for
describing the main roles (Table 2), 10 codes for the tasks (Table 3), and 7
codes for the challenges (Table 4.) In this section, we explain in more detail
the concepts that these codes represent, and our decisions for eliciting a
consistent set of codes.

For coding the roles, we observed that the terminology is rather diverse for
the technical roles. For example, participants mentioned terms such as
engineers, coders, developers, and data scientists for the role of developer.

Some participants identified themselves or their collaborators as researchers.
We questioned the inclusion of code for the role of researcher. However, such
code can be ambiguous as the research topics could either concern the
technical development of algorithms, or other domains such as governance or
social security. Thus, we decided to group under the code “developer” the
researchers who focus on the technical development of algorithms. Researchers
that contributed from other domains sometimes assumed roles other than
developers, such as advisor or manager.

For the role of manager, participants mentioned terms such as innovation
managers, product owners, program managers, project managers, or CTO (Chief
Technology Officer). These terms were often used interchangeably. We decided
to group all management-related roles under the same code “manager”, without
using specific codes for each job title or hierarchical level.

In Table 2 the descriptions of the main roles can be found. For example,
managers are, respectively, those who “supervise the projects for the
development of the system and oversee documentation checks and balances”. In
our use cases, the managers often worked in the same team as developers and
were either hired externally or internally by a (public) requester.

The request for the model —associated with the “requester” role— often came
from ministries, and they were only mentioned for funding or initiating a
project.

The “data subject” role as well as the “requester” role were never described
as the end-users. In Table 2 it is also stated that data subjects are “an
organization or entity that is impacted by the system, service or product”
[45]. The data subjects were, in almost all of our use cases (10 out of 11),
citizens. The advisor role was often presented as advising on 1) domain
knowledge, 2) technical knowledge, or 3) ethical knowledge. Overall, as
illustrated in Fig. 2, the developer role was mentioned the most (N=189),
followed by End-users (N=107) and Policy Makers (N=92).

Figure 2: Main Roles Identified. The number of times a Role is mentioned. Note
that Developers are mentioned most (N = 189) and Data Subjects least (N = 22).

In Section 4.2.2, we describe which roles occurred the most for which phase
(4.2.2) of the algorithm’s life-cycle.

Role types |  Description
---|---
Developer |  Research, design, and/or develop algorithms
Policy-maker |  Responsible for designing and overseeing the carrying out of policy and social decisions
Manager |  Supervise the projects for the development of the system and oversee documentation checks and balances
End-user |  (In)directly engage with the system and use algorithms within their business processes to offer products and services to others
Data subject |  Organization or entity that is impacted by the system, service, or product
Advisor |  Give constructive feedback on the system throughout the life-cycle
Requester |  Who are the main client and investor for the use-case
Table 2: Descriptions of Main Roles  Task |  Description
---|---
Technical Decision |  Decision-making for technical aspects in the development, training, and testing of the AI model
Consulting |  Advising on domain, technical, or ethical knowledge aspects of the AI model
Fairness & Risks |  Controlling the negative social impacts of the AI model such as discrimination
Involving Stakeholders |  Actively involving other actors throughout the AI algorithm’s life-cycle (e.g., a task of manager roles)
Researching |  Examining, studying or investigating aspects of the AI algorithms’ life-cycle
Goal Formulation |  Planning and deciding on the purpose of the AI model and its requirements
Bias Analysis |  Analysing the systematic differences between populations or individuals in the AI model output
Model Usage |  Operationalizing the AI model
Go - no go |  Deciding to proceed with the development and or implementation of the model
Auditing |  Documenting and controlling the process of and around the AI model
Table 3: Descriptions of Main Tasks

In Table 3, we describe the main tasks identified from the qualitative coding
analysis. For example, the task ”Consulting” refers to advising on domain,
technical, or ethical knowledge aspects of the AI model. In Section 4.2.2, we
describe which roles occurred the most for which task.

Challenge |  Description
---|---
Interpretation |  Misunderstanding of technical information regarding the AI model, such as evaluation metrics
Involvement |  Lack of participation and collaboration between actors throughout the algorithm’s life-cycle
Risk Oversight |  Problems concerning governance, legal, ethical, and procedural aspects
Resources |  Insufficient time, planning, infrastructure, money, and documentation
Feedback |  Lack of substantial input, information exchange between actors
Bias |  Problems with the analysis of prejudice towards individuals or groups
Role |  Unclear function or duty division among actors
Table 4: Descriptions of Main Challenges

In Table 4, we describe the main challenges identified from the qualitative
coding analysis. For example, ”Interpretation” issues refer to the
misunderstanding or misevaluation of information regarding the AI model. In
Section 4.2.2, we describe which roles occurred the most for which challenge.

####  4.2.2 Co-occurrences

Figures 3, 4, and 5 illustrate which Roles were mentioned the most, based on
co-occurrence with Phases (Section 4.2.2), Tasks (Section 4.2.2), and
Challenges (Section 4.2.2).

Figure 3: Co-occurrences of Roles and Phases. The number of times a Phase is
mentioned is shown on the y-axis (in decreasing order). Note that the
developer role (blue) is mentioned the most in all phases and that the data
subject role (orange) is mentioned the least.

##### Roles and Phases

Figure 3 shows that developers are most prominent in the development,
evaluation, and formulation phases, but less prominent in the deployment and
monitoring phases. Developer (P1) mentioned that “we don’t monitor what the
municipalities are doing with the results.” and “feedback is needed on how the
results will be used in deployment”. Conversely, stakeholders other than
developers could be more involved in the development phase. Another developer
(P9) mentioned “For the future, we could incorporate stakeholders at earlier
stages in the development to see what the potential sources of bias are.”

End-users and policy-makers were the second highest in occurrences for phases.
Moreover, Figure 3 demonstrates that the monitoring phase (N = 10) was
mentioned the least throughout the interviews whereas the evaluation phase was
mentioned the most (N = 90).

Data subjects were seldom mentioned to be involved. Data subjects could be
more involved throughout the phases of an algorithm’s life-cycle e.g. P5
mentioned “it depends on the type of AI. If it has an impact on citizens or
uses a lot of data from citizens, it would be relevant to include a focus
group of citizens from the beginning but it is less relevant for road
repairs.”. The role of the requester was only mentioned in the formulation
phases but rarely as being involved throughout other phases. Advisor roles
were often mentioned to be involved in the evaluation phase before deployment,
or when the project is halted.

Figure 4: Co-occurrences of Roles and Tasks. The number of times a Task is
mentioned is shown on the y-axis (in increasing order). Note that technical
decision-making is mentioned the most. Developer roles (blue) are mentioned
the most for all tasks except for model usage.

##### Roles and Tasks

Figure 4 shows that the developer role was mentioned the most for all tasks
(e.g., technical decision-making, consulting, dealing with fairness and
risks). This indicates that actors taking on developer roles were the most
prominent in making decisions throughout the algorithm’s life-cycle. About the
typical tasks developers handle, developer (P9) mentioned that they “decided
on how to improve accuracy and handling issues. For instance, gathering more
diverse data to handle bias”. About their collaboration with other roles,
another developer (P1) mentioned that they “define and chose metrics for the
models” and that these ”are defined in collaboration with the municipality but
choosing metrics and trimming down after input was decided by the two of their
team”. The developer role was not mentioned for tasks related to model usage.

Regarding the task of stakeholder involvement, managers are the main decision-
makers. Within teams, managers are sometimes the only ones in direct contact
with roles other than developers. Managers were often mentioned to supervise
developers in technical decision-making, and they often rely on the
developers’ judgment for bias and risk oversight. A manager we interviewed
(P2) mentioned that for handling error rates and biases they “rely on the
technical teams’ judgment” and that “the technical colleagues give advice when
the model is good enough, but it’s a bit of a grey area. We also rely on
literature”. Another manager (P4) mentioned that “it is time intensive to
explain [bias analysis] to stakeholder users. Bias analysis is sometimes so
complex, even as an expert I sometimes don’t understand it, and it takes a lot
of time”.

Actors with a developer role also sometimes assume advisor roles. When
technical advisors are missing, managers can hire a third-party developer to
analyze the code, give technical advice, or even build the model. An advisor
we interviewed (P5) mentioned that “an external company was hired to develop
the model for the municipality”, which made the “data ecosystem quite
complex”. Another manager we interviewed (P3) added that they “hired an
external bureau for auditing and investigating the algorithm”, e.g., as they
“could not get reliable predictions because the social domain changes all the
time, and it’s hard to keep track of these changes—for example in social
support—and how that impacts the system”.

Advisor (P10) mentioned that they “were involved to give feedback as an
involved bystander. But it was hard for someone like me to understand what the
difference between implementation and design is and what that means for real-
life implications”. This demonstrates that roles other than developers lack
the technical skills to participate in decisions.

Figure 5: Co-occurrences of Roles and Challenges. The number of times a
Challenge is mentioned is shown on the y-axis (in increasing order). The roles
of developer (blue), end-users (green), and policy-makers (red) are mentioned
most.

##### Roles and Challenges

In Figure 5 the co-occurrences of roles and challenges are shown. Most
communication challenges were associated with the roles of developers, end-
users, and policy-makers.

The role of the end-user was frequently mentioned for challenges related to
interpretation and role. This means that most challenges were related to
either the (mis)understanding or (mis)evaluation of information or an unclear
function or duty division amongst actors. Several participants mentioned that
more input is needed from end-users on the interpretation and use of the
results envisioned in the deployment phase. For instance, a manager (P3)
mentioned that it is challenging that “we don’t know if governments and
municipalities can understand the model”. A developer (P1) also mentioned that
“it’s hard to get a focused answer on how they are going to use the model and
what the results will be’, and that “the municipality is too loosely involved
in the project.

Regarding the challenges with bias, risk oversight, and the interpretation of
model output, the role of policy maker was frequently mentioned for the
challenges related to risk oversight. This concerns problems in governance,
legal, ethical, and procedural aspects. The role of developer was frequently
mentioned for challenges related to resources, input, and bias. It was
mentioned that more input is needed on the analysis of feature selection and
bias in the development phase from developers to end-users and policy makers.
Both end-users and policy-makers were often mentioned as missing the technical
skills to understand the uncertainty of predictions and limitations of the
model in real-world settings. A developer (P6) mentioned about end-users that
“people could trust the model blindly and mistake it for a decision-making
tool” and another advisor (P8) where inspectors were the end users mentioned
that they were “not sure if the inspectors fully understood why certain cases
were flagged as misuses or put on the list [of potential frauds]”.

With regard to challenges for bias and risk oversight, an advisor (P8)
mentioned that “there should be more focus on asking users what policy-makers
perceive as risks and biases” and that it is “difficult for them to understand
that there are many different interpretations. What it really means to be a
’true positive’, is this person really a fraud, or was this person not able to
fill in the forms properly?”. Interpretation challenges by end-users and
policy-makers were mentioned most in the monitoring and deployment phases. A
developer (P6) mentioned that “training for users is needed, to remind users
not to rely on the tool but that the decision is up to them.”.

Data subjects (e.g. citizens) were also mentioned for involvement and risk
oversight challenges. Participants mentioned a  need for more citizen
involvement and for being more transparent to citizens throughout the phases
of an algorithm’s life-cycle.  Managers are looking for appropriate frameworks
for (fruitful) collaborations with citizens. Manager (P4) mentioned that
“there is a long history with the citizen council for consultation and it is
usually conflict-based. It’s hard to make fruitful collaboration, getting them
to understand the issues and getting them out of anger mode”. An advisor (P7)
mentioned about previous involvement with citizens that “they [citizens] said
no on the feasibility of the model from the municipality. They did not get it.
It was more of a general no to technology instead of asking a targeted
question”.

####  4.2.3 Key Insights from Qualitative Coding Analysis

From the qualitative coding analysis, we conclude that: (1) Actors with
developer roles are predominant in most phases and tasks while potentially
lacking the required guidance from domain experts. (2) End-users and policy
makers often lack the technical skills to interpret a systems output or
estimate potential fairness issues. (3) Citizens filling the role of data
subjects are seldom mentioned to be involved throughout the phases of the
algorithm’s life-cycle. In the next section, we analyze these challenges
further by characterizing the relations between the main elements of
communication patterns in a conceptual framework.

###  4.3 Modeling Communication Patterns

The communication patterns emerging from the interviews are not easily
described with qualitative analysis in written form only. The codes may
identify the key elements of communication patterns, but not their
relationships. Counting the (co-)occurrences of codes could not fully capture
these relationships. We thus elicited a conceptual framework that models the
relationships between the elements of the communication patterns (e.g.,
between actors, roles, and skills). This also allowed us to explore the
perspective of socio-technical systems (Section 2), in which AI models and
fairness-related decisions arise through the interactions between actors. The
conceptual framework we elicited is shown in Figure 6 and detailed in Table 5.

Figure 6: The basic concepts selected from our qualitative coding analysis,
and used to characterize the communication patterns underlying fairness-
related decisions, and the challenges we identified.

We elicited 6 elements to describe the communication patterns: Phase, Role,
Task, Skill, Actor, and Information Exchange. At least two concepts were
needed to characterize the communication process: the stakeholders who
exchange information (represented by the concept Actor), and the act of
communicating (represented by the concept Information Exchange). Describing
the context of the communications requires at least 4 additional concepts
(Phase, Task, Role, Skill) to underlie fairness-related human decision-making
and their challenges. For example, Tasks may be missing at certain Phases of a
system’s life-cycle. Or Actors may not have the right Skill or Role when
making a fairness-related decision. Skill was added as a key element of the
communication pattern because the interviews showed that the challenges that
stakeholders face often arise from the mismatch between their role and skill.

Concept |  Description |  Relation |  Property
---|---|---|---
Task |  Actions that Actors perform |  Involves Role, Skill, Info exchange, During Phase, Actors contribute to |  is missing
Role |  Function filled by Actor |  Actors Have Role, Task involves Role, Role involves Skill |  is missing
Actor |  Entities that perform Tasks, actively or passively |  have Role, Skill, Contributes to Task, Info Exchange between Actors |  affiliation: public (national institution, ministry, municipality) or private, is missing
Phase |  Indicates the evolution of the system from conception through retirement |  Task during Phase |  is missing
Skill |  Professional ability, expertise, or knowledge needed in practice to complete a specific task |  Task involves Skill, Actor(s) have Skill, Role involves Skill |  is missing
Information exchange |  Communication transfer between Actors |  between Actor(s), Task involves Info exchange |  is missing
Table 5: Description of concepts used to characterize the communication
patterns underlying fairness-related decisions

To provide a temporal overview of the communication processes, we link the
Tasks to the Phases of the system’s life-cycle in which they take place (e.g.,
to reflect on the fairness-related) Tasks that are executed at specific
Phases). We link the Tasks to the Actors and Information Exchange they
involve, to represent the stakeholder collaborations for each Task. Finally,
we relate the Actors to their Roles and Skills, and also link the Roles to the
Skills they require.

Adding the property is missing to any of the 6 elements in the communication
model is of great interest for documenting the challenges mentioned in the
interviews. We chose to represent the communication patterns using these 6
elements precisely because challenges arise if any of them are missing. For
example, an Actor may miss specific Skills, or a fairness-related Task may be
entirely missing.

Adding information on the affiliation of Actors is also of interest to better
describe the stakeholders involved in fairness-related decisions, and to
identify potential issues with conflict of interests, privacy, accountability,
or legal frameworks.

The elements and properties of this conceptual framework were sufficient to
represent the communication patterns we observed in the interviews. Adding
more elements or properties would come at the risk of making it harder to
generalize to new contexts.

In the next section, we apply this conceptual framework to illustrate three
relevant patterns observed in the challenges.

####  4.3.1 Pattern 1: Actors with a developer role are predominant and miss
guidance from domain experts.

Several participants mentioned challenges with the involvement and role of
stakeholders with domain expertise (Table 4). Actors with a developer role are
predominant, especially at the beginning of an algorithm’s life-cycle, i.e.,
formulation, development, and evaluation phases (Figure 3). Developers make
decisions that seem technical but have crucial implications for fairness and
public policy. Yet, actors with domain expertise may not be involved in
guiding such seemingly technical decisions. Actors with technical skills
become the main decision-makers, while they potentially miss domain expertise
skills and stakeholders with the roles of advisor and policy-makers.

Such technical decisions with fairness implications include, e.g., balancing a
model’s False Positive and False Negative rates, or fairness metrics based on
these error rates. Domain expertise is needed to assess the practical
implications of each type of error121212For instance in punitive use cases
[1], False Positives (e.g., accusing innocents of fraud) can be more
problematic than False Negatives (e.g., undetected fraud). In assistive use
cases, False Negatives (e.g., failing to help individuals in need) can be more
problematic than False Positives (e.g., helping less vulnerable individuals)..

Figure 7: Example of challenges with stakeholder involvement and role that constitute Pattern 1. Apparent technical decisions, such as defining which AI method to use and with which data features, have domain implications in practice but are made by developers alone. Other stakeholders with domain expertise are not involved in guiding the technical decisions. The missing information exchange is about the representativeness of the data features and their applicability to the use case. Participant |  Quote
---|---
P2 |  [For handling bias and error rates] ”the technical colleagues give advice when the model is good enough, but it’s a bit of a grey area. We also rely on literature and on the technical teams’ judgment”.
P3 |  “Hired an external bureau for auditing and investigating the algorithm”. [Also because they] “couldn’t get reliable predictions because the social domain changes all the time, and it’s hard to keep track of these changes”.
P5 |  ”An external company was hired to develop the model for the municipality, which made the data ecosystem quite complex”
P8 |  ”There should be more focus on asking users what policy-makers perceive as risks and biases”
P9 |  ”Involvement and direct information of the operators who work with AI system is needed, which particular change or improvement would be most useful for them”
Table 6: Quotes illustrating the communication pattern in Figure 7 where
information on data quality was missing.

Our conceptual framework (Figure 6) can be used to represent such
communication issues. For example, Figure 7 illustrate the quotes from Table
6. Developers must decide which data features are suitable for a use case, and
how to use them withing AI systems. Domain experts could inform developers
about the context in which the data features are representative of specific
social groups. Without such information, developers may decide to use data
features in ways that produce biased results for specific social groups.

####  4.3.2 Pattern 2: End-users and policy-makers may lack the technical
skills to interpret the system’s limitations and uncertainty

Several participants mentioned challenges with the interpretation of a
system’s limitations and uncertainty (Table 4). At the deployment phase,
actors with end-users and policy-maker roles may question whether the system
delivers what it is supposed to perform, and how to interpret the validity of
its results. Yet, they may not have the technical skills to understand the
limitations of a system. They may miss guidance from actors with a developer
role, who have the technical skills to understand the uncertainty and the
practical limitations they entail.

Our conceptual framework (Figure 6) can be used to represent this
communication challenge. For example, Figure 8 illustrates the quotes from
Table 7.

This finding highlights a need for increased input between end-users, policy-
makers, and developers at the right phase. It echoes Pattern 1, where actors
with a developer role lack guidance on the implications of their technical
choices. These directly impact a system’s limitations and uncertainty.

Figure 8: Example of challenges with model interpretation: the actors that use AI models, or make policies involving AI models, may miss the skills to understand model limitations and error metrics. They may also miss information exchange with developers who can explain the limitations and uncertainty.  Participant |  Quote
---|---
P1 |  Most important risk is that the model will not be used or is misinterpreted. For example, mixing up correlation and causality might lead to not helping people at risk of poverty.”
P3 |  ”We don’t know if governments and municipalities can understand the model. 131313Upon asking their consent for publishing the quote, P3 added: ”We don’t know if governments and municipalities [have the capacity in time and competence] to fully understand the model [so they can use it for policy tasks].”
P5 |  [The most difficult challenge is] “the gap between data scientists and policy-makers. How to make sure that what is developed is being well understood and useful for those of non-tech background.”
P6 |  ”Training for users is needed, to remind users not to rely on the tool but that the decision is up to them.”
P8 |  ”The difficult for them to understand that there are many different interpretations. What it really means to be a ’true positive’, is this person really a fraud, or was this person not able to fill in the forms properly?”
Table 7: Quotes illustrating the communication pattern in Figure 8.  Figure 9:
Example of challenges with the involvement of citizens: they are structurally
absent throughout the algorithm’s life-cycle, although they are the data
subjects whose data is collected and processed, and who are impacted by the
deployment of algorithmic systems. Information exchange is missing for them to
understand and comment on the many design choices that impact fairness.

####  4.3.3 Pattern 3: Citizens are structurally absent throughout the
algorithm’s life-cycle.

Some participants mentioned challenges with the lack of involvement from
citizens, e.g., who have the role of data subjects. For instance, in the
formulation phase, citizen participation may be missing to give feedback on
the design choices of the model. It is interesting to note that most
participants did not mention citizens, and may thus overlook issues with their
participation in a system’s design or evaluation.

Our conceptual framework (Figure 6) can be used to represent this
communication challenge. For example, Figure 9 illustrates the quotes from
Table 8. A lack of citizen involvement may lead to unbalanced fairness-related
decisions that do not include key practical considerations.

Participant |  Quote
---|---
P4 |  “There is no direct citizen participation.”
P5 |  “it depends on the type of AI. If it has an impact on citizens or uses a lot of data from citizens, it would be relevant to include a focus group of citizens from the beginning but it is less relevant for e.g. road repairs.”
P7 |  [On previous involvement with citizens] “they said no on the feasibility of the model from the municipality. They did not get it. It was more of a general no to technology instead of asking a targeted question”.
Table 8:  Quotes illustrating the communication pattern in Figure 9.

##  5 Discussion

By characterizing the relations between the concepts in a conceptual
framework, we demonstrated that unclear or undefined governance structures for
roles, tasks, and skills can lead to misinterpretation of the system’s
limitations and uncertainty, and even to misuses of the algorithmic system.
From our use cases, we also saw that it was not always clear who makes final
(mostly policy) decisions on the further development or use of algorithms, or
what is the (legal, procedural, information) basis for such decisions. When
there is a lack of actors filling the right roles at the right phase, actors
can take on multiple roles at once for which they may not be fully equipped,
which can lead to a discretionary imbalance.

It is possible that participants forgot to mention involved stakeholders in
the development process or did not see some social participants as influential
for choices, practices, and protocols. Forgetting a particular role or actor
does not necessarily reflect the actual governance structure or experienced
communication challenges. Participants may not have oversight, be unwilling to
provide specific details, or were perhaps steered by how the interview
questions were formulated.

We also recognize that, in practice, formulations for roles and groups can
vary and can be diffused. For instance, in some cases, actors identified as
developers would primarily identify themselves as a researcher who would also
carry out“developer tasks”. We stress again that counting (co-)occurrences
alone is not enough to assess the structure of the communication process that
underlies fairness-related decisions. As mentioned in the results, sometimes
an occurrence would be counted for role and phase when actually ‘role X was
missing in phase Y’, and thus the relations between concepts needed to be
characterized to provide context to our findings. The conceptual framework we
constructed covers comprehensively the relations that appear in the
interviews, yet, it may not be sufficient for other scenarios. Fortunately,
the incremental method applied for its construction allows for easy extension.

The number of interviews was limited (N=11) and based on participants who
collaborated on Dutch social domain use cases. The fact that all use cases
reside in the Netherlands was for the sole reason that it was more readily
available to us. It is important to emphasize that every (public sector) use
case will have its own (normative) context, specific governance, and
communication structure. With our use cases, we tried to appreciate these
local conditions and resist “the portability trap” of stating that every AI
use case will function the same from one context to another [48, 49].

Regardless of the stated limitations, our findings confirm and complement
earlier work emphasizing the growing autonomy and discretion of developers in
public sector operations, as well as the unclear role divisions for the usage
of automated decision tool [20, 31, 30, 29]. In terms of conceptual
reorganization and synthesis, the number of interviews was sufficient to
demonstrate the value of studying communication process underlying the choices
and criteria for fairness-related decisions. Besides, the methodology we
applied, consisting of (a) qualitative research, (b) qualitative coding
analysis, (c) incremental construction of a conceptual framework, and (d) the
application of the conceptual framework on acknowledged challenges, is rather
generic, and we do not foresee constraints to its reuse in different, wider
contexts. Yet, in terms of more general factual knowledge, More research is
needed to investigate other local governance structures and communication
processes around fairness-related decisions.

##  6 Conclusion

In this research, we investigate fairness-related decisions through
communication processes between diverse stakeholders that work on AI
algorithms in the public sector. We conducted semi-structured interviews to
analyze the divisions of roles and tasks, the required skills, and the
perceived challenges throughout the algorithm’s life-cycle. We applied
qualitative coding analysis, to identify key elements of the communication
processes that underlie fairness-related decisions. The results are formulated
in a conceptual framework that represents these key elements as well as
missing elements such as actors who miss skills or collaborators for certain
tasks. To evaluate the adequacy and value of this methodology for the study of
communication processes concerning fairness-related decisions, we applied it
to social domain use cases in public organizations based in the Netherlands.
The results we found are potentially relevant for policy interventions, as
they generally indicate a lack of involvement and feedback between developer,
end-user, and policy-maker roles. More precisely, we have captured the
following key observations: (i) Developers play the most prominent role in
most tasks and phases of the system’s life-cycle. They may miss guidance from
stakeholders with advisor and policy-maker roles, and domain expertise skills.
(ii) End-users and policy-makers often lack the technical skills to interpret
the system’s limitations and uncertainty and to estimate potential fairness
issues. They rely on the technical skills of developers for making apparent
technical decisions such as feature selection and balancing error rates, which
potentially influence policy outcomes. (iii) Lastly, we observed that citizens
are structurally absent throughout the system’s life-cycle, even though it is
mentioned that their involvement is needed in the future for balanced
fairness-related decision-making. These findings indicate that model
governance is currently inadequate, and that there is a potential inability to
recognize and address fairness issues throughout an algorithm’s life-cycle.
This can lead to misinterpretation and misuse of algorithms, with critical
implications for the impacted populations. The conceptual framework we derived
can help to address such issues before deployement, highlighting where to
intervene (e.g. with adequate communications, gathering necessary skills
currently missing, or introducing new roles), before the algorithm goes
actually in production.

Table 9: Questionnaire Questions |  Notes
---|---
Institution / Department |  Name of entity/department
What is your (team’s) role? |  Description team/staff involved brief
Who do you work with (directly)? |
Domain and topic of use case |
Start and end date |
What type of system is being developed for the use case? |  Intended use/aim
What is the goal of the system? |
External partners developing technology for the use case (if any) |
Who are the (end) users - are they directly involved in the development process? |
II. Development process |
Could you guide us through the process of development by mapping out phases - and specific actions in each phase? |
Could you guide us through the decisions made about the system and by whom? |  e.g. involvement in deciding on: the goal of the system — design of the system (metrics/labeling/test/training/error) — evaluation of system — monitoring — deploying system
What kind of decisions do you and your team make? (could you give an example) |
What input is needed /do you use to make decisions as a reference point |  e.g. handbook, training, expert group?
What kinds of exchanges are needed in your decision process? |
Are there other teams or (external) stakeholders involved in the decision-making process? |
How do you support the decision process of your collaborators with your output? |
III. Considerations |
How can the development process be improved for the following, from your perspective? |  e.g., information exchange, role division, handling error rates and biases, handling risks, responsibilities
What are the most difficult challenges and risks of failures for the system? |
How are these challenges and risks measured assessed and monitored? |
What information is needed (by whom) to handle these challenges and risks? |
Who is consulted for this information? |
What is your role in the process of addressing challenges and risks? |
Could any issues occur that might halt the development process, (if so could you give an example of how are these go/no-go decisions determined?) |
In real-life applications, could there be specific risks or negative impacts for individuals or social groups? |
Is error analysis / Bias analysis performed for negative impacts (if so how is this done and could you give an example)? |
Once the algorithm is deployed in practice, what kind of human oversight is available to control for error, bias, or negative impacts? |
What procedures and recourses, if any, are available for addressing the negative impacts of the system? |
Do you have access to explanations or training on the risks for individuals and social groups, e.g., from your colleagues or from external experts? |
IV. Follow-up questions |
Who is in charge / responsible for mitigating measures on respecting privacy and data protection? For instance, is there a valid legal basis for processing personal data? |
Are there cybersecurity or privacy-preserving measures deployed to preserve privacy and data security? |
If no challenges (or very few) concerns are mentioned in 4, provide a scenario? |  e.g. complaints about the output; security breaches; what if the training set is not representative, high error rates

## References

  *   * Rodolfa et al. [2021] Rodolfa, K.T., Lamba, H., Ghani, R.: Empirical observation of negligible fairness–accuracy trade-offs in machine learning for public policy. Nature Machine Intelligence 3(10), 896–904 (2021)
  * Rodolfa et al. [2020] Rodolfa, K.T., Salomon, E., Haynes, L., Mendieta, I.H., Larson, J., Ghani, R.: Case study: Predictive fairness to reduce misdemeanor recidivism through social service interventions. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 142–153. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372863 . https://doi.org/10.1145/3351095.3372863
  * Williamson [2016] Williamson, B.: Digital education governance: data visualization, predictive analytics, and ‘real-time’policy instruments. Journal of education policy 31(2), 123–141 (2016)
  * Van Veenstra et al. [2019] Van Veenstra, A.F.E., Djafari, S., Grommé, F., Kotterink, B., Baartmans, R.F.W.: Quickscan AI in the Publieke dienstverlening (2019). http://resolver.tudelft.nl/uuid:be7417ac-7829-454c-9eb8-687d89c92dce
  * Hoekstra et al. [2021] Hoekstra, Chideock, Veenstra, V.: TNO Rapportage Quickscan AI in the Publieke sector II (2021). https://www.rijksoverheid.nl/documenten/rapporten/2021/05/20/quickscan-ai-in-publieke-dienstverlening-ii
  * Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys (CSUR) 54(6), 1–35 (2021)
  * Fass et al. [2008] Fass, T.L., Heilbrun, K., DeMatteo, D., Fretz, R.: The lsi-r and the compas: Validation data on two risk-needs tools. Criminal Justice and Behavior 35(9), 1095–1108 (2008)
  * Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self assessment. Publications Office (2020). https://doi.org/10.2759/002360 . https://data.europa.eu/doi/10.2759/002360
  * European Commission and Technology [2019] European Commission, C. Directorate-General for Communications Networks, Technology: Ethics Guidelines for Trustworthy Artificial Intelligence. Publications Office (2019). https://doi.org/10.2759/346720 . https://data.europa.eu/doi/10.2759/346720
  * Commission [2021] Commission, E.: Proposal for a regulation of the European parliament and of the council: laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206
  * Suresh and Guttag [2021] Suresh, H., Guttag, J.V.: A framework for understanding sources of harm throughout the machine learning life cycle. In: EAAMO 2021: ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, Virtual Event, USA, October 5 - 9, 2021, pp. 17–1179. ACM, New York, NY, USA (2021). https://doi.org/10.1145/3465416.3483305 . https://doi.org/10.1145/3465416.3483305
  * Lee et al. [2019] Lee, M.K., Kusbit, D., Kahng, A., Kim, J.T., Yuan, X., Chan, A., See, D., Noothigattu, R., Lee, S., Psomas, A., et al.: Webuildai: Participatory framework for algorithmic governance. Proceedings of the ACM on Human-Computer Interaction 3(CSCW), 1–35 (2019)
  * Amershi et al. [2019] Amershi, S., Begel, A., Bird, C., DeLine, R., Gall, H., Kamar, E., Nagappan, N., Nushi, B., Zimmermann, T.: Software engineering for machine learning: A case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300 (2019). IEEE
  * Haakman et al. [2020] Haakman, M., Cruz, L., Huijgens, H., Deursen, A.: Ai lifecycle models need to be revised. an exploratory study in fintech. arXiv preprint arXiv:2010.02716 (2020)
  * Barocas et al. [2019] Barocas, S., Hardt, M., Narayanan, A.: Fairness and Machine Learning: Limitations and Opportunities. The MIT Press, Cambridge, Massachusetts (2019). http://www.fairmlbook.org
  * Saleiro et al. [2018] Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K.T., Ghani, R.: Aequitas: A Bias and Fairness Audit Toolkit. arXiv (2018). https://doi.org/10.48550/ARXIV.1811.05577 . https://arxiv.org/abs/1811.05577
  * Stapleton et al. [2022] Stapleton, L., Saxena, D., Kawakami, A., Nguyen, T., Ammitzbøll Flügge, A., Eslami, M., Holten Møller, N., Lee, M.K., Guha, S., Holstein, K., et al.: Who has an interest in “public interest technology”?: Critical questions for working with local governments & impacted communities. In: Companion Publication of the 2022 Conference on Computer Supported Cooperative Work and Social Computing, pp. 282–286 (2022)
  * Filgueiras [2022] Filgueiras, F.: New pythias of public administration: ambiguity and choice in ai systems as challenges for governance. Ai & Society 37(4), 1473–1486 (2022)
  * Madaio et al. [2022] Madaio, M., Egede, L., Subramonyam, H., Wortman Vaughan, J., Wallach, H.: Assessing the fairness of ai systems: Ai practitioners’ processes, challenges, and needs for support. Proceedings of the ACM on Human-Computer Interaction 6(CSCW1), 1–26 (2022)
  * Fest et al. [2022] Fest, I., Wieringa, M., Wagner, B.: Paper vs. practice: How legal and ethical frameworks influence public sector data professionals in the netherlands. Patterns 3(10), 100604 (2022)
  * Saldaña [2013] Saldaña, J.: The Coding Manual for Qualitative Researchers. International series of monographs on physics. SAGE, California, USA (2013)
  * Ropohl [1999] Ropohl, G.: Philosophy of socio-technical systems. Society for Philosophy and Technology Quarterly Electronic Journal 4(3), 186–194 (1999)
  * Latour [1999] Latour, B.: On recalling ant. The sociological review 47(1_suppl), 15–25 (1999)
  * Latour [1992] Latour, B.: Where are the missing masses? the sociology of a few mundane artifacts. Shaping technology/building society: Studies in sociotechnical change 1, 225–258 (1992)
  * Latour [1994] Latour, B.: On technical mediation. Common knowledge 3(2) (1994)
  * Chopra and SIngh [2018] Chopra, A.K., SIngh, M.P.: Sociotechnical systems and ethics in the large. In: Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. AIES ’18, pp. 48–53. Association for Computing Machinery, New York, NY, USA (2018). https://doi.org/10.1145/3278721.3278740 . https://doi.org/10.1145/3278721.3278740
  * Dolata et al. [2022] Dolata, M., Feuerriegel, S., Schwabe, G.: A sociotechnical view of algorithmic fairness. Information Systems Journal 32(4), 754–818 (2022)
  * Slota et al. [2021] Slota, S.C., Fleischmann, K.R., Greenberg, S., Verma, N., Cummings, B., Li, L., Shenefiel, C.: Many hands make many fingers to point: challenges in creating accountable ai. AI & SOCIETY, 1–13 (2021)
  * Poel and Royakkers [2011] Poel, v.d., Royakkers: Ethics, Technology, and Engineering : an Introduction. Wiley-Blackwell, United States (2011)
  * Bovens and Zouridis [2002] Bovens, M., Zouridis, S.: From street-level to system-level bureaucracies: How information and communication technology is transforming administrative discretion and constitutional control. Public Administration Review 62(2), 174–184 (2002) https://doi.org/10.1111/0033-3352.00168 https://onlinelibrary.wiley.com/doi/pdf/10.1111/0033-3352.00168
  * Kalluri [2020] Kalluri, P.: Don’t ask if artificial intelligence is good or fair, ask how it shifts power. Nature 583(7815), 169–169 (2020) https://doi.org/10.1038/d41586-020-02003-2
  * Danaher [2016] Danaher, J.: The threat of algocracy: Reality, resistance and accommodation. Philosophy & Technology 29(3), 245–268 (2016)
  * Hickok [2022] Hickok, M.: Public procurement of artificial intelligence systems: new risks and future proofing. AI & society, 1–15 (2022)
  * Siffels et al. [2022] Siffels, L., Berg, D., Schäfer, M.T., Muis, I.: Public values and technological change: Mapping how municipalities grapple with data ethics. New Perspectives in Critical Data Studies, 243 (2022)
  * Jonk and Iren [2021] Jonk, E., Iren, D.: Governance and communication of algorithmic decision making: A case study on public sector. In: 2021 IEEE 23rd Conference on Business Informatics (CBI), vol. 1, pp. 151–160 (2021). IEEE
  * Wieringa [2020] Wieringa, M.: What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. FAT* ’20, pp. 1–18. Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3351095.3372833 . https://doi.org/10.1145/3351095.3372833
  * Bovens [2007] Bovens, M.: 182 public accountability. In: The Oxford Handbook of Public Management. Oxford University Press, Oxford, United Kingdom (2007). https://doi.org/%****␣sn-article.bbl␣Line␣575␣****10.1093/oxfordhb/9780199226443.003.0009 . https://doi.org/10.1093/oxfordhb/9780199226443.003.0009
  * Spierings and van der Waal [2020] Spierings, J., Waal, S.: Algoritme: de mens in de machine - Casusonderzoek naar de toepasbaarheid van richtlijnen voor algoritmen (2020). https://waag.org/sites/waag/files/2020-05/Casusonderzoek_Richtlijnen_Algoritme_de_mens_in_de_machine.pdf
  * Cobbe et al. [2023] Cobbe, J., Veale, M., Singh, J.: Understanding accountability in algorithmic supply chains. arXiv preprint arXiv:2304.14749 (2023)
  * Fujii [2018] Fujii, L.A.: Interviewing in Social Science Research, A Relational Approach. Routledge, New York, NY; Abingdon, Oxon (2018)
  * Goede et al. [2019] Goede, D., Bosma, Pallister-Wilkins: Secrecy and Methods in Security Research A Guide to Qualitative Fieldwork. Routledge, New York, NY; Abingdon, Oxon (2019)
  * Strauss [1987] Strauss, A.L.: Qualitative Analysis for Social Scientists. Cambridge university press, Cambridge (1987)
  * Noy and McGuinness [2001] Noy, N., McGuinness, B.: Ontology development 101: A guide to creating your first ontology. Stanford Knowledge Systems Laboratory (2001). https://protege.stanford.edu/publications/ontology_development/ontology101.pdf
  * van Hage et al. [2011] Hage, W., Malaisé, V., Segers, R., Hollink, L., Schreiber, G.: Design and use of the simple event model (sem). Web Semantics: Science, Services and Agents on the World Wide Web 9, 128–136 (2011) https://doi.org/10.1093/oxfordhb/9780199226443.003.0009
  * Golpayegani et al. [2022] Golpayegani, D., Pandit, H.J., Lewis, D.: AIRO: An Ontology for Representing AI Risks Based on the Proposed EU AI Act and ISO Risk Management Standards vol. 55, p. 51 (2022). IOS Press
  * Franklin et al. [2022] Franklin, J.S., Bhanot, K., Ghalwash, M., Bennett, K.P., McCusker, J., McGuinness, D.L.: An ontology for fairness metrics. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 265–275 (2022)
  * Tamburri et al. [2020] Tamburri, D.A., Van Den Heuvel, W.-J., Garriga, M.: Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. In: 2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), pp. 391–394 (2020). IEEE
  * Selbst et al. [2019] Selbst, A.D., Boyd, D., Friedler, S.A., Venkatasubramanian, S., Vertesi, J.: Fairness and abstraction in sociotechnical systems. In: Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 59–68 (2019)
  * Barocas et al. [2021] Barocas, S., Guo, A., Kamar, E., Krones, J., Morris, M.R., Vaughan, J.W., Wadsworth, W.D., Wallach, H.: Designing disaggregated evaluations of ai systems: Choices, considerations, and tradeoffs. In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pp. 368–378 (2021)