# Trustworthy AI in practice: an analysis of practitioners’ needs and
challenges

Maria Teresa Baldassarre  0000-0001-8589-2850 University of Bari ”A.
Moro”BariItaly<EMAIL_ADDRESS>,  Domenico Gigante
0000-0003-3589-6970 Ser&Practices SrlBariItaly<EMAIL_ADDRESS>,  Marcos
Kalinowski  0000-0003-1445-3425 Pontifical Catholic University of Rio de
Janeiro (PUC-Rio)Rio de JaneiroBrazil<EMAIL_ADDRESS>,  Azzurra
Ragone  0000-0002-3537-7663 University of Bari ”A. Moro”BariItaly
<EMAIL_ADDRESS>and  Sara Tibidò  Scuola IMT Alti Studi LuccaBariItaly
<EMAIL_ADDRESS>
(2024)

###### Abstract.

Recently, there has been growing attention on behalf of both academic and
practice communities towards the ability of Artificial Intelligence (AI)
systems to operate responsibly and ethically. As a result, a plethora of
frameworks and guidelines have appeared to support practitioners in
implementing Trustworthy AI applications (TAI). However, little research has
been done to investigate whether such frameworks are being used and how. In
this work, we study the vision AI practitioners have on TAI principles, how
they address them, and what they would like to have – in terms of tools,
knowledge, or guidelines – when they attempt to incorporate such principles
into the systems they develop. Through a survey and semi-structured
interviews, we systematically investigated practitioners’ challenges and needs
in developing TAI systems. Based on these practical findings, we highlight
recommendations to help AI practitioners develop Trustworthy AI applications.

Artificial Intelligence, Software Engineering, Trustworthy AI, Mixed-method
Research, Systematic Investigation, Survey, Semi-structured Interview

††journalyear: 2024††copyright: acmlicensed††conference: 28th International
Conference on Evaluation and Assessment in Software Engineering; June 18–21,
2024; Salerno, Italy††booktitle: 28th International Conference on Evaluation
and Assessment in Software Engineering (EASE 2024), June 18–21, 2024, Salerno,
Italy††doi: 10.1145/3661167.3661214††conference: The 28th International
Conference on Evaluation and Assessment in Software Engineering; 18–21 June,
2024; Salerno, Italy††price: 15.00††isbn: 979-8-4007-1701-7/24/06††ccs:
Computing methodologies Artificial intelligence††ccs: Software and its
engineering Software creation and management

##  1\. Introduction

Artificial intelligence (AI) systems increasingly exert an extensive impact on
various facets of our existence, encompassing the realm of healthcare and the
quality of education we receive (Bosch et al., 2016; Esteva et al., 2017;
Holstein et al., 2018), the determination of which news articles or social
media posts we encounter (Alvarado and Wærn, 2018; Bucher, 2017; Rader and
Gray, 2015), the allocation of employment opportunities (HireVue.com, 2019;
pymetrics.ai, 2018), the detention decisions (Chouldechova, 2016), and the
intensification of policing efforts in some areas (Lum and Isaac, 2016; Veale
et al., 2018), just to name a few. With this expansion, the risk of AI
increasing social inequities has generated escalating attention across several
communities, including the media. Indeed it is common to observe reports in
mainstream media of systemic dangerous behaviors observed in widely used AI
systems, such as a smart algorithm guiding assistance for tens of millions of
people biased against dark-skinned
patients111https://www.nature.com/articles/d41586-019-03228-6, or an AI
chatbot suspended for making homophobic slurs and leaking user
information222https://www.ic3.gov/Media/News/2021/210310-2.pdf. These risks
are even more pronounced with the recent advent of Generative AI and the
impact these systems have on various societal aspects (Baldassarre et al.,
2023).

In this context, the concept of Trustworthy Artificial Intelligence (TAI) has
been defined: ”Trustworthy AI has three components, which should be met
throughout the system’s entire life cycle: (1) it should be lawful, complying
with all applicable laws and regulations (2) it should be ethical, ensuring
adherence to ethical principles and values and (3) it should be robust, both
from a technical and social perspective since, even with good intentions, AI
systems can cause unintentional harm. Each component in itself is necessary
but not sufficient for the achievement of Trustworthy AI” (High-Level Expert
Group on AI (2018a), AIHLEG).

Several public and private organizations have responded to these societal
fears by developing different kinds of resources: ethical requirements,
principles, guidelines, best practices, tools, and frameworks (Barletta et
al., 2023; Pardo et al., 2011). As the field progresses, integrated toolkits
are being developed with the objective of rendering these methods more broadly
accessible and usable (e.g., Enisa’s Machine Learning Security (enisaml,
2021), Aequitas (aequitas.dssg.io, 2018) or Google’s What-if-Tool (whatiftool,
2018)).

Despite growth in the development and dissemination of toolkits, there has
been little research investigating how AI practitioners actually use them in
practice throughout the entire Software Development Life Cycle (SDLC). A few
studies have already explored practitioners’ perceptions and desires around
open-source toolkits but are focused only on the Fairness aspect of TAI (Lee
and Singh, 2021; Richardson et al., 2021). However, neither of these two works
investigated what professionals need during the entire SDLC.

Our research builds on the results of our previous mapping study (Barletta et
al., 2023) and aims to investigate how AI professionals deal with TAI issues
on a day-to-day basis. To better understand practitioners’ needs we
distributed a survey and conducted semi-structured one-on-one interviews,
collecting data from a total of 34 practitioners employed in companies of
different sizes.

As the main contribution, this work intends to deeply investigate
practitioners’ views, needs, and challenges in developing TAI systems
throughout the entire SDLC. The novel contribution can be summarized as
follows:

  * •

We have analyzed the existing procedures that development teams adopt when
implementing Trustworthiness in AI;

  * •

We have investigated the impediments encountered in the attempt to implement
Trustworthiness in AI;

  * •

We have identified a range of practitioners’ needs that academic and
industrial research should seek to address.

Our study has identified a range of practitioner needs that have thus far been
overlooked in the literature. For example, the majority of our interviewees
report a lack of tools for the late stages of the SDLC, as well as of
knowledge bases and practical guidelines with suggestions on implementing TAI
across the entire SDLC.

The paper is organized as follows. Section 2 provides some background
definitions. Section 3 describes the methodology we adopted to conduct this
study, including research questions and how we collected, extracted, and
analyzed the data. Section 4 presents the quantitative and qualitative results
together with some preliminary findings. Section 5 discusses the results, and
practical implications and recommendations for the AI industry and research
community. Section 6 addresses threats to validity, followed by the conclusion
in Section 7.

##  2\. Background

###  2.1. AI Principles proliferation

National and global entities have established specialized expert committees in
the field of Artificial Intelligence (AI) to address the associated risks
stemming from AI development. These committees often have the mandate of
formulating policy documents. Prominent examples of such organizations include
the High-Level Expert Group on Artificial Intelligence initiated by the
European Commission (High-Level Expert Group on AI (2018b), AIHLEG), the
UNESCO Ad Hoc Expert Group (AHEG) tasked with the Recommendation on the Ethics
of Artificial Intelligence (UNESCO Ad Hoc Expert Group (2021), AHEG), the
Advisory Council on the Ethical Use of Artificial Intelligence and Data in
Singapore (Holborn Law LLC, 2018), the NASA Artificial Intelligence Group
(NASA Artificial Intelligence Group, 2018) and the UK AI Council (UK AI
Council, 2019), among others.

These committees bear the crucial role of generating comprehensive reports and
guidelines about Trustworthy AI (TAI). A parallel endeavour is observable
within the commercial landscape, particularly among enterprises heavily
reliant on AI technologies. Corporations such as
Sony333https://www.sony.com/en/SonyInfo/sony_ai/responsible_ai.html and
Meta444https://ai.facebook.com/blog/facebooks-five-pillars-of-responsible-ai/
have made their AI policies and principles publicly accessible. Concurrently,
professional organizations and non-profit entities, such as UNI Global
Union555http://www.thefutureworldofwork.org/media/35420/uni_ethical_ai.pdf and
the Internet
Society666https://www.internetsociety.org/resources/doc/2017/artificial-
intelligence-and-machine-learning-policy-paper/, have issued statements and
recommendations.

The substantial efforts of this diverse spectrum of stakeholders in crafting
TAI principles and policies not only underscore the imperative need for
ethical guidance but also exemplify their vested interest in shaping AI ethics
to align with their specific objectives (Greene et al., 2019). It is
noteworthy that the private sector’s engagement in the gap of AI ethics has
undergone a thorough check, with contentions suggesting that high-level soft
policies may be employed to either transform a social issue into a purely
technical one (Greene et al., 2019) or to potentially circumvent regulatory
measures (bay, 2018; Jobin et al., 2019).

Nevertheless, a set of research endeavors has brought attention to the
divergent nature of these proposals, giving rise to a complex challenge often
referred to as ”principle proliferation”777It is ”the proliferation of soft-
law efforts”. Jobin’s analysis (Jobin et al., 2019) shows the emergence of an
apparent cross-stakeholder convergence on promoting the ethical principles […]
[but] unclarity remains as to which ethical principles should be prioritized,
how conflicts between ethical principles should be resolved, who should
enforce ethical oversight on AI and how researchers and institutions can
comply with the resulting guidelines. (Jobin et al., 2019). Consequently,
efforts have been undertaken to address this challenge. For instance, Jobin et
al. (Jobin et al., 2019) conducted a comprehensive study, that culminated in
the identification of a global convergence around five ethical principles:
transparency, justice and fairness, non-maleficence, responsibility, and
privacy. Jobin et al. (Jobin et al., 2019) also observed that, while no single
document they reviewed encompassed all of these ethical principles, these five
principles were mentioned in over half of the sources examined. Furthermore,
their detailed thematic analysis unveiled significant semantic and conceptual
variations in the interpretation of these principles and the specific
recommendations or areas of concern derived from each one.

###  2.2. Trustworthy AI principles definitions

As set out in Section 2.1, a notable degree of ambiguity and subtlety exists
in demarcating the principles that predominantly characterize Trustworthy AI
(TAI). Notably, TAI is sometimes used interchangeably with Responsible or
Ethical AI. In our investigation, we confront the challenge of principle
proliferation by choosing to focus on a specific subset of principles that
characterize TAI. Specifically, we concentrate on the most recurrent four
principles identified by Jobin et al. (Jobin et al., 2019), while opting to
exclude the principle of responsibility due to its infrequent occurrence and
lack of a clear, universally accepted definition.

Furthermore, in this work, we have decided to adopt the definitions put forth
by the High-Level Expert Group on Artificial Intelligence (AIHLEG) — an entity
established by the European Commission — explained in their ”Ethics guidelines
for trustworthy AI” (High-Level Expert Group on AI (2018a), AIHLEG).

Given these premises, we mapped the four selected principles identified by
Jobin et al. (Jobin et al., 2019), with the formal definition delineated by
the AIHLEG (High-Level Expert Group on AI (2018a), AIHLEG). The mapping has
been carried out based on the contents of the definitions and not merely on
the nomenclatures, as, in most cases, they differ. For the sake of simplicity,
we have shortened and labeled each TAI principle as follows: Transparency,
Fairness, Security, and Privacy. We will use these labels throughout the
paper. Definitions of the terms adopted for each principle are provided in our
online appendix (Baldassarre et al., 2024a).

##  3\. Method

The goal of our study is to investigate the state of the practice to
understand common practices as well as challenges and difficulties encountered
by AI practitioners in implementing TAI systems through the entire SDLC.

Since traditional phases of the SDLC do not necessarily map well with the
activities required to develop an AI-enabled system, we have extended each
phase with one or more activities mentioned by Zhengxin et al. (Zhengxin et
al., 2023). Table 1 graphically shows how each traditional SDLC phase’s
definition has been extended.

Table 1. Activities integrated into the traditional SDLC phases to support AI-enabled systems development. Traditional SDLC phase | Integrated activity
---|---
Requirements Elicitation | Model Requirement
Design |  | Data Collection,
---
Data Preparation
Development |  | Feature Engineering,
---
Model Training
Test | Model Evaluation
Deployment | Model Deployment
Monitoring | Model Monitoring

Given our main goal, and based on the analysis of the shortcomings and common
practices that emerged from the mapping study carried out in our previous work
(Barletta et al., 2023), we defined the following Research Questions (RQs):

  * •

RQ1: What vision/opinion do practitioners have about untrustworthiness issues
and how often do they encounter them?

  * •

RQ2: How do practitioners address untrustworthiness issues?

  * •

RQ3: What tools/support do practitioners desire to address untrustworthiness
problems better?

###  3.1. Data collection

For our study, we recruited practitioners888By ”AI practitioners” we mean
those who work in any role on a team developing products or services involving
AI. working on AI products and services through a combination of purposive and
snowball sampling (Heckathorn, 2011). To recruit participants for our study,
we chose to not widely distribute our survey online, but rather to conduct a
brief screening procedure to define the inclusion criteria and target suitable
participants for the study. The target participants for the survey were AI
practitioners involved in the development of AI-based systems, with at least
some basic knowledge of TAI principles and/or who had previously addressed TAI
in their professional work. We started by sending personal emails to contacts
within our network, working in industry or academia, explaining the purpose of
the study and the inclusion criteria, we asked them to help us recruit other
participants by spreading the invitation through their networks. Next, we
verified that the inclusion criteria were met by asking some specific
questions in the demographics section of the survey.

The invitation to participate in the study was sent by email and included an
explanation of the study’s purpose. In order to meet all needs (e.g.,
restrictions related to tight schedules, time zones, commitments), we gave
participants the chance to choose between (a) survey (asynchronous
interaction) or (b) semi-structured interviews (synchronous interaction) mode.
This allowed us to include a broader amount of subjects and collect a higher
number of answers. All participants answered the same set of questions and had
the opportunity to add any non-listed practices or suggestions/feedback
related to the closed-ended question in the open-text fields (asynchronous
mode) whereas, interviewed participants (synchronous) could “discuss their
answers out loud” and further elaborate their considerations with the
interviewers. The answers were all transcribed to be included later in the
thematic analysis.

Participation was on a voluntary basis and not rewarded by any means. The
survey can be accessed at (Baldassarre et al., 2024b). Overall, we obtained 23
answers for the survey and 11 participants attended the interview.

Section 4.5 provides details about participants’ demographics and their
relevant experience. Specific details about their companies and working
environment have been abstracted to preserve anonymity.

###  3.2. The survey

The survey contains six main sections (Baldassarre et al., 2024b):

1\. Informed consent request. This page asks the participants to provide their
informed consent and explains the purpose of the research, the participants’
requirements, confidentiality rules, participation on a voluntary basis, and
the time needed to complete the survey.

2\. Preliminary concepts knowledge. We clarified the semantics and
interpretation of each TAI principle for participants by providing a
definition for each principle. By listing a definition, we wanted to build a
shared understanding of each principle to answer the remaining questions. In
addition, we asked the participants about their vision and previous experience
with TAI.

3\. Practices in preventing untrustworthiness in AI. We inquired participants
about the main strategies they adopt to prevent TAI issues (e.g., balancing
the dataset or choosing a specific algorithm).

4\. Practices in discovering untrustworthiness issues in AI. We asked
participants to share their strategies to find possible sources of
untrustworthiness (e.g., do auditing tasks, compute metrics, learn from user
feedback).

5\. Practices in addressing trustworthiness issues. We investigated the
different approaches used to address TAI issues (e.g., dataset augmentation,
instance weighting).

6\. Demographics and background information. Participants were asked about
gender, level of education, country, role, years of experience, size of the
organization, etc.

All the questions of the survey sections (3), (4), (5), illustrated above, as
well as the options for the answers, were inspired from and based on our
previous work (Barletta et al., 2023), which set current shortcomings and
common practices in literature.

The survey was anonymous and did not ask for any directly identifying
information. Most of the survey questions were closed-answer and mandatory,
but there were also optional open-text ones. Through the latter, we were able
to collect further qualitative data as many of the participants provided
information on the practices they usually implement. All survey data and raw
material can be accessed in the online appendix (Baldassarre et al., 2024a).

###  3.3. Interview Study Protocol

Before scheduling the interviews with participants, to understand the
challenges and requirements for conducting remotely semi-structured
interviews, as well as to help us refine our study protocol, we conducted two
pilot interviews. These preliminary interviews helped us define the protocol
and the setting we applied to the final, larger sample.

All interviews were conducted via Microsoft
Teams999https://www.microsoft.com/it-it/microsoft-teams; after asking each
participant for their consent, we enabled Recording & Transcription Teams
features. Once the interview ended, we proof-checked the transcription in
order to correct any misspellings, anonymized any Personal Identifiable
Information (PII) and finally deleted the recording.

The interview study consisted of think-aloud semi-structured interviews, each
one lasting between 45 and 90 minutes. During the live interview, we
periodically asked participants to elaborate on their responses, especially
for the open ones. We also encouraged participants to ”think aloud” (Kupis et
al., 2019; Someren et al., 1994) and discuss the information that was being
displayed and how their understanding of the question was developing.

To give a standard structure to each interview, we used the survey as a
canvas.

###  3.4. Data Analysis

In this step, we extracted all relevant data using quantitative and
qualitative data analysis techniques to summarize and interpret the collected
data. For quantitative data, we used descriptive statistics (George and
Mallery, 2018), and for qualitative data, we used thematic analysis (Cruzes
and Dybå, 2011).

We used an inductive thematic analysis approach (Braun and Clarke, 2006, 2012)
to analyze about 11.5 hours of video recordings and their corresponding
(automatically generated and manually proof-checked) transcripts. The entire
analysis was done through Atlas.ti101010https://atlasti.com/. Two authors
worked independently and used the tool to conduct an open coding of the
transcripts for each quotation. Next, they manually reviewed each code and
decided which to include/exclude annotating any comments. Once this step was
completed, they joined to compare and discuss results. The total number of
analyzed quotations was 23. The calculated Cohen’s Kappa (Bakeman and Quera,
2011) is 0.259. All the details about the coding procedure and the generated
codes are provided in the online appendix (Baldassarre et al., 2024a).

Once finalized, the codes were shared with the entire research team and
grouped into higher-level themes concerning the practitioners’ knowledge and
practices. In Section 4 we discuss the findings identified from these codes
and themes, together with implications for future TAI developments.

##  4\. Results and findings

We present findings from our think-aloud interviews study and the survey
answers, divided into three main sections,

  * •

Practices in preventing untrustworthiness in AI

  * •

Practices in discovering untrustworthiness issues

  * •

Practices in addressing untrustworthiness issues

Across all three phases, we discovered different nuances of practitioners’
needs around TAI issues. We supplement data from the closed-ended questions
(quantitative results) with the thematic analysis performed on the answers
from all the open-ended questions (qualitative results). We performed analysis
on the disaggregated data with respect to subgroups such as company size,
gender, education, and number of projects deployed. To understand if the
differences were statistically significant, we conducted pairwise comparisons
using Fisher’s exact test coupled with the Benjamini-Hochberg (Benjamini and
Hochberg, 1995) correction to obtain the adjusted p-values. Since there is no
statistical significance in any of the cases except for company size, detailed
in Section 4.2, the graphs in the paper report the results of the analyses in
aggregated form.

###  4.1. Preliminary concepts knowledge

All discussed tables and graphs, from now on, bring together the answers from
both the interviews (11) and the survey (23). In the survey section ”2.
Preliminary concepts knowledge”, we observed that the TAI principle
participants have encountered most frequently in their projects is Privacy (20
answers), followed by Transparency (18 answers) and Security (17 answers),
while the least experienced is Fairness (13 answers). This answer should be
further investigated because perhaps sometimes practitioners may not recognize
or be aware of the need to address some issues related to these principles.

In addition, Table 2 shows the reasons participants agreed on concerning why
they care about TAI. The most agreed reasons were Avoid violating legal
requirements and Improve the overall quality. While, the least agreed one was
Retain users/avoid losing the activity, with eight disagreements.

Table 2. Reasons why participants care about TAI principles. Reason |  | Disagreement
---
(1-2)
| Neutral
---
(3)
| Agreement
---
(4-5)
N/A
| Doing something
---
about trustworthiness
in AI/ML
4 | 6 | 23 | 1
| Avoid violating
---
legal requirements
2 | 3 | 27 | 2
| Avoid reputational
---
damages
4 | 8 | 21 | 1
| Improve the overall
---
quality
3 | 3 | 26 | 2
| Retain users/avoid
---
losing business
8 | 4 | 19 | 3

Other important factors related to the reasons for caring about TAI emerged:
i) ”need to solve mission-critical tasks”; ii) ”[need to provide] models
usable in real-world contexts”, and this demonstrated that black box models
are not allowed in some specific contexts; iii) ”the robustness of the AI
explanations themselves”, which shows consciousness about the fact that all
TAI aspects contribute to making the model more robust; and iv) ”[need to]
desire to commercially assemble AI systems to improve society”.

Finally, Fig. 1 shows that most of the participants address TAI principles
during the Design and Development SDLC phases. In contrast, very few
participants reported that they had addressed TAI principles during the
Requirements Elicitation and, especially, Deploy phases.

Figure 1. TAI principles addressed by SDLC phase.

###  4.2. Practices in Preventing untrustworthiness in AI

In the survey section ”3.Practices in preventing untrustworthiness in AI”, as
Fig. 2 shows, the most recurrent strategy employed by our participants to
ensure trustworthiness is ”algorithm that can best explain the decision”. On
the other hand, the least employed practice appears to be ”inject malicious
data points”. In analyzing the disaggregated data, we found a statistically
significant difference only in the responses related to the strategy
“Algorithm that can best explain the decision” for the company size subgroup.
Indeed, for medium-sized companies, we found more positive responses than for
small and large enterprises. For medium ones, no negative answers were given
and 63% of the participants chose ”Always”, reflecting the wide use of this
strategy in medium-sized companies.

Figure 2. Strategies employed to ensure trustworthiness in AI. N/A answers
have been removed.

Some participants mentioned other strategies, such as ”[conduct an] in-depth
study of the state of the art [prior to start designing the system]” and ”[use
the] post-processing phase […] to apply human-friendly deterministic rules to
check whether a result is in line with the sense of the application domain”.

Additionally, when we asked the participants to rate the utility of various
hypothetical tools assuming their team had access to them, the participants
rated as the most valuable the tool able to ”[…] generate an explanation of a
model after its creation […]”. On the other hand, they rated as least useful
the tool to ”decide how much data you need for particular
subgroups/subpopulations”. These results are shown in Fig. 3.

Figure 3. Perceived usefulness of hypothetical tools to prevent TAI issues.
N/A answers have been removed.

Here too, some important insights emerged, including ”tools to improve
software architectures” and ”[a tool for] referencing the best architecture
(dependencies, docker files, instant compute, …) to perform the task with the
lowest possible costs”.

The answers to this section show practitioners are prone to use techniques and
tools to prevent trustworthiness issues, focusing mainly on ensuring
Transparency (a.k.a. Explainability).

###  4.3. Practices in Discovering Untrustworthiness Issues

In the survey section ”4.Practices in Discovering untrustworthiness in AI”, we
investigated which strategies participants mainly employed to discover TAI
issues.

The data shows that the most used strategies are ”Metrics/KPIs”, ”learn from
user feedback”, and ”examine AI/ML model’s input features” (see Fig.4).
Examples of Metrics/KPIs related to fairness are, just to cite a few,
Demographic Parity, Accuracy, F1-Score. Whereas, user feedback is intended as
having a feedback form where the users can report misbehavior by the algorithm
(Gemini Mistake, 2024). What stands out is that the strategy less employed by
the participants is ”generate specific adversarial/malicious samples” (4/10
negative answers).

Figure 4. Strategies employed to discover untrustworthiness in AI.

Moreover, when asked for other (not mentioned) strategies they employ, one
participant answered ”post processing studies to evaluate possible model
’discriminations’”. As a free thought, another declared ”apply confidence
criteria such that it is possible to measure how often the model fails to
respond reliably”.

What emerges in this section is that participants employ both qualitative
(e.g., auditing tasks in the human labeling/scoring process) and quantitative
(e.g., metrics and KPIs) strategies.

###  4.4. Practices in Addressing Untrustworthiness Issues

Regarding survey section ”5.Practices in Addressing Untrustworthiness in AI”,
participants reported that after finding a TAI issue only in 35% of the cases
(12/34) the team addressed it directly, while 15% (5/34) of the participants
stated that it was not addressed by them, but handled by a third party. Worth
noting is the fact that in 50% (17/34) of the cases participants reported that
they did not fix the issue after finding it. The reasons why participants did
not solve the issue after finding it are asked in a subsequent question (see
Table 3).

When participants addressed any issues found, they declared the most
implemented strategies were ”improving the quality of the dataset (e.g.,
removing spurious samples, paradoxical values)” (8 answers) and ”augmenting
the dataset (e.g., with artiﬁcial, manually generated data points)” (6
answers). On the other hand, the less implemented strategy was ”searching for
a tool which automates a specific trustworthiness issue-fixing process” (1
answer). One interviewee also mentioned that they usually approach
explainability by ”using [only] white box models”. More details can be found
in Figure A2 in the online appendix (Baldassarre et al., 2024a).

Regarding the reasons why participants did not solve a TAI issue after finding
it, Table 3 shows the most frequent reasons are ”the issue solution required
too much time to be implemented” (58.3%) and ”the issue solution was likely to
decrease the performance of the system (e.g., decreasing accuracy)” (50%). On
the other hand, none of the participants answered: ”no one had an idea on how
to solve the issue”; this is a positive result since demonstrates
practitioners are conscious of untrustworthiness problems and can formulate
hypotheses on how to address them. During the interviews, one participant also
mentioned ”data availability” as an impediment.

Table 3. Reasons which prevented participants from addressing/ﬁxing AI trustworthiness issues.  Impediment | Yes | No | N/A
---|---|---|---
No one had idea on how to solve the issue | 0 | 9 | 3
| The issue solution required high human effort,
---
which we could not afford
5 | 5 | 2
| The issue solution was too expensive
---
(ﬁnancially) to address
4 | 5 | 3
| The issue solution required too much time to
---
be implemented
7 | 3 | 2
| The issue solution was likely to decrease the
---
performance of the system
(e.g., decreasing accuracy)
6 | 3 | 3
| There was not a tool which automated the
---
fixing process
5 | 5 | 2

Finally, when we again asked the participants to rate the utility of various
hypothetical tools — assuming their team had access to them — the participants
rated as the most valuable a tool able to ”[…] help […] monitoring the AI
model after its release to the public”, followed by ”best practices that can
actively guide your team through the model’s SDLC”, ”tools to help the team in
the data pre-processing steps (e.g., decide whether one needs to add/remove
data points from your training set, and what kind of data you need to
add/remove)”, and ”a knowledge book in which are mapped trustworthiness
problems and […] solutions”. On the other hand, they rated as least useful
tools ”[…] to help your team doing an ex-post TAI audit” and tools able to […]
help your team deciding which AI model best respects the TAI principles […].
These results are graphically shown in Fig. 5.

Figure 5. Perceived usefulness of hypothetical tools to address TAI issues.
N/A answers have been removed.

###  4.5. Demographics and background information

In terms of demographics of the sample, the study participants’ gender is
represented by 68% male, 21% female, 3% non-binary or gender diverse, and 9%
preferred not to respond. Concerning academic qualifications, 56% of the
participants have a master’s degree, 41% have earned a Ph.D., and 3% have
completed their bachelor’s studies in computer science-related fields. The
notable prevalence of advanced education within the respondent pool aligns
with our expectations, reflecting the elevated cognitive expertise required by
the complexities of this particular field.

The vast majority of the participants are employed in medium-large companies,
specifically, 32.3% (11/34) work for companies with less than 50 employees
(small), 23.5%, (8/34) for companies between 50 and 500 employees (medium),
while 44.1%, (15/34) are employed in large companies with more than 500
employees.

Participants have on average five years of experience in their role and two
years of experience in the AI field.

Regarding the technology area that best describes with which AI
products/services the participants work, the four most prevalent are ”Decision
support” (15 answers), ”Natural Language Processing” (13 answers), ”Computer
Vision / Image Analysis” (12 answers), and ”Recommender Systems” (9 answers).

Finally, regarding the number of AI-enabled projects developed and deployed
into a production environment, we observed that most of the participants
(19/34, 56%) declared that just a small percentage of the developed projects —
from 1 to 30% — are deployed in a production environment, while only 3% of the
participants declared that most of the projects — from 90 to 100% — are
deployed into a production environment. This reveals the fact that most of
these types of projects are still in an experimental stage.

Due to space constraints, we have not included tabular representation of
demographics in the paper which are, however, all available in the online
appendix (Baldassarre et al., 2024a)111111Appendix Table A4 (interviewees’
self-reported technology areas and team roles); Table A5 (participants count
grouped by company size); Table A6 (participants count grouped by years of
experience in their current role and in developing AI-enabled systems).

###  4.6. Summary of key findings

Here we summarize some key findings from our study.

F1. The study reveals that participants care a lot about Privacy and
Transparency. Indeed, among the most used strategies to ensure trustworthiness
are ”post-hoc explainability” and ”algorithm that can best explain the
decision” (Fig. 2). In addition, tools that ”generate an explanation” and that
help in deciding ”the more clear and explainable model” are among the tools
perceived as most useful (Fig. 3).

F2. Acting on the dataset is one of the most used strategies to solve the
found TAI issues. Indeed the most implemented strategies are ”improving the
quality of the dataset (e.g., removing spurious samples, paradoxical values)”
and ”augmenting the dataset (e.g., with artiﬁcial, manually generated data
points)”.

F3. Business constraints — like the time required to implement the solution or
the unacceptable performance drop — often represent impediments to
implementing trustworthy AI applications (see Table 3).

F4. Analyzing the tools practitioners lack the most in addressing TAI issues,
a need for tools for after-deploy monitoring and best practices and TAI
knowledge books that can actively guide a team through the SDLC emerges (Fig.
5).

F5. Many TAI projects are developed but not deployed in a production
environment, which reveals that in some cases practitioners are still
experimenting with this field.

##  5\. Discussion

RQ1. What vision/opinion do practitioners have about untrustworthiness
problems and how often do they address them?

Our findings reveal that the most addressed principle is Privacy, probably
because it is contained in various regulations that exist and must necessarily
be complied with (e.g., in Europe the GDPR (European Parliament and Council of
the European Union, 5 04)). Transparency is also often taken into
consideration, probably because there are domains where it is a fundamental
and unavoidable feature required by the law, such as in Healthcare and
Financial Services. On the other hand, the one less addressed is Fairness,
perhaps because there are still no clear and shared regulations for this
dimension of trustworthiness and everything is left to the initiative and
ethical values of those implementing these systems. As a result, even large
and well-established companies in the industry are often caught up in scandals
that damage their reputation and show how even the most popular and widely
used algorithms suffer from
unfairness121212https://www.bloomberg.com/graphics/2023-generative-ai-bias/.

It is notable that when we asked practitioners why their team cares about
trustworthiness in AI, the motivation ”retain users/avoid losing business”
found most disagreement among them. This may reveal that they believe that TAI
issues do not lead to losing users and/or business. Moreover, during
interviews, it emerged that addressing TAI is, in some cases, even mandatory
and not an option.

Finally, our study suggests that TAI is mainly addressed in the early stages
of the SDLC. While this is good — since the earlier certain decisions are
made, the more effective they are in the design of the final model — this also
reveals that practitioners are most likely not aware of tools and best
practices to be used in the final stages of the lifecycle. In fact, Deploy is
one of the least addressed phases. Indeed, one interviewee also mentioned the
need for guidelines on the best cloud provider compliant with TAI practices.
These elements allow us to infer that the choice of deployment infrastructure
is often left to chance or, in the best case, to routines and/or trust in a
specific cloud provider.

RQ2. What do practitioners do to address untrustworthiness problems?

Based on the answers, it is clear that when practitioners want to solve TAI
issues they mainly act on the dataset, as the most implemented strategies are
related to ”improving the quality of the dataset” and ”augmenting it with
artificial data points”. As shown by Fig. 1 and Fig. 4, TAI is mainly
addressed in Design (e.g., examining model’s input features) and Development
phase, without disregarding the Monitoring one (e.g., learning from user
feedback). Answers to question 22131313Q22: Which of the following strategies
has your team evaluated, and which strategies were actually implemented?
sustain this trend and highlight that practitioners also give little
consideration to the possibility of searching for tools that automate their
manual activities, perhaps because they feel safer with manual analyses or
because they are not familiar with automatic tools. Finally, answers to
question 24141414Q24: What prevented your team from addressing/ﬁxing these AI
trustworthiness issues? demonstrate that business constraints (e.g., time,
money) often hinder the resolution of TAI issues: if solving a problem takes
too long or may cause a slight performance degradation, practitioners tend to
not address it.

RQ3. What do practitioners desire to better address untrustworthiness
problems?

What emerged from the data is that the tools most in demand are those that can
generate an explanation of the model after its training. However, literature
provides several mature models to explain both traditional ML algorithms — see
SHAP (Lundberg and Lee, 2017) and LIME (Ribeiro et al., 2016) — and neural
networks — such as (Kenny et al., 2021). Perhaps these answers could be due
either to the unsuitability of such tools for specific issues to be solved or
to scarce knowledge about explainability tools landscape, which would also
explain the fact that post-hoc explainability strategies are infrequently used
(see Fig. 2). On the contrary, the tools deemed less useful are the ones that
help decide how much data they need for particular subgroups/subpopulations,
probably because there are other factors — not related to TAI requirements —
that constrain practitioners while collecting and/or pre-processing the data.
For instance, sometimes, collecting more data about a specific sub-group could
be simply infeasible or very difficult.

Furthermore, it is evident that practitioners feel the need for tools to
monitor the model after it has been released to the public, they especially
express a need for guidelines and a knowledge base to help them in
implementing TAI throughout the SDLC. Moreover, practitioners believe a single
tool to perform post-hoc analyses to be of little use, probably because they
feel that it is too late to worry about TAI once the system has been released
on the market to end users.

###  5.1. Practical implications and recommendations

Based on the previous discussion, we summarize some key practical implications
and recommendations for the AI industry and the AI research community.

P1. Practitioners should take TAI principles into account throughout the
entire SDLC and not just in the early stages, such as the Design phase, as
shown by Fig. 1 and answers to question 18151515Q18: Please indicate if you
use one of the following strategies to discover trustworthiness issues.. Even
if valuable mitigation measures can be put in place early in SDLC, there is a
huge amount of mitigations that can only be implemented in the latter stages
of the SDLC. For instance, using a tool that helps to choose the best cloud
provider to be TAI-compliant.

P2. Our study reveals that, on numerous occasions, even when TAI issues are
identified, they are not addressed by practitioners — either directly or by
third parties — due to business constraints such as limited time, financial
constraints, or declining performance. Nevertheless, the oversight of these
issues may result in significant economic and reputational consequences,
incurring substantial costs for companies. Hence, it is imperative for the
industry to commit to addressing TAI issues detected at various stages of the
SDLC. In addition, such actions are crucial for compliance with emerging
regulations, such as the AI Act (European Union, 2023).

P3. As a general remark, there is a pressing need for guidelines, knowledge
bases, and tools that can help practitioners implement TAI principles
throughout the entire SDLC. They need guidance and practical advice on which
tools to use at each stage of SDLC and to address which principles. Often, as
pointed out in our study, although these tools exist, practitioners may not be
aware of them. Moreover, as pointed out in our previous work (Barletta et al.,
2023), there is a significant gap that should be filled between high-level AI
ethics principles and low-level concrete practices for practitioners. For this
reason, as a research community, we should rethink how to design these
guidelines and best practices, so that they are readily available and usable
by professionals and provide actionable guidelines that can be put into
practice while implementing trustworthy AI applications.

##  6\. Threats to validity

In this section, we discuss the threats to the validity (Wohlin et al., 2012)
of our study. We delineate the threats to validity and constraints on the
outcomes of our study arising from the research methodology we employed.

Construct validity The thematic analysis was executed by two researchers,
introducing the potential for subjective judgment. To address this concern, we
implemented the negotiated agreement technique (Spadini et al., 2018) between
the first and second researcher, fostering consensus, which was achieved after
a careful examination of 25 comments.

Internal validity. Threats to internal validity pertain to unconsidered
factors that might impact the variables and relationships under scrutiny. In
our investigation, we conducted interviews with AI practitioners to gain
insights into their perspectives on Trustworthy AI (TAI) issues. Each
practitioner possesses a distinct background and ethical standpoint,
potentially diverging from the practices of their peers. We sought to mitigate
this issue by interviewing practitioners from diverse companies and different
countries. Furthermore, interviewees’ viewpoints may be influenced by
additional factors, such as existing literature on TAI, potentially leading to
social desirability bias (Furnham, 1986), or practices adopted in company
projects they are involved in. To counteract this, we consistently reminded
interviewees that the discussion focused specifically on the TAI issues they
encounter in their daily work. At the conclusion of the interviews, we
encouraged them to freely express their broader thoughts on TAI.

Generalizability – Transferability. One notable threat lies in the
generalizability of findings, as our sample, albeit diverse, may not fully
represent the broader population of professionals engaged in AI development.
To mitigate this threat, we solicited opinions from a heterogeneous
participants sample: we took practitioners from corporates, characterized by
different sizes with different years of experience — both in their current
role and in AI-enabled systems development; with different genders, level of
education, and job roles; working in diverse application domains and
technology areas. All details can be found in the online appendix (Baldassarre
et al., 2024a).

##  7\. Conclusion

In this study, we conducted think-aloud interviews and a survey with 34 AI
practitioners to explore how they handle and implement Trustworthy AI
applications. The study highlights how, among the TAI principles considered,
practitioners mainly focus on Privacy at the expense of Fairness, even if many
practitioners acknowledge to be aware of how important this last dimension is.
Noteworthy is the fact that half of the participants stated that they did not
fix TAI issues after discovering them in their projects. Indeed only half of
them declared to have addressed the issue, either by themselves (35%) or
revolving to a third party (15%). From the study it also emerged that TAI is
mainly addressed in the initial phases of the SDLC (mainly during the Design
and Development) and few practitioners declared to also address it in later
stages, like Deploy. The strategies most employed to build TAI applications
focus on data quality enhancement or choosing the most self-explanatory
algorithm. Moreover, data and algorithm design are conducted by exploiting
manual analysis, without using any automated tool. As one could expect, the
most common cause of obstruction in implementing TAI is business constraints
(e.g., time, money).

Finally, we also identified that practitioners feel the need for tools to
monitor the model after production deployment, and knowledge bases and
actionable guidelines to help them implement trustworthiness throughout the
entire SDLC. While different frameworks, tools, and guidelines may exist
(Barletta et al., 2023), these are either composed of high-level statements
that are sometimes difficult to translate into concrete implementation
strategies or may be unknown to AI practitioners (as shown by all the answers
to the questions related to the need/usefulness of tools to address TAI
issue). As future directions, we intend to complement and expand this work
with more interviews with a larger sample of AI practitioners to investigate
some of the critical points that emerged in this study more in-depth.

###### Acknowledgements.

This study has been partially supported by the following projects: SSA (Secure
Safe Apulia - Regional Security Center, Codice Progetto 6ESURE5) and KEIRETSU
(Codice Progetto V9UFIL5) funded by ”Regolamento regionale della Puglia per
gli aiuti in esenzione n. 17 del 30/09/2014; SERICS (PE00000014) under the MUR
National Recovery and Resilience Plan funded by the European Union –
NextGenerationEU.

## References

  * (1)
  * bay (2018) 2018\.  _Wagner, B. in Being Profiled: Cogitas Ergo Sum. 10 Years of ‘Profiling the European Citizen‘_.  Amsterdam University Press.  https://mediarep.org/handle/doc/14277
  * aequitas.dssg.io (2018) aequitas.dssg.io 2018.  _aequitas.dssg.io_.  Retrieved Sep 20, 2023 from http://aequitas.dssg.io/
  * Alvarado and Wærn (2018) O. Alvarado and A. Wærn. 2018.  Towards Algorithmic Experience: Initial Efforts for Social Media Contexts.  _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (2018).  https://api.semanticscholar.org/CorpusID:5053741
  * Bakeman and Quera (2011) Roger Bakeman and Vicenç Quera. 2011.  _Observer Agreement and Cohen’s Kappa_.  Cambridge University Press, 57–71.
  * Baldassarre et al. (2023) M.T. Baldassarre, D. Caivano, B. Fernandez Nieto, D. Gigante, and A. Ragone. 2023.  The Social Impact of Generative AI: An Analysis on ChatGPT. In _Proceedings of the 2023 ACM Conference on Information Technology for Social Good_ (Lisbon, Portugal) _(GoodIT ’23)_. Association for Computing Machinery, New York, NY, USA, 363–373.
  * Baldassarre et al. (2024a) M. T. Baldassarre, D. Gigante, M. Kalinowski, A. Ragone, and S. Tibidò. 2024a.  Online appendix.  https://figshare.com/s/099d7f7a0b2da08653e6.
  * Baldassarre et al. (2024b) M. T. Baldassarre, D. Gigante, M. Kalinowski, A. Ragone, and S. Tibidò. 2024b.  Survey link.  https://forms.office.com/e/GVeeWf1Pqz.
  * Barletta et al. (2023) V.S. Barletta, D. Caivano, D. Gigante, and A. Ragone. 2023.  A Rapid Review of Responsible AI Frameworks: How to Guide the Development of Ethical AI. In _Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering_ (Oulu, Finland) _(EASE ’23)_. Association for Computing Machinery, New York, NY, USA, 358–367.  https://doi.org/10.1145/3593434.3593478
  * Benjamini and Hochberg (1995) Yoav Benjamini and Yosef Hochberg. 1995.  Controlling the false discovery rate: a practical and powerful approach to multiple testing.  _Journal of the Royal statistical society: series B (Methodological)_ 57, 1 (1995), 289–300.
  * Bosch et al. (2016) Nigel Bosch, Sidney K. D’Mello, R. Baker, Jaclyn L. Ocumpaugh, Valerie J. Shute, Matthew Ventura, Lubin Wang, and Weinan Zhao. 2016.  Detecting Student Emotions in Computer-Enabled Classrooms. In _International Joint Conference on Artificial Intelligence_.  https://api.semanticscholar.org/CorpusID:14637074
  * Braun and Clarke (2006) V. Braun and V. Clarke. 2006.  Using thematic analysis in psychology.  _Qualitative Research in Psychology_ 3 (01 2006), 77–101.
  * Braun and Clarke (2012) V. Braun and V. Clarke. 2012.  _Thematic analysis._ 57–71.
  * Bucher (2017) Taina Bucher. 2017.  The algorithmic imaginary: exploring the ordinary affects of Facebook algorithms.  _Information, Communication & Society_ 20, 1 (2017), 30–44.
  * Chouldechova (2016) Alexandra Chouldechova. 2016.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  _Big data_ 5 2 (2016), 153–163.
  * Cruzes and Dybå (2011) Daniela Cruzes and Tore Dybå. 2011.  Recommended Steps for Thematic Synthesis in Software Engineering.  _International Symposium on Empirical Software Engineering and Measurement_ , 275 – 284.  https://doi.org/10.1109/ESEM.2011.36
  * enisaml (2021) enisaml 2021.  _Securing Machine Learning Algorithms_.  Retrieved Sep 20, 2023 from https://www.enisa.europa.eu/publications/securing-machine-learning-algorithms
  * Esteva et al. (2017) Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin M. Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. 2017.  Dermatologist-level classification of skin cancer with deep neural networks.  _Nature_ 542 (2017), 115–118.
  * European Parliament and Council of the European Union (5 04) European Parliament and Council of the European Union. 2016-05-04.  Regulation (EU) 2016/679 of the European Parliament and of the Council.  https://data.europa.eu/eli/reg/2016/679/oj
  * European Union (2023) European Union. 2023.  AI Act.  https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence
  * Furnham (1986) Adrian Furnham. 1986.  Response bias, social desirability and dissimulation.  _Personality and Individual Differences_ 7 (1986), 385–400.  https://api.semanticscholar.org/CorpusID:144370219
  * Gemini Mistake (2024) Gemini Mistake 2024.  _Gemini image generation got it wrong. We’ll do better._ Retrieved Mar 13, 2024 from {{https://blog.google/products/gemini/gemini-image-generation-issue}}
  * George and Mallery (2018) Darren George and Paul Mallery. 2018.  _IBM SPSS Statistics 25 Step by Step: A Simple Guide and Reference_.
  * Greene et al. (2019) Daniel Greene, Anna Lauren Hoffmann, and Luke Stark. 2019.  Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning. In _Hawaii International Conference on System Sciences_.
  * Heckathorn (2011) Douglas D. Heckathorn. 2011.  Comment: Snowball versus Respondent-Driven Sampling.  _Sociological Methodology_ 41 (2011), 355 – 366.  https://api.semanticscholar.org/CorpusID:117640237
  * High-Level Expert Group on AI (2018a) (AIHLEG) High-Level Expert Group on AI (AIHLEG). 2018a.  Ethics guidelines for trustworthy AI | Shaping Europe’s digital future.  https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
  * High-Level Expert Group on AI (2018b) (AIHLEG) High-Level Expert Group on AI (AIHLEG). 2018b.  High-Level Expert Group on AI (AIHLEG).  https://digital-strategy.ec.europa.eu/en/policies/expert-group-ai
  * HireVue.com (2019) HireVue.com 2019.  _HireVue.com_.  Retrieved Sep 20, 2023 from https://www.hirevue.com/
  * Holborn Law LLC (2018) Holborn Law LLC. 2018.  Advisory Council on the Ethical Use of Artificial Intelligence and Data in Singapore.  https://www.cms-holbornasia.law/en/sgh/publication/singapore-to-form-advisory-council-for-ethical-use-of-ai
  * Holstein et al. (2018) Kenneth Holstein, Bruce M. McLaren, and Vincent Aleven. 2018.  Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms. In _International Conference on Artificial Intelligence in Education_.
  * Jobin et al. (2019) A. Jobin, M. Ienca, and E. Vayena. 2019.  The global landscape of AI ethics guidelines.  _Nature Machine Intelligence_ 1, 9 (01 Sep 2019), 389–399.  https://doi.org/10.1038/s42256-019-0088-2
  * Kenny et al. (2021) Eoin M. Kenny, Courtney Ford, Molly Quinn, and Mark T. Keane. 2021.  Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and error-rates in XAI user studies.  _Artificial Intelligence_ 294 (2021), 103459.
  * Kupis et al. (2019) Julia Kupis, Sydney Johnson, Gregory Hallihan, and Dana Lee Olstad. 2019.  Assessing the Usability of the Automated Self-Administered Dietary Assessment Tool (ASA24) among Low-Income Adults.  _Nutrients_ 11, 1 (2019).
  * Lee and Singh (2021) Michelle Seng Ah Lee and Jat Singh. 2021.  The Landscape and Gaps in Open Source Fairness Toolkits. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (Yokohama, Japan) _(CHI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 699, 13 pages.
  * Lum and Isaac (2016) Kristian Lum and William Isaac. 2016.  To Predict and Serve?  _Significance_ 13 (10 2016), 14–19.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions.  In _Advances in Neural Information Processing Systems 30_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765–4774.
  * NASA Artificial Intelligence Group (2018) NASA Artificial Intelligence Group. 2018.  NASA Artificial Intelligence Group.  https://ai.jpl.nasa.gov/
  * Pardo et al. (2011) César Pardo, Francisco Pino, Félix García, Francisco Romero Romero, Mario Piattini, and Maria Teresa Baldassarre. 2011.  HProcessTOOL: A support tool in the harmonization of multiple reference models.  _Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)_ 6786 LNCS, PART 5 (2011), 370 – 382.  https://doi.org/10.1007/978-3-642-21934-4_30
  * pymetrics.ai (2018) pymetrics.ai 2018.  _pymetrics.ai_.  Retrieved Sep 20, 2023 from https://www.pymetrics.ai//
  * Rader and Gray (2015) Emilee J. Rader and Rebecca M Gray. 2015.  Understanding User Beliefs About Algorithmic Curation in the Facebook News Feed.  _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (2015).
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016.  ”Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_. 1135–1144.
  * Richardson et al. (2021) Brianna Richardson, Jean I. Garcia-Gathright, Samuel F. Way, Jennifer Thom-Santelli, and Henriette Cramer. 2021.  Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits.  _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ (2021).
  * Someren et al. (1994) Maarten Someren, Yvonne Barnard, and Jacobijn Sandberg. 1994.  _The Think Aloud Method - A Practical Guide to Modelling CognitiveProcesses_.
  * Spadini et al. (2018) Davide Spadini, Maurício Aniche, Margaret-Anne Storey, Magiel Bruntink, and Alberto Bacchelli. 2018.  When Testing Meets Code Review: Why and How Developers Review Tests. In _Proceedings of the 40th International Conference on Software Engineering_ (Gothenburg, Sweden) _(ICSE ’18)_. Association for Computing Machinery, New York, NY, USA, 677–687.
  * UK AI Council (2019) UK AI Council. 2019.  UK Government.  https://www.gov.uk/government/groups/ai-council
  * UNESCO Ad Hoc Expert Group (2021) (AHEG) UNESCO Ad Hoc Expert Group (AHEG). 2021.  Recommendation on the Ethics of Artificial Intelligence.  https://www.unesco.org/en/artificial-intelligence/recommendation-ethics
  * Veale et al. (2018) Michael Veale, Max Van Kleek, and Reuben Binns. 2018.  Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_ (Montreal QC, Canada) _(CHI ’18)_. Association for Computing Machinery, New York, NY, USA, 1–14.
  * whatiftool (2018) whatiftool 2018.  _The What-If Tool: Code-free probing of machine learning models_.  Retrieved Sep 20, 2023 from https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html
  * Wohlin et al. (2012) Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012.  Experimentation in software engineering. Springer Science & Business Media.
  * Zhengxin et al. (2023) Fang Zhengxin, Yuan Yi, Zhang Jingyu, Liu Yue, Mu Yuechen, Luan Qinghua, Xu Xiwei, Wang Jeff, Wang Chen, Zhang Shuai, and Chen Shiping. 2023.  MLOps Spanning Whole Machine Learning Life Cycle: A Survey.  _ArXiv_ abs/2304.07296 (2023).