# Towards a Responsible AI Development Lifecycle: Lessons From Information
Security

Erick Galinkin<EMAIL_ADDRESS>1234-5678-9012 Rapid7120 Causeway
stBostonMassachusettsUSA02114

(2022)

###### Abstract.

Legislation and public sentiment throughout the world have promoted fairness
metrics, explainability, and interpretability as prescriptions for the
responsible development of ethical artificial intelligence systems. Despite
the importance of these three pillars in the foundation of the field, they can
be challenging to operationalize and attempts to solve the problems in
production environments often feel Sisyphean. This difficulty stems from a
number of factors: fairness metrics are computationally difficult to
incorporate into training and rarely alleviate all of the harms perpetrated by
these systems. Interpretability and explainability can be gamed to appear
fair, may inadvertently reduce the privacy of personal information contained
in training data, and increase user confidence in predictions – even when the
explanations are wrong. In this work, we propose a framework for responsibly
developing artificial intelligence systems by incorporating lessons from the
field of information security and the secure development lifecycle to overcome
challenges associated with protecting users in adversarial settings. In
particular, we propose leveraging the concepts of threat modeling, design
review, penetration testing, and incident response in the context of
developing AI systems as ways to resolve shortcomings in the aforementioned
methods.

Explainable Artificial Intelligence, Interpretable Machine Learning, Fairness,
AI Ethics

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: 2020 AAAI/ACM Conference on AI, Ethics, and
Society; August 1–3, 2022; Oxford, England††booktitle: AIES ’22: 5th AAAI/ACM
Conference on AI, Ethics, and Society††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Security and privacy Human and societal aspects
of security and privacy††ccs: Social and professional topics User
characteristics††ccs: Social and professional topics Computing / technology
policy

##  1\. Introduction

Increasing adoption of artificial intelligence in public life has sparked
tremendous interest in the fields of AI ethics, algorithmic fairness and bias,
and model explainability and interpretability. These ideas did not spring out
of thin air, but rather are a response to difficult questions about when,
where, and how it is appropriate to use artificial intelligence to perform
tasks. Much of the extant literature has aimed to provide constructive
movement in the direction of ensuring the principles of fairness,
accountability, and transparency are upheld by machine learning algorithms.
For practitioners, there are three dominant approaches in AI ethics:

  1. (1)

Fairness metrics

  2. (2)

Interpretability

  3. (3)

Explainability

Throughout this work, we use the term “AI system” to mean products and
services that leverages artificial intelligence as a decision making
component. The term “Responsible AI” then is an AI system that is built with a
notion of minimizing the potential harm. The term “harm”, used throughout the
paper, we use in accordance with Crawford’s (Crawford, 2017) use of the term
to mean both allocative and representational harms. Allocative harms are harms
which result in an improper distribution of resources on the basis of group
membership. Representational harms are more difficult to quantify than
allocative harms and result in the reinforcement of subordination of some
group on the basis of identify, e.g. race, gender, class, etc.

The topics of fairness, interpretability, and explainability are not merely of
interest to the academic world. The European Union has begun work on their “AI
Act” (Commission et al., 2021b), a law that seeks to legislate and harmonize
regulations of technologies and products that leverage artificial
intelligence. In the United States, the National Institute of Standards and
Technology has begun work on a risk management framework for artificial
intelligence (of Standards and Technology, 2021), and a number of states have
passed legislation regulating uses of artificial intelligence (of State
Legislatures, 2021). Consequently, all at once, we are developing methods for
determining and achieving fairness and explainability, implementing these
methods in industry, and seeing regulation of the technologies that encourage
or require those same methods. Unfortunately, standardization of these topics
is ongoing, there are no one-size-fits-all solutions, and there are
significant methodological and computational hurdles to overcome.

We look to the field of information security as one potential model for
success due to similarities between the two fields. In particular, information
security deals with a number of competing theories (Moody et al., 2018) and
standards (Weidman et al., 2020; Sulistyowati et al., 2020; Roy, 2020) that
make it challenging to harmonize controls. Moreover, information security,
like ethical AI, aims to find heuristics, stopgaps, and proxies for
computationally intractable problems (Cohen, 1987; Chess and McGraw, 2004)
with important human impacts. In information security, it is widely accepted
that even in the best case for mitigation, vulnerabilities and compromises
cannot be avoided entirely. To this end, information security seeks to
optimize mitigation, detection, and response. In this work, we demonstrate how
practitioners in ethical AI can use the framework of mitigation, detection,
and response to operationalize fairness, interpretability, and explainabiltiy
frameworks.

##  2\. Background and Related Works

This work builds on research in economics, AI ethics, software engineering,
and information security, drawing inspiration from Howard and Lipner’s
Security Development Lifecycle (Howard and Lipner, 2006) and from the many
ways that their work has been refined, implemented, and evolved in the
software industry. In this section, we provide background on fairness,
interpretability, and explainability with an eye towards the insufficiency of
existing methods. Crucially, in the realm of interpretability and
explainability, the presence of explanations increases user confidence in the
predictions of the model, even when the explanations are incorrect (Bansal et
al., 2021).

###  2.1. Fairness

In the economics literature, there are a variety of fairness metrics that have
been established. For many fairness metrics in the continuous case, the
problems are rarely able to be solved efficiently (Robertson and Webb, 1998)
and for indivisible goods, envy-free allocation – allocation where nobody
would rather have someone else’s good – is NP-hard (Lipton et al., 2004).
Kleinberg et al. (Kleinberg et al., 2017) explore the COMPAS risk tool and
show that for integer values, risk assignment is NP-Complete; although for
non-integer values, the problem of fair risk assignment in polynomial-time
remains open. Fairness in classification was examined by Dwork et al. (Dwork
et al., 2012), who developed fairness constraints in a classification context,
identifying statistical parity as one way to determine fairness in
classifiers. Yona and Rothblum (Yona and Rothblum, 2018) tackle the issue of
generalization from training to test sets in machine learning by relaxing the
notion of fairness to an approximation and demonstrating generalization of
metric-fairness. Like the aforementioned works, much of the literature focuses
on the fairness of a single algorithm making classifications on groups of
agents.

In the multi-agent setting, where motivations of individual agents may worsen
the outcomes of other agents, the problem becomes even more difficult. The
work of Zhang and Shah (Zhang and Shah, 2014) attempts to resolve fairness in
this multi-agent setting via linear programming and game theoretic approaches.
The game theoretic approach used tries to find a Nash equilibrium for the two-
player zero-sum setting, a problem which is known to be PPAD-Complete (Chen
and Deng, 2006) and conjectured not to be in P unless P = NP. This suggests
that in general, any attempt at algorithmic fairness is a substantial
computational problem on top of whatever problem we are aiming to solve.

In addition to computational difficulties, the work of Fazelpour and Lipton
(Fazelpour and Lipton, 2020) addresses shortcomings in the ideological
foundations of formalizing fairness metrics by connecting the existing work to
the political philosophy of ideal and non-ideal approaches. As in much of the
fair machine learning literature, ideal models in political theory imagine a
world that is perfectly just. By using this as a target, we aim to measure –
and correct for – the deviation from this ideal. However, developing this
fairness ideal in algorithmic settings necessitates comparison to other groups
and consequently, a “fair” approach may actually be worse for all groups and
yield new groups that need to be protected. Further work by Dai et al. (Dai et
al., 2021) shows that fairness allocations with narrow desiderata can lead to
worse outcomes overall when issues like the intrinsic value of diversity
(Steel et al., 2021) are not accounted for. This suggests that because the
term “fairness” is not well-defined, collaboration between developers of AI
systems and social scientists or ethicists is important to ensure any metric
for measuring fairness captures a problem-specific definition of the term.

###  2.2. Interpretability

Recent work on model interpretability has indicated that users find simpler
models more trustworthy (Schmidt and Biessmann, 2019). This is built on the
definition of Lipton (Lipton, 2018) that presumes users are able to comprehend
the entire model at once. However, the ability to interpret high-dimensional
models is limited, even when those models are linear (Molnar, 2020). This
initially suggests that model interpretability limits the available models to
low-dimensional linear models and short, individual decision trees.

Spurred by these notions, Generalized Linear Models and Generalized Additive
Models have been developed and seek to be sufficiently robust to be useful in
practice while retaining strong notions of human-interpretability. These
methods allow for linear and non-linear models that are inherently
interpretable. However, as Molnar notes (Molnar, 2020), high-dimensional
models are inherently less interpretable even when those models are linear.
Moreover, even the most interpretable models rely on assumptions about the
stability of the data generation process and any violation of those
assumptions renders interpretation of those weights invalid.

###  2.3. Explainability

Post-hoc explanations have proven very popular due to their intelligibility
and their ability to be used with complex machine learning models,
particularly neural networks. Explainablity methods tend to be model agnostic
and are more flexible than model-specific interpretation methods. We refer
readers interested in the technical details of these methods to other
resources, such as the book by Molnar (Molnar, 2020) or appropriate survey
literature (Carvalho et al., 2019). In practice, explainability methods
manifest in a variety of ways:

  1. (1)

Partial Dependence Plots

  2. (2)

Individual Conditional Explanations

  3. (3)

Accumulated Local Effects

  4. (4)

Feature Interaction

  5. (5)

Feature Importance

  6. (6)

Global Surrogates

  7. (7)

Local Surrogates

  8. (8)

Shapley values

  9. (9)

Counterfactual Explanations

  10. (10)

Adversarial Examples

  11. (11)

Attention Layer Visualization

The above methods can be broadly grouped into two buckets: global explanations
and local explanations. Global explanations seek to provide overall model
interpretability for models that are otherwise difficult to understand. These
methods will demonstrate, for example, how certain features are weighted more
heavily than others or show how correlation between variables can cause a
particular prediction. Local methods, on the other hand, purport to provide
explanations for individual predictions. The most popular among these are LIME
(Ribeiro et al., 2016), GradCAM (Selvaraju et al., 2017), and SHAP (Lundberg
and Lee, 2017), which leverage local surrogate models, gradient-based
localization, and Shapley values respectively to foster explanations. In
response to their popularity, the robustness of these methods have been
investigated. Slack et al. (Slack et al., 2020) demonstrated that these
methods do not work well in an adversarial setting – that is, they can be
fooled by a modeler who wishes to provide convincing explanations that appear
innocuous while maintaining a biased classifier. Further work by Agarwal et
al. (Agarwal et al., 2021) attempts to establish foundations for robustness in
explanation methods, finding that there are some robustness guarantees for
some methods, but those guarantees are subject to variance in the
perturbations and gradients. Beyond these issues, substantial critiques have
been leveraged against the use of Shapley values for feature importance (Kumar
et al., 2020b) based on their inconsistency across distributions and the lack
of a normative human evaluation for the values (Kaur et al., 2020).

Counterfactual explanations offer a particularly useful line of explanation,
effectively answering the question: “what would need to be different to get a
different outcome?” Humans desire counterfactual explanations, since they
provide a direction to create a different outcome in the future (Pearl and
Mackenzie, 2018). As an example, when a person applies for a bank loan and is
denied on the basis of their credit score, they expect a counterfactual
explanation that says what factors, specifically, contributed to the denial
and would need to improve in order to approve the loan. Though metacognition –
thinking about thinking – has been studied in computer science, and
particularly in cognitive architectures (Marshall, 1999), recent attempts have
been made (Kumar et al., 2021; Babu and Suresh, 2012; Vilone and Longo, 2021)
toward a metacognition for explaining difficult to interpret models, largely
in the mold of providing counterfactual explanations. However, to date,
counterfactal explanations and artificial metacognition have not developed
sufficiently to allow for their use.

###  2.4. Attacks on AI systems

There is a deep connection between security and fairness in machine learning
systems. Aside from clear connections like the link between differential
privacy and fairness in classification (Dwork et al., 2012), techniques like
adversarial examples (Szegedy et al., 2013) – inputs to models that are
similar to humans but are perturbed to cause misclassification – can be used
to evaluate the robustness of model fairness, interpretability, and
explainability. Adjacent to our taxonomy of allocative and representational
harms, we also have a taxonomy of harms that our model can perpetrate against
users and third parties: one, the harms caused by the system itself, including
the aforementioned allocative and representational harms; two, the harms
caused to users by other users of the system. The first case is well-studied,
though strategies for renumerating and redressing uncovered harms outside of
calibration primarily prescribe putting human-in-the-loop or mandating
explanations for a human gatekeeper. The harms caused to users by other users
of the system tend to align more closely with attacks on AI systems, which we
provide a high-level overview of below and refer readers to surveys on attacks
in machine learning (Pitropakis et al., 2019) and threats to privacy in
machine learning (Al-Rubaie and Chang, 2019) for additional details. These
user-on-user harms largely align with four overarching categories:

  1. (1)

Classification-level attacks

  2. (2)

Model-level attacks

  3. (3)

System-level attacks

  4. (4)

Privacy attacks

Classification-level attacks are those attacks which seek to cause
misclassification. These attacks include adversarial examples in images, but
also distinct techniques like “Bad Characters” (Boucher et al., 2022) that use
imperceptible characters to bypass text content filters. Essentially, these
attacks allow one user to harm another by causing an input to be misclassified
without altering the model, data, or anything else. These attacks would also
include attacks like the one used against Tesla’s Traffic Aware Cruise Control
(Povolny and Trivedi, 2020) where a malicious individual could easily modify a
35 mph speed limit sign with a small piece of black tape and cause the model
to incorrectly classify the sign as an 85 mph speed limit sign.

Model-level attacks differ from classification-level attacks in that they
alter the model itself. The most common example of this is a poisoning attack
– an attack in which the training data of the model are altered to cause
consistent misclassification. This often requires access to the model or the
data itself, making the attack challenging. However, in the online setting, a
number of online data poisoning attacks (Zhang et al., 2020; Kloft and Laskov,
2007) have been demonstrated to great effect. A malicious user then, could
poison the model and cause problems for all users.

System-level attacks are intend not to simply affect the predictions of the
model, but rather damage the system itself. An example here is that of sponge
examples (Shumailov et al., 2021), model inputs that are generated to maximize
energy consumption and inference time to degrade the functionality of the
system. This can also include exploitation of conventional vulnerabilities
which could allow for tampering with model inputs or outputs to harm users.

Privacy attacks include membership inference (Shokri et al., 2017; Choquette-
Choo et al., 2021) and model inversion (Fredrikson et al., 2015; Carlini et
al., 2021). Membership inference attacks seek to identify whether or not
individuals are present in the training data of a model, potentially damaging
user privacy. Model inversion then, is a step further. Rather than ask whether
or not a user’s data is present in the training data of the model, model
inversion seeks to extract training data directly from the model – a
phenomenon that has been observed in generative models. Both of these attacks
can facilitate harms to users and are within the purview of responsible AI to
limit.

##  3\. Adapting the Secure Software Development Lifecycle to Artificial
Intelligence

As discussed, attempts to satisfy fairness criteria can be limiting from a
computational perspective. Within information security, there is a notion of
formal verification (Ringer et al., 2020), a computationally intensive process
of ensuring that under any input, the program behaves as expected. This leads
to more reliable software that is less prone to exploitable bugs. Note
however, that the mission statement of formal verification – designing a
program that halts when a bug is detected – is undecidable because it is
exactly the halting problem. This has led to extensive work in both automated
and interactive verification to overcome this theoretical barrier by solving
subproblems, approximations of the problem, or writing domain-specific
automation. In many cases, formal verification for software is a larger
engineering effort than the software project itself and as a result, most
software is not formally verified. How then, do we ensure that software is not
riddled with exploitable bugs? In general, the presence of exploitable bugs in
software (Wysopal et al., 2006; Dowd et al., 2006) is reduced through a number
of steps in the secure software development lifecycle. For our purposes, we
identify analogies between ethical AI development and the following:

  1. (1)

Design Review

  2. (2)

Threat Modeling

  3. (3)

Penetration Testing

These principles reduce risk that may be introduced in software development
and produce more robust code without the overhead of formal verification
methods. In ethical artificial intelligence, we also seek to reduce the risk
of negative outcomes and discrimination. As such, we adapt these secure
software development lifecycle principles to ethical AI. One point of
disagreement in the security community that may be reflected here is whether
to perform threat modeling ahead of design review. The idea of performing
threat modeling first is to provide a thorough view of the threats so that the
risks uncovered in design review are threat-centric. We follow the convention
of performing design review ahead of threat modeling based on the rationale
that defining the threats for a system that has not yet been designed makes
the scope too broad to be useful. We note that both approaches are valid and
can be tailored to fit the maturity and preferences of the organization.

###  3.1. Design Review

In information security, a design review looks at the system under development
and assesses the architecture, design, operations, and their associated risks
(Dowd et al., 2006) allowing for implementation of systems-level security
controls such as authentication, encryption, logging, and validation. When
developing AI systems, a similar sort of design review should be conducted,
with a view toward AI risks. This means that during the design review process,
we should explore questions like:

  * •

How can we check for distribution drift?

  * •

Are we logging model queries in a way that allows us to find reported bad
behaviors?

  * •

What features do we input to the model, and do they introduce potential
issues?

  * •

Are there other data sources we should be incorporating into this model?

  * •

What actions, if any, are taken automatically as a result of model
predictions?

This step provides a system-level view of how data goes into and predictions
come out of the system and is ideally conducted before the system is deployed.
The idea, at the design review step, is to identify data flows and consider
how the system could be refactored or rearchitected to avoid potential risks.
Things like data pre-processing or calibration (Barocas et al., 2017) should
be discussed at this step, and if they are not needed or not sufficient, there
should be documentation as to why they are omitted. This goes beyond the
actual model and training pipeline to include where data is derived from, what
additional data is collected, where predictions and logs are stored, and other
system-level issues.

An important part of the design review process is a discussion of how data
related to the system is generated, processed, and stored. This is an
important part of the system that is often viewed through a lens of privacy
and policy, but not always with a view of how to responsibly manage data.
While data management and mismanagement can cause one to run afoul of data
privacy legislation, there are a variety of personal data misuses (Kröger et
al., 2021) that can cause harm. This means that the privacy of data per se is
not the entirety of the discussion, but how the data moves through the system
to become a classification needs to be uncovered. An investigation into this
requires analysis of all data used in predictions, whether these are raw data,
proxy features that stand in for data that is not directly available, or
transformed features like those yielded from principle component analysis.

###  3.2. Threat Modeling

Threat modeling is the phase of the development process that aims to predict
the threats that a system may face (Shostack, 2014). Akin to how one might
imagine ways to secure a home by evaluating the locks, windows, and entrances
to their home, threat modeling seeks to evaluate how attackers may gain entry
to a system. Since AI systems are software, the security threat modeling
conducted should incorporate those systems. By analogy, we want to think not
only of threats to our system, but how our system could pose a risk to users.
This comes in two forms: malicious users of our system harming other users,
and harms that our system could hypothetically cause.

When it comes to harming other users, we look to AI security and data privacy
for potential harms (Kröger et al., 2021). Essentially, we must assess if
users are fully independent and if not, the ways in which one user could
potentially harm another. As an example, malicious users could extract
training data from trained models or infer individuals membership in the
training data (Carlini et al., 2021; Choquette-Choo et al., 2021) which could
then be used to harm those individuals privacy. Another example is malicious
users conducting data poisoning attacks (Ahmed and Kashmoola, 2021),
particularly for online machine learning systems (Zhang et al., 2020) that
might lead to bad outcomes for other users. This is one way that AI security
directly influences AI ethics.

On the other hand, enumerating ways in which a system using AI could harm
users is also critical. Some harms may be expected: a self-driving car that
does not recognize a pedestrian (Shepardson and Jin, 2021), a discriminatory
bail-setting algorithm (Angwin et al., 2016), an image cropping algorithm
suffering from the “male gaze” (Birhane et al., 2022). However, other harms
could rear their head. For example, the EMBER malware (malicious software)
dataset (Anderson and Roth, 2018) includes a large number of features for
Windows Portable Executable files, including the language of the system the
malware was compiled on. One could conclude, based on the command and control
infrastructure and the compilation language of the malware, that the presence
of Chinese language is indicative of maliciousness and correspondingly
restrict access to Chinese language websites. One harm this could introduce,
however, is inadvertent discrimination against Chinese-speaking users who may
wish to visit legitimate webpages or run legitimate software. Ultimately, we
may conclude that the benefit of deploying the system outweighs the risk – but
identifying this possible harm is still an important part of the threat
modeling process that we will revisit in our section on Incident Response.

###  3.3. Penetration Testing

The concept of a penetration test is simple – a trusted individual or team
with adversarial skills seeks to find weaknesses in a system in accordance
with the same techniques an attacker would use. In the context of developing
ethical AI systems, a “penetration test” then approaches our AI system with
the same tools and intent as a malicious actor. This test should evaluate an
attacker’s ability to harm the system, harm the users of a system, and also
uncover harms latent in the system. Much like the Twitter Algorithmic Bias Bug
Bounty (Chowdhury and Williams, 2021), we can and should directly evaluate our
algorithms from an adversarial perspective, even if only internally. Though
the term penetration testing has a particular meaning in the information
security context, we use it here to refer to the use of adversarial techniques
to uncover potential harms in AI systems. Additionally, we eschew the phrase
“algorithmic bias assessment” since bias is only one potential cause for harm
and we seek to use a more task-oriented term.

Conducting these sort of assessments require both AI security skills and
sociotechnical knowledge. As of 2021, only 3 out of 28 organizations surveyed
conducted security assessments on their machine learning systems (Kumar et
al., 2020a), suggesting that many organizations are not currently well-
equipped to evaluate these vulnerabilities and would need to cultivate teams
capable of performing algorithmic harm assessments. Utilities like Counterfit
(Pearce and Kumar, 2021) and PrivacyRaven (Hussain, 2020) have lowered the
barrier to entry for security professionals to use adversarial examples and
membership inference attacks on machine learning models, but many
organizations still do not assess their machine learning security. These same
utilities are critical to conducting these assessments against models.
Additionally, simple tactics like using so-called beauty filters can also
demonstrate bias in machine learning systems (Fingas, 2021). In order to
devise new tactics to target these algorithms, AI assessors need to understand
both the technical and social factors included in these systems. Importantly,
the act of testing these systems assists us not only in identifying potential
harms but also in assessing the robustness of our system.

Another key to penetration testing is the need to test the full system as
deployed. Since the algorithm is not deployed in a vacuum, there may be
feature engineering, allow and block-listing, preprocessing, post-processing,
and other steps that could allow problems to creep into the system. Many so-
called “AI systems” are not single algorithms deployed behind an API, but are
instead a tapestry of data engineering, multiple algorithms, and post-
processing systems. In some cases, an algorithm may be biased against a
particular group, but some calibration (Barocas et al., 2017) in a post-
processing system corrects for the identified issue. In other cases, the added
complexity of the overall system may actually amplify small changes to inputs
and cause a larger effect that one might observe on the individual algorithm.

##  4\. Incident Response

An often overlooked discussion is how to deal with a harm perpetrated by an AI
system once it is identified. In the field of information security, there is
the concept of a breach – a successful intrusion by an attacker into our
system – and when this occurs, we begin the incident response process.
Typically an incident response process occurs alongside execution of a
business continuity plan, a predefined plan for how to continue execution when
there is a security event or natural disaster. The incident response process
involves eliminating the attacker’s access to systems, patching
vulnerabilities that were exploited, and taking steps to ensure that the
attacker does not get back in. Similarly, we should be prepared in the field
of AI to respond to events where our system creates or perpetuates harm.

There are a number of ways harms can be identified even after design review,
threat modeling, and penetration testing such as through a bias bounty, a news
report, or a user reporting that they have been harmed. Once the existence of
a harm is identified, the work of incident response begins with identifying
what the actual harm is. This can be an acute damage or harm to an individual,
a systemic bias problem, or the potential for a third-party to harm other
users of the system. A self-driving car that strikes a pedestrian is a
commonly-used example because the harm is clear: there exists a configuration
of vehicles, pedestrians, and other distractions such that the vehicle does
not stop before a pedestrian is struck. Other harms, such as bias against
racial and gender minorities as observed in the cases of COMPAS (Angwin et
al., 2016) and Amazon’s hiring algorithm (Dastin, 2018) are less obvious until
we conduct research into exactly what harms occurred. Whether the harm
identified is an acute damage to an individual or an ongoing systemic harm, we
must take immediate action to:

  1. (1)

Continue operations if possible

  2. (2)

Perform root cause analysis

  3. (3)

Remediate the harms caused

###  4.1. Continuity Planning

Once a harm is established, all reasonable efforts to prevent another incident
should be taken. In many cases, this means removing a system from production
for a period of time while the remainder of the incident response process is
executed. Some sort of procedure should be established to allow for continuity
of operations during this period that is contingent on the severity of the
harm. For example, a self driving car that strikes a pedestrian may require
temporarily suspending self-driving across a fleet or limiting where it can be
used. In the case of something like a discriminatory sentencing algorithm, we
may simply allow judges to operate as they did before the tool was available,
suspending its use. Other cases, such as Twitter’s image cropping algorithm’s
“male gaze” bias or its aversion to non-Latin text may not rise to the need
for continuity response and can remain in production.

In many cases, the scale of these harms – bad user experience, emotional pain
and suffering, loss of life – can be anticipated, even if the specific harm
cannot. This provides the ability to set up risk-based continuity planning.
Essentially, we seek to answer the question: “if we have to remove this system
from production, what will we do instead?” to ensure that those who depend in
some way on these systems are still able to leverage them, even with limited
functionality.

###  4.2. Root Cause Analysis

In security, root cause analysis is used to ask and answer questions about the
series of events which led to a security incident, often with a particular
focus on the vulnerabilities exploited and why they were not patched. Even in
so-called blameless post-mortems, the root cause analysis seeks to determine
what was the weak link in the chain and how said weak link could have been
avoided. In the case of algorithmic harms, a root cause analysis is likely to
be much more involved, due to the large number of pieces at play.

The first place to look when a harm occurs is what, if anything, has changed
in the system since it most recently functioned at an acceptable level. If
there was an update the morning before an incident, it is prudent to
investigate whether or not the previous version of the system would have
caused the harm. If not, an ablation study should be conducted across the
pipeline to identify what components, if any, could be changed to mitigate the
harm. This root cause analysis then informs future penetration tests and
threat models to ensure that another incident is not caused by the same cause.

###  4.3. Remediating Harms

After a root cause is identified, it is prudent to remediate both the harms
themselves and the causes of said harms. Remediating a harm depends a lot on
the particulars of the harms caused and is currently an issue being openly
discussed. For the teenagers harmed by the promotion of eating disorders on
social media (O’Sullivan et al., 2021), it is unlikely that they will be
directly compensated by the organization perpetuating the harm. Remediating
the harm itself is a difficult task that asks much larger questions about who
is responsible for these incidents, how the costs are handled, and who, if
anyone, owes harmed parties reparations for said harms. For harms at the scale
of COMPAS, the questions grow even larger. However, as governments like the EU
consider revising their product liability regimes to incorporate AI
(Commission et al., 2021a), developers and purveyors of these systems should
develop a plan for how to address potential claims against their systems.

Remediating the cause of the harm then, is the more straightforward task –
though by no means is the task simple. Remediating the harm extends the work
of root cause analysis and opens the question of how to fix the root cause. In
the case of bias, this could be a matter of finding a new dataset, calibrating
according to sensitive attributes, leveraging multicalibration (Hébert-Johnson
et al., 2018), decision calibration (Zhao et al., 2021), or some other method.
In other cases, the cause of the harm may necessitate pre- or post-processing
of data and decisions to create guardrails. Yet other cases may require a
fundamental reconsideration of the system in use and whether or not it is
feasible to have a safe, fair system. These harms and remediations must be
documented to ensure that future projects do not fall into the same trap and
can be evaluated using similar methods.

##  5\. A Responsible AI Development Lifecycle

Figure 1. An Illustration of the Responsible AI Development Lifecycle

Given the analogies between security and ethical artificial intelligence in
Sections 3 and 4, we propose a framework for a responsible AI development
lifecycle, illustrated in Figure 1. Organizations working with artificial
intelligence and machine learning have existing processes for design,
development, training, testing, and deployment of AI systems. Though we do not
detail those processes here, the proposed framework aims not to replace any
part of that process, but rather to augment existing processes with ethical
principles. The responsible AI development lifecycle consists of five steps:

  1. (1)

Planning and Review

  2. (2)

Design Review

  3. (3)

Harm Modeling

  4. (4)

Penetration Testing

  5. (5)

Incident Response

Planning and review, as both the first and last step in the process, is
intended to revise existing systems and inform the development of new systems.
This step should occur before any new code is written, whether that is in the
process of remediating harms or designing a new system. The planning and
review process should look back at prior findings in this system and other
systems and set forth the steps that need to be taken in developing this new
system. Additionally, this step is where business continuity planning as
outlined in Section 4.1 should occur. One critical part of planning and review
is the process of documentation – this involves documenting plans and
findings, capturing learnings from incidents, and identifying structures that
should be in place ahead of development.

This process leads into the design review step where the overall design of the
system is set out. The intent of the system should be clear, the algorithmic
components should be well understood, and data sources should be documented.
In a more mature organization, a design review should include datasheets
(Gebru et al., 2021) or model cards (Mitchell et al., 2019) to nail down
specific places where there are known issues. As discussed in Section 3.1, the
design review should consider not only the artificial intelligence component,
but the scaffolding around it and the system as a whole to include logging and
auditing, pre-processing, post-processing, and in-processing.

Coming out of the design review step, training and tuning models, deployment,
and testing can occur in parallel with harm modeling. As discussed in Section
3.2, this is a good place for counterfactual reasoning – asking all of the
“what if” scenarios to understand what can go wrong. This process should
involve stakeholders from throughout the organization to identify potential
harms to users of the system and to external parties. Where possible, this
process should also identify mitigations that can be put in place before the
system goes live. The reason to put mitigations in place ahead of time is
twofold: first, having mitigations in place ahead of deployment reduces the
likelihood that an individual experiences an identified harm; second, it
reduces the cost of putting the mitigation in place since there is no downtime
needed to implement it.

When the system is deployed or ready for deployment, we can conduct
penetration testing of the system, as in Section 3.3. This penetration test
differs from a traditional penetration test in a number of respects, but is
not entirely divorced from the original notion. Specifically, we are still
concerned with discovering security bugs since many of those can be leveraged
to cause harm. Where this approach differs is in taking a broader view of what
is in-scope for a penetration test, since we are concerned not only with the
possibility of attacks on our models, but also with potential representational
harms that are difficult to uncover in security testing.

Assuming that we have established a continuity plan and done our job in the
design review phase, there should be good feedback mechanisms for uncovering
harms in our systems. These can be monitored and spot-corrected over time to
ensure that the system is functioning as intended and is not perpetrating
harms. When something goes out of wack or a user or third-party reports a
problem, we should initiate our incident response. As mentioned in Section 4,
we should begin by identifying the scope and scale of the harm, then proceed
to initiate our business continuity plan if necessary. Once we have determined
the root cause of the harm, we develop a plan to alleviate the problem and
proceed back to the top of the process – reviewing our findings and planning
our fixes.

As teams cycle through this process for a particular system, each trip through
the cycle should be shorter and easier. In some cases, the cycle may terminate
entirely if the outcome of the review and planning step is a decision not to
use an AI system. When evaluating the risks and benefits of deploying an AI
system, it is important to always consider the reference point of simply not
using artificial intelligence

##  6\. Conclusion

This work establishes a framework for developing AI systems responsibly. We
identify two parallel taxonomies of harm: allocative harm versus
representational harm and system-on-user harm versus user-on-user harm. These
two taxonomies allow us to develop methods of uncovering, identifying, and
classifying harms to users. Since AI development occurs not in a vacuum but
rather as part of a broader development cycle, we view the proposed framework
as something that can easily work alongside existing AI system production
methods.

The proposed system consists of 5 steps that mirror the standard lifecycle of
a software system – design, development, training, testing, and deployment. As
development proceeds, our framework helps AI developers and stakeholders
evaluate their system for potential harms and address them. This framework
differs from existing prescriptions of fairness metrics, explainability
methods, and interpretable models due to the shortcomings of those methods
computationally, epistemologically, and practically. Since nearly all systems
inevitably change, fail, or prove insufficient, this framework offers an
opportunity for iterative ethical improvements without sacrificing
practicality. We achieve this by analogy with information security, and our
framework built on the foundations of the secure software development
lifecycle.

This framework makes a number of strong assumptions by necessity.
Specifically, we assume that a “harm” can be identified and that given
sufficient care, can be resolved. Crucially, we also assume that organizations
actually want to identify and remediate harms to users and are willing to
expend effort to improve their systems to that end. Finally, we assume that
the skills to evaluate these models is present in organizations that wish to
adopt these principles – something that we know is untrue for the majority of
organizations.

In future work, we aim to address the feasibility of automating parts of this
process to reduce the need for specially-skilled individuals. Today, the
creation of documentation around bias, robustness, and other important
features of responsible AI is cumbersome and manual. By offering opportunities
to automate and operationalize some of this work, adoption of these processes
is eased.

## References

  * (1)
  * Agarwal et al. (2021) Sushant Agarwal, Shahin Jabbari, Chirag Agarwal, Sohini Upadhyay, Zhiwei Steven Wu, and Himabindu Lakkaraju. 2021.  Towards the Unification and Robustness of Perturbation and Gradient Based Explanations.  _International Conference on Machine Learning_.
  * Ahmed and Kashmoola (2021) Ibrahim M Ahmed and Manar Younis Kashmoola. 2021.  Threats on Machine Learning Technique by Data Poisoning Attack: A Survey. In _International Conference on Advances in Cyber Security_. Springer, 586–600.
  * Al-Rubaie and Chang (2019) Mohammad Al-Rubaie and J Morris Chang. 2019.  Privacy-preserving machine learning: Threats and solutions.  _IEEE Security & Privacy_ 17, 2 (2019), 49–58.
  * Anderson and Roth (2018) Hyrum S Anderson and Phil Roth. 2018.  Ember: an open dataset for training static pe malware machine learning models.  _arXiv preprint arXiv:1804.04637_ (2018).
  * Angwin et al. (2016) Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016.  Machine bias: There’s software used across the country to predict future criminals. And it’s biased against blacks. ProPublica (2016).  _Google Scholar_ (2016), 23\.
  * Babu and Suresh (2012) G Sateesh Babu and Sundaram Suresh. 2012.  Meta-cognitive neural network for classification problems in a sequential learning framework.  _Neurocomputing_ 81 (2012), 86–96.
  * Bansal et al. (2021) Gagan Bansal, Tongshuang Wu, Joyce Zhou, Raymond Fok, Besmira Nushi, Ece Kamar, Marco Tulio Ribeiro, and Daniel Weld. 2021\.  Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * Barocas et al. (2017) Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2017\.  Fairness in machine learning.  _Nips tutorial_ 1 (2017), 2017.
  * Birhane et al. (2022) Abeba Birhane, Vinay Uday Prabhu, and John Whaley. 2022\.  Auditing Saliency Cropping Algorithms. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 4051–4059.
  * Boucher et al. (2022) Nicholas Boucher, Ilia Shumailov, Ross Anderson, and Nicolas Papernot. 2022.  Bad characters: Imperceptible nlp attacks.  _43rd IEEE Symposium on Security and Privacy_ (2022).
  * Carlini et al. (2021) Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021\.  Extracting training data from large language models. In _30th USENIX Security Symposium (USENIX Security 21)_. 2633–2650.
  * Carvalho et al. (2019) Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. 2019.  Machine learning interpretability: A survey on methods and metrics.  _Electronics_ 8, 8 (2019), 832.
  * Chen and Deng (2006) Xi Chen and Xiaotie Deng. 2006.  Settling the complexity of two-player Nash equilibrium. In _2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS’06)_. IEEE, 261–272.
  * Chess and McGraw (2004) Brian Chess and Gary McGraw. 2004.  Static analysis for security.  _IEEE security & privacy_ 2, 6 (2004), 76–79.
  * Choquette-Choo et al. (2021) Christopher A Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. 2021.  Label-only membership inference attacks. In _International Conference on Machine Learning_. PMLR, 1964–1974.
  * Chowdhury and Williams (2021) Rumman Chowdhury and Jutta Williams. 2021.  Introducing Twitter’s first algorithmic bias bounty challenge.  https://blog.twitter.com/engineering/en_us/topics/insights/2021/algorithmic-bias-bounty-challenge.  [Online; accessed 17-January-2022].
  * Cohen (1987) Fred Cohen. 1987\.  Computer viruses: theory and experiments.  _Computers & security_ 6, 1 (1987), 22–35.
  * Commission et al. (2021a) EU Commission et al. 2021a.  Civil liability – adapting liability rules to the digital age and artificial intelligence.  https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/12979-Civil-liability-adapting-liability-rules-to-the-digital-age-and-artificial-intelligence/public-consultation_en.  (2021).  [Online; accessed 3-February-2022].
  * Commission et al. (2021b) EU Commission et al. 2021b.  Proposal for a regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts.  _COM (2021)_ 206 (2021).
  * Crawford (2017) Kate Crawford. 2017\.  The Trouble With Bias.  https://nips.cc/Conferences/2017/Schedule?showEvent=8742 Thirty-first Conference on Neural Information Processing Systems Keynote Presentation.
  * Dai et al. (2021) Jessica Dai, Sina Fazelpour, and Zachary Lipton. 2021\.  Fair machine learning under partial compliance. In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_. 55–65.
  * Dastin (2018) Jeffrey Dastin. 2018\.  Amazon scraps secret AI recruiting tool that showed bias against women.
  * Dowd et al. (2006) Mark Dowd, John McDonald, and Justin Schuh. 2006.  _The art of software security assessment: Identifying and preventing software vulnerabilities_.  Pearson Education.
  * Dwork et al. (2012) Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012.  Fairness through awareness. In _Proceedings of the 3rd innovations in theoretical computer science conference_. 214–226.
  * Fazelpour and Lipton (2020) Sina Fazelpour and Zachary C Lipton. 2020.  Algorithmic fairness from a non-ideal perspective. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 57–63.
  * Fingas (2021) Jon Fingas. 2021\.  Twitter’s AI bounty program reveals bias toward young, pretty white people.  https://www.engadget.com/twitter-ai-bias-beauty-filters-133210055.html.  [Online; accessed 19-January-2022].
  * Fredrikson et al. (2015) Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. 2015\.  Model inversion attacks that exploit confidence information and basic countermeasures. In _Proceedings of the 22nd ACM SIGSAC conference on computer and communications security_. 1322–1333.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021\.  Datasheets for datasets.  _Commun. ACM_ 64, 12 (2021), 86–92.
  * Hébert-Johnson et al. (2018) Ursula Hébert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. 2018.  Multicalibration: Calibration for the (computationally-identifiable) masses. In _International Conference on Machine Learning_. PMLR, 1939–1948.
  * Howard and Lipner (2006) Michael Howard and Steve Lipner. 2006.  _The security development lifecycle_. Vol. 8.  Microsoft Press Redmond.
  * Hussain (2020) Suha Hussain. 2020\.  PrivacyRaven Has Left the Nest.  https://blog.trailofbits.com/2020/10/08/privacyraven-has-left-the-nest/.  [Online; accessed 19-January-2022].
  * Kaur et al. (2020) Harmanpreet Kaur, Harsha Nori, Samuel Jenkins, Rich Caruana, Hanna Wallach, and Jennifer Wortman Vaughan. 2020.  Interpreting interpretability: understanding data scientists’ use of interpretability tools for machine learning. In _Proceedings of the 2020 CHI conference on human factors in computing systems_. 1–14.
  * Kleinberg et al. (2017) Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2017\.  Inherent Trade-Offs in the Fair Determination of Risk Scores. In _8th Innovations in Theoretical Computer Science Conference (ITCS 2017)_. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
  * Kloft and Laskov (2007) Marius Kloft and Pavel Laskov. 2007.  A poisoning attack against online anomaly detection. In _NIPS Workshop on Machine Learning in Adversarial Environments for Computer Security_ , Vol. 19. Citeseer.
  * Kröger et al. (2021) Jacob Leon Kröger, Milagros Miceli, and Florian Müller. 2021.  How Data Can Be Used Against People: A Classification of Personal Data Misuses.  _Available at SSRN 3887097_ (2021).
  * Kumar et al. (2020b) I Elizabeth Kumar, Suresh Venkatasubramanian, Carlos Scheidegger, and Sorelle Friedler. 2020b.  Problems with Shapley-value-based explanations as feature importance measures. In _International Conference on Machine Learning_. PMLR, 5491–5500.
  * Kumar et al. (2020a) Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. 2020a.  Adversarial machine learning-industry perspectives. In _2020 IEEE Security and Privacy Workshops (SPW)_. IEEE, 69–75.
  * Kumar et al. (2021) Sannidhi P Kumar, Chandan Gautam, and Suresh Sundaram. 2021\.  Meta-Cognition-Based Simple And Effective Approach To Object Detection. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 3795–3799.
  * Lipton et al. (2004) Richard J Lipton, Evangelos Markakis, Elchanan Mossel, and Amin Saberi. 2004.  On approximately fair allocations of indivisible goods. In _Proceedings of the 5th ACM Conference on Electronic Commerce_. 125–131.
  * Lipton (2018) Zachary C Lipton. 2018\.  The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.  _Queue_ 16, 3 (2018), 31–57.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions. In _Proceedings of the 31st international conference on neural information processing systems_. 4768–4777.
  * Marshall (1999) James B Marshall. 1999\.  _Metacat: A self-watching cognitive architecture for analogy-making and high-level perception_.  Indiana University.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the conference on fairness, accountability, and transparency_. 220–229.
  * Molnar (2020) Christoph Molnar. 2020\.  _Interpretable machine learning_.  Lulu. com.
  * Moody et al. (2018) Gregory D Moody, Mikko Siponen, and Seppo Pahnila. 2018\.  Toward a unified model of information security policy compliance.  _MIS quarterly_ 42, 1 (2018).
  * of Standards and Technology (2021) National Institute of Standards and Technology. 2021.  AI Risk Management Framework Concept Paper.  https://www.nist.gov/itl/ai-risk-management-framework.  [Online; accessed 10-January-2022].
  * of State Legislatures (2021) National Conference of State Legislatures. 2021\.  Legislation Related to Artificial Intelligence.  https://www.ncsl.org/research/telecommunications-and-information-technology/2020-legislation-related-to-artificial-intelligence.aspx.  [Online; accessed 10-January-2022].
  * O’Sullivan et al. (2021) Donie O’Sullivan, Clare Duffy, and Sarah Jorgensen. 2021\.  Instagram promoted pages glorifying eating disorders to teen accounts.  https://www.cnn.com/2021/10/04/tech/instagram-facebook-eating-disorders/index.html.  [Online; accessed 27-January-2022].
  * Pearce and Kumar (2021) Will Pearce and Ram Shankar Siva Kumar. 2021.  AI Security Risk Assessment Using Counterfit.  https://www.microsoft.com/security/blog/2021/05/03/ai-security-risk-assessment-using-counterfit/.  [Online; accessed 19-January-2022].
  * Pearl and Mackenzie (2018) Judea Pearl and Dana Mackenzie. 2018.  _The book of why: the new science of cause and effect_.  Basic books.
  * Pitropakis et al. (2019) Nikolaos Pitropakis, Emmanouil Panaousis, Thanassis Giannetsos, Eleftherios Anastasiadis, and George Loukas. 2019.  A taxonomy and survey of attacks against machine learning.  _Computer Science Review_ 34 (2019), 100199.
  * Povolny and Trivedi (2020) Steve Povolny and Shivangee Trivedi. 2020.  Model hacking ADAS to pave safer roads for autonomous vehicles.  _McAfee Advanced Threat Research_ (2020).
  * Ribeiro et al. (2016) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016\.  ” Why should i trust you?” Explaining the predictions of any classifier. In _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_. 1135–1144.
  * Ringer et al. (2020) Talia Ringer, Karl Palmskog, Ilya Sergey, Milos Gligoric, and Zachary Tatlock. 2020.  QED at large: A survey of engineering of formally verified software.  _arXiv preprint arXiv:2003.06458_ (2020).
  * Robertson and Webb (1998) Jack Robertson and William Webb. 1998.  _Cake-cutting algorithms: Be fair if you can_.  CRC Press.
  * Roy (2020) Prameet P Roy. 2020\.  A High-Level Comparison between the NIST Cyber Security Framework and the ISO 27001 Information Security Standard. In _2020 National Conference on Emerging Trends on Sustainable Technology and Engineering Applications (NCETSTEA)_. IEEE, 1–3.
  * Schmidt and Biessmann (2019) Philipp Schmidt and Felix Biessmann. 2019.  Quantifying interpretability and trust in machine learning systems.  _AAAI-19 Workshop on Network Interpretability for Deep learning_ (2019).
  * Selvaraju et al. (2017) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization. In _Proceedings of the IEEE international conference on computer vision_. 618–626.
  * Shepardson and Jin (2021) David Shepardson and Hyunjoo Jin. 2021.  U.S. probing fatal Tesla crash that killed pedestrian.  https://www.reuters.com/business/autos-transportation/us-probing-fatal-tesla-crash-that-killed-pedestrian-2021-09-03/.  _Reuters_ (2021).  [Online; accessed 17-January-2022].
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017.  Membership inference attacks against machine learning models. In _2017 IEEE symposium on security and privacy (SP)_. IEEE, 3–18.
  * Shostack (2014) Adam Shostack. 2014\.  _Threat modeling: Designing for security_.  John Wiley & Sons.
  * Shumailov et al. (2021) Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, and Ross Anderson. 2021\.  Sponge examples: Energy-latency attacks on neural networks. In _2021 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 212–231.
  * Slack et al. (2020) Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju. 2020.  Fooling lime and shap: Adversarial attacks on post hoc explanation methods. In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_. 180–186.
  * Steel et al. (2021) Daniel Steel, Sina Fazelpour, Bianca Crewe, and Kinley Gillette. 2021.  Information elaboration and epistemic effects of diversity.  _Synthese_ 198, 2 (2021), 1287–1307.
  * Sulistyowati et al. (2020) Diah Sulistyowati, Fitri Handayani, and Yohan Suryanto. 2020\.  Comparative Analysis and Design of Cybersecurity Maturity Assessment Methodology Using NIST CSF, COBIT, ISO/IEC 27002 and PCI DSS.  _JOIV: International Journal on Informatics Visualization_ 4, 4 (2020), 225–230.
  * Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013\.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ (2013).
  * Vilone and Longo (2021) Giulia Vilone and Luca Longo. 2021.  Notions of explainability and evaluation approaches for explainable artificial intelligence.  _Information Fusion_ (2021).
  * Weidman et al. (2020) Jake Weidman, Igor Bilogrevic, and Jens Grossklags. 2020\.  Nothing Standard About It: An Analysis of Minimum Security Standards in Organizations. In _European Symposium on Research in Computer Security_. Springer, 263–282.
  * Wysopal et al. (2006) Chris Wysopal, Lucas Nelson, Elfriede Dustin, and Dino Dai Zovi. 2006.  _The art of software security testing: identifying software security flaws_.  Pearson Education.
  * Yona and Rothblum (2018) Gal Yona and Guy Rothblum. 2018.  Probably approximately metric-fair learning. In _International Conference on Machine Learning_. PMLR, 5680–5688.
  * Zhang and Shah (2014) Chongjie Zhang and Julie A Shah. 2014.  Fairness in multi-agent sequential decision-making. In _Advances in Neural Information Processing Systems_. 2636–2644.
  * Zhang et al. (2020) Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. 2020\.  Online data poisoning attacks. In _Learning for Dynamics and Control_. PMLR, 201–210.
  * Zhao et al. (2021) Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. 2021.  Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration.  _Advances in Neural Information Processing Systems_ 34 (2021).