# Trustworthy AI: From Principles to Practices

Bo Li<EMAIL_ADDRESS>JD AI ResearchBeijingChina ,  Peng Qi<EMAIL_ADDRESS>JD
AI ResearchMountain ViewCAUSA ,  Bo Liu<EMAIL_ADDRESS>JD AI ResearchMountain
ViewCAUSA ,  Shuai Di<EMAIL_ADDRESS>JD AI ResearchBeijingChina ,  Jingen Liu
<EMAIL_ADDRESS>JD AI ResearchMountain ViewCAUSA ,  Jiquan Pei
<EMAIL_ADDRESS>JD AI ResearchBeijingChina ,  Jinfeng Yi<EMAIL_ADDRESS>JD
AI ResearchBeijingChina and  Bowen Zhou<EMAIL_ADDRESS><EMAIL_ADDRESS>JD
AI ResearchBeijingChina

###### Abstract.

Fast developing artificial intelligence (AI) technology has enabled various
applied systems deployed in the real world, impacting people’s everyday lives.
However, many current AI systems were found vulnerable to imperceptible
attacks, biased against underrepresented groups, lacking in user privacy
protection, etc., which not only degrades user experience but erodes the
society’s trust in all AI systems. In this review, we strive to provide AI
practitioners a comprehensive guide towards building trustworthy AI systems.
We first introduce the theoretical framework of important aspects of AI
trustworthiness, including robustness, generalization, explainability,
transparency, reproducibility, fairness, privacy preservation, alignment with
human values, and accountability. We then survey leading approaches in these
aspects in the industry. To unify the current fragmented approaches towards
trustworthy AI, we propose a systematic approach that considers the entire
lifecycle of AI systems, ranging from data acquisition to model development,
to development and deployment, finally to continuous monitoring and
governance. In this framework, we offer concrete action items to practitioners
and societal stakeholders (e.g., researchers and regulators) to improve AI
trustworthiness. Finally, we identify key opportunities and challenges in the
future development of trustworthy AI systems, where we identify the need for
paradigm shift towards comprehensive trustworthy AI systems.

trustworthy AI, robustness, generalization, explainability, transparency,
reproducibility, fairness, privacy protection, value alignment, accountability

††ccs: Computing methodologies Artificial intelligence††ccs: General and
reference Surveys and overviews††ccs: Computing methodologies Machine learning

##  1\. Introduction

The rapid development of artificial intelligence (AI) is bringing great
economic and social prospects to human society. With the widespread
application of AI in areas such as transportation, finance, medicine,
security, and entertainment, there is rising societal awareness that we need
these systems to be trustworthy. This is because the breach of stakeholders’
trust can lead to severe societal consequences given the pervasiveness of
these AI systems. In contrast, AI practitioners, including researchers,
developers, decision-makers, etc., have traditionally been pursuing system
performance (a.k.a. accuracy) as the main metric in their workflows. This
metric is far from sufficient to reflect the requirement of AI
trustworthiness. Beyond system performance, various aspects of AI systems
should be carefully considered to improve their trustworthiness, including but
not limited to aspects such as robustness, algorithmic fairness,
explainability, transparency, etc.

While most active academic research on AI trustworthiness focuses on the
algorithmic properties of models, we find that development in algorithmic
research alone is insufficient to build trustworthy AI products. From an
industrial perspective, the lifecycle of an AI product involves multiple
stages, including data preparation, algorithmic design, development,
deployment, as well as operation, monitoring, and governance. Achieving
trustworthiness in any single aspect (e.g., robustness) involves efforts at
multiple stages in the system lifecycle, e.g., data sanitization, robust
algorithms, anomaly monitoring, risk auditing, etc. On the other hand, the
breach of trust in any single link or any single aspect could undermine the
trustworthiness of the entire system. Therefore, AI trustworthiness should be
established and assessed systematically throughout the whole lifecycle of an
AI system.

In addition to pursuing AI trustworthiness by establishing trustworthy
requirements in different trustworthiness aspects, the interaction between
these aspects is an important and under-explored topic for real-world
trustworthy AI systems. For instance, the need for data privacy might
interfere with the desire to explain system output in detail, and the pursuit
of algorithmic fairness could be detrimental to the accuracy and robustness
experienced by some groups. As a result, greedily pursuing these different
aspects alone does not necessarily result in an optimal solution towards more
trustworthy AI systems. Instead, trustworthy AI should be established by
trading-off and jointly optimizing multiple trustworthiness aspects.

The above facts suggest that a systematic approach is necessary to shift the
current AI paradigm for trustworthiness. This requires awareness and
cooperation from multidisciplinary stakeholders who work on different
trustworthiness aspects and different stages of the the system lifecycle. To
help develop this systematic approach, we organize multidisciplinary knowledge
in an accessible manner for AI practitioners to learn about AI
trustworthiness, and provide an operational and systematic guidance towards
building trustworthy AI systems. Our main contributions include:

  * •

We survey and expand on recent discussions regarding AI trustworthiness
(kumar2020trustworthy, ; wickramasinghe2020trustworthy, ;
thiebes2020trustworthy, ; wing2020trustworthy, ; floridi2019establishing, ;
cammarota2020trustworthy, ; shneiderman2020bridging, ; aihleg2018ethics, ;
brundage2020toward, ; singh2021trustworthy, ; varshney2019trustworthy, ;
liu2021trustworthy, ) and establish our desiderata of trustworthy AI systems
from an industrial perspective, which include robustness, generalization,
explainability, transparency, reproducibility, fairness, privacy protection,
value alignment, and accountability (Section 2).

  * •

We broadly review various stakeholders’ attempts at realizing these
desiderata, including active academic research, industrial development
techniques, and governance and management mechanisms. This diverse and
comprehensive collection of approaches helps provide an overall picture of AI
trustworthiness and bridge the knowledge gap between practitioners from
different backgrounds (Section 3).

  * •

We dissect the entire development and deployment lifecycle of AI systems in
industrial applications, and discuss how AI trustworthiness can be enhanced at
each stage – from data to AI models, from system deployment to its operation.
We propose a systematic framework to organize the multidisciplinary and
fragmented approaches towards trustworthy AI, and further propose to pursue AI
trustworthiness as a continuous workflow to incorporate feedback at each stage
of the AI system lifecycle. We also analyze the relationship between different
trustworthiness aspects in practice (mutual enhancement and sometimes trade-
offs). With this, we aim to provide AI practitioners such as researchers,
developers, operators, and legal experts an accessible and comprehensive guide
to quickly understand the approaches towards AI trustworthiness (Section 4).

  * •

We discuss outstanding challenges of trustworthy AI that the research
community and industrial practitioners should focus on solving in the near
future. We identify several key issues, including the need for deeper
fundamental understanding of several aspects of AI trustworthiness (e.g.,
robustness, fairness, and explainability), the importance of user awareness,
as well as the promotion of interdisciplinary and international collaboration
(Section 5).

With the above contributions, we aim at providing practitioners and
stakeholders of AI systems not only a comprehensive introduction to the
foundations and future of AI trustworthiness, but also an operational
guidebook towards how to construct AI systems that are trustworthy.

Figure 1. The relation between different aspects of AI trustworthiness
discussed in this survey. Notice that implicit interaction widely exists
between different aspects and we only cover the representative explicit
interaction.

##  2\. AI Trustworthiness: Beyond Prediction Accuracy

The success of ML technology in the last decades has largely benefited from
the accuracy-based performance measurement. By denoting task performance with
quantitative accuracy or loss, training AI models becomes tractable in the
manner of optimization. Meanwhile, prediction accuracy is widely adopted to
indicate the superiority of an AI product. However, with the recent widespread
applications of AI, the limitation of such accuracy-only measurement is
exposed to a number of new challenges. These challenges range from malicious
attacks against AI systems to misuses of AI which violates human values. To
mitigate these problems, in the last decade the AI community has realized that
more factors beyond accuracy should be considered and improved when building
an AI system. Recently a number of enterprises, academia, and public sector
organizations have identified these factors and summarized them as principles
of AI trustworthiness, which includes robustness, security, transparency,
fairness, safety, etc (jobin2019global, ). In this paper, we provide an
overview of these principles with the following categorization:

  * •

We consider representative requirements which present technical challenges to
current AI systems, including robustness, explainability, transparency111We
follow (turilli2009ethics, ) to exclude transparency as an ethical
requirement., reproducibility and generalization.

  * •

We consider ethical requirements with global consensus (jobin2019global, ),
including fairness, privacy, and accountability. We also consider the
requirement of value alignment which concerns the alignment of generic human
ethics in an AI system.

In this section, we illustrate the motivation and definition of each
requirement. In addition, we survey the evaluation approaches for each
requirement. It should also be noted that the selected requirements are not
orthogonal and some of them are closely correlated. We explain the
relationships with the corresponding requirements in this section. We also use
Figure 1 to visualize the relationships between different aspects, which
includes trade-offs, contributions, and manifestation.

###  2.1. Robustness

In general, robustness refers to the ability of an algorithm or system to deal
with execution errors, erroneous inputs, or unseen data (ieee-std-robustness,
). As for an AI system, robustness is an important factor affecting its
performance in realistic environments. The lack of robustness might also cause
unintended or harmful behavior to the system, diminishing its safety and
trustworthiness.

In the context of machine learning systems, the term of robustness is
applicable in diverse situations. In this review, we summarize the robustness
of an AI system by categorizing the vulnerabilities of an AI system and
describe the representatives as follows.

Distributional shifts. With the widespread application of AI systems, the
environment where an AI model is deployed becomes more complicated and
diverse. If an AI model is trained without considering the diverse
distributions of data in different scenarios, its performance might be
significantly affected. The robustness against distributional change has been
a common problem in various applications of AI (amodei2016concrete, ). In
high-stake applications, this problem is even more critical due to the
negative effect on safety and security. For example, in the area of autonomous
driving, besides developing a perceptual system using data from sunny scenes,
academia and industry are conducting numerous development and testing
strategies to enhance the perceptual performance in rainy scenes to guarantee
the system reliability under various weather conditions.

Adversarial attacks. It is widely recognized that AI systems might be
vulnerable to attacks by adversaries with malicious intentions. Among the
various forms of attacks, the adversarial attack and its defense has been an
area which raises concern in both academia and industry recently. Literature
has categorized the threat of adversarial attack in several typical aspects
and proposed various defense approaches (chakraborty2018adversarial, ;
silva2020opportunities, ; akhtar2018threat, ; yuan2019adversarial, ;
li2020backdoor, ). We briefly describe these threats as categorized in
(chakraborty2018adversarial, ). Consider a face recognition system as an
example. Evasion attack creates perturbed samples to mislead the verification
without affecting the given model. A person wearing adversarial pattern
glasses might hide himself/herself from being recognized. Poisoning attack
injects carefully designed samples into the training data to change the
system’s response to specific samples or patterns. Exploratory attack attempts
to steal knowledge about the models. Though it does not directly change the
model behavior, the stolen knowledge is of significant value to generate
adversarial samples.

Illegal inputs. Besides the above two types of vulnerability, the system-level
robustness against illegal inputs should also be carefully considered in
realistic AI products. The cases of illegal inputs can be extremely diverse in
practical situations. For example, a face verification system might have
printed face images or masks as inputs, a.k.a. presentation attack
(ramachandra2017presentation, ), which violates the original use case design.
Unexpected erroneous inputs are also a typical source of system-level
vulnerability. An image of very high resolution might hang an imperfect image
recognition system. An autonomous vehicle lidar perception system might
perceive laser beams emitted by lidars from other vehicles and produce
corrupted inputs.

To prevent AI systems from vulnerabilities, various approaches have been
explored. The objective of defense can be either proactive or reactive
(machado2020adversarial, ). Proactive defense attempts to optimize the AI
system to be more robust to various inputs, while reactive defense aims at
detecting the potential security issues such as changing distribution or
adversarial samples. Representative approaches to improve the robustness of an
AI system will be introduced in Section 4.

##### Evaluation

Evaluating the robustness of an AI system serves as an important approach to
avoid vulnerabilities and control risks. We briefly describe two groups of
evaluations: robustness test and mathematical verification.

Robustness test. Testing has served as an essential approach to evaluate and
enhance the robustness of not only conventional software but also AI systems.
Conventional functional test methodology such as monkey test
(exforsys2011what, ) provide effective approaches to evaluate robustness
against illegal input. Moreover, as will be introduced in Section 4.3.1,
software testing methodology is further extended in recent literature to
evaluate the robustness against adversarial attacks (pei2017deepxplore, ;
ma2018deepgauge, ).

In addition to functional tests, performance tests, i.e., benchmarking serves
as a more widely adopted approach in the ML area to evaluate system
performance on various dimensions. Test datasets collected by design serve as
the most widely used means to evaluate the robustness on the various
distributions of data in ML research. Benchmarking is also used to evaluate
robustness against typical illegal inputs such as face spoofing
(ramachandra2017presentation, ). In terms of adversarial attack, the minimal
adversarial perturbation is a core robustness metric and is widely evaluated
by its empirical upper bound, a.k.a. empirical robustness, on a test dataset
(su2018robustness, ; carlini2017towards, ). From the attacker’s perspective,
the attack success rate also intuitively measures the robustness of the system
(su2018robustness, ).

Mathematical verification. Inherited from the theory of formal method,
certified verification on the adversarial robustness of an AI model has raised
growing interest in the research of ML. For example, by deriving a non-trivial
and certified lower bound of the minimum distortion to attack an AI model, the
adversarial robustness is simultaneously reflected (zhang2018efficient, ;
boopathy2019cnn, ). We further introduce this direction in Section 4.2.1 and
4.3.4 .

###  2.2. Generalization

A core requirement of trustworthy AI systems is the capability to distill
knowledge from limited training data to make accurate predictions on data
unseen during training. On the one hand, this requires AI systems to make
accurate predictions on realistic unseen data, preferably even from domains or
distributions they are not trained on. On the other hand, they should be able
to generalize without the need to exhaustively collect and annotate large
amounts of data for various domains.

Generalization has been a long-lasting topic in ML research. The canonical
research of generalization theory focuses on the prediction of unseen data
which typically share the same distribution as the training data
(goodfellow2016machine, ). Even though AI models are able to achieve
reasonable accuracy on training datasets, it is known that a gap (a.k.a.
generalization gap) exists between the training accuracy and the practical
testing accuracy. Various approaches have been studied in the area from
statistic learning to deep learning to analyze this problem and enhance the
model generalization. Typical approaches like cross-validation,
regularization, and data augmentation can be found in many ML textbooks
(goodfellow2016machine, ).

The creation of a modern data-driven AI model requires a large amount of data
and annotations at the training stage. It brings high cost to manufacturers
and users to re-collect and re-annotate data to train models for each task.
This raises the demand to generalize the knowledge of a model to different
tasks, which not only reduces data cost, but also improves model performance
in many cases. Various research directions have been explored to address the
knowledge generalization under different scenarios and configurations. For
example, domain adaption (zhou2021domain, ; wang2021generalizing, ) aims at
finding domain invariant features such that an algorithm can achieve close
performance across different domains. As another example, the goal of few
shots learning is to develop algorithms to enable the model’s generalizability
of learning from only a few examples (wang2020generalizing, ; cheng2019few, ;
yu2018diverse, ). To some extent, some learning paradigms like pre-training
and meta-learning also provide a mechanism to transfer prior knowledge from
the old task to different new tasks [63,391] for better reproducibility
(wang2020generalizing, ; brown2020language, ). These studies explore the
possibility to produce an AI model with low cost and high flexibility, which
provides a better experience to AI stakeholders including developers,
operators, and users.

The inclusive concept of generalization is closely related to some other
aspects of AI trustworthiness, especially robustness. In the context of ML,
the aforementioned robustness against distributional shifts in Section 2.1 is
also deemed to be a problem of generalization. This implies that the
requirements of robustness and generalization have some overlapping aspects.
In terms of adversarial robustness, its relationship with generalization is
more complicated. As demonstrated in (xu2012robustness, ), an algorithm that
is robust to small perturbations shall have better generalization. Some recent
research (tsipras2018robustness, ; raghunathan2019adversarial, ), however,
observe that improving adversarial robustness by adversarial training may hurt
the test accuracy and result in worse generalization. To explain this
phenomenon, (fawzi2018adversarial, ) argues that the adversarial robustness
corresponds to different data distributions, which may hurt a model’s
generalization.

##### Evaluation

Benchmarking on variously distributed test datasets serves as a widely adopted
approach to evaluate the generalization of an AI model in realistic scenarios.
In statistical learning, cross-validation has been used as an effective
approach for long time to evaluate generalization on the same distribution of
training and testing datasets. In terms of domain generalization, a summary of
commonly used datasets and benchmarks can be found in (zhou2021domain, ),
covering tasks of object recognition, action recognition, segmentation, and
face recognition.

In terms of theoretical evaluation, the past decades of ML studies have
developed rich theories to measure a model’s generalization by estimating its
generalization error bound. For example, Rademacher complexity
(bartlett2002rademacher, ) is commonly used to tell how well a model can fit a
random assignment of class labels. In addition, Vapnik-Chervonenkis (VC)
dimension (vapnik2013nature, ) is a measure of the capacity/complexity of a
learnable function set. A higher VC dimension indicates a larger capacity of
the hypothesis. The generalization error has an upper bound in terms of either
Rademacher complexity or VC dimension.

The growth of DNN is further bringing new development to the theory of
generalization. (zhang2016understanding, ) observes that the modern deep
learning models can achieve small generalization gap despite of their massive
capacity. This phenomenon has led to active academic discussions on this open
problem of the generalization of DNN (arpit2017closer, ; krueger2017deep, ;
kawaguchi2017generalization, ; belkin2018reconciling, ). For example,
(belkin2018reconciling, ) looks into the generalization from the perspective
of bias-variance tradeoff to explain and evaluate the generalization of DNN.

###  2.3. Explainability and Transparency

The opaqueness of complex ML models has brought widespread concerns to
academia, industry, and society. From the theoretical aspect, the problem of
how DNN models outperform other conventional ML approaches has been puzzling
researchers. From the practical aspect, the information opacity of an AI
product inevitably harms its trustworthiness. The requirements of
explainability and transparency on an AI system address the concerns of AI
opaqueness in the above two aspects respectively and are recognized to help
build the trust of the society on AI technology (lipton2018mythos, ).

####  2.3.1. Explainability

Explainability, i.e., understanding how an AI model makes its decision, stays
at the core place of modern AI research, and serves as a fundamental factor
that determines the trust in AI technology. The motivation of AI
explainability comes from various aspects (arrieta2020explainable, ). From the
perspective of scientific research, it is meaningful to understand all bits
and bytes of the data, parameters, procedures, and outcomes in an AI system,
which provides insight into the nature of both machine intelligence and human
intelligence. From the perspective of building AI products, explainability
helps understand the AI system in outlining its defects and also reduce the
by-product trade-offs (kim2018introduction, ). Explainability does not
directly increase the safety or robustness of an AI system, but helps expose
the risk issues (kim2018introduction, ). From the ethical point of view,
explainability also helps alleviate the concern about responsibility
distinction of the black box model (doshi2017towards, ).

Explaining ML models has been an active area in not only ML research but also
psychological research in the past five years (arrieta2020explainable, ;
guidotti2018survey, ; bodria2021benchmarking, ; dovsilovic2018explainable, ;
lipton2018mythos, ; molnar2020interpretable, ; dovsilovic2018explainable, ;
adadi2018peeking, ). Although the definition of explainability of an AI model
is still an open question, we have seen an amount of research addressing this
problem from the aspects of artificial intelligence
(rosenfeld2019explainability, ; guidotti2018survey, ) and psychology
(gunning2019darpa, ; miller2019explanation, ). In summary, these works
converge to a division of the following two levels on explainability.

  * •

Model explainability by design. A series of fully or partially explainable ML
models have been designed in the past half century of ML research. Typical
representatives include linear regression, decision trees, KNN, rule-based
learners, generalized additive model (GAM), and Bayesian models
(arrieta2020explainable, ). The design of a new explainable model is currently
an active area in ML.

  * •

Post-hoc model explainability. Despite the good explainability of the above
conventional models, more complex models such as DNN or GDBT are showing more
preferable performance in the recent success of industrial AI systems. Since
approaches are still not found to obtain a holistic explainability of these
complex models, researchers instead turn to post-hoc explanation which
addresses a model’s behavior by analyzing its input, intermediate result, and
output. One representative category of this aspect approximates the decision
surface either globally or locally by an explainable ML model, i.e.,
explainer, such as linear model (ribeiro2016should, ; lundberg2017unified, )
and rules (ribeiro2018anchors, ; guidotti2018local, ). For deep learning
models like CNN or transformer, the inspection on intermediate features is
also a widely adopted means to explain model behavior (tu2020select, ;
yang2018hotpotqa, ).

Explainability approaches have been an active area in ML and are
comprehensively surveyed in a variety of literature (arrieta2020explainable, ;
guidotti2018survey, ; bodria2021benchmarking, ; dovsilovic2018explainable, ;
molnar2020interpretable, ; dovsilovic2018explainable, ). Representative
algorithms to achieve the above two levels of explainability will be reviewed
in Section 4.2.2.

##### Evaluation

In addition to the problem of explaining AI models, a unified evaluation of
explainability has also been a recognized challenge. One of the major reasons
lies in the ambiguity of the psychological outlining of explainability. To
sidestep this problem, a variety of works use qualitative metrics to evaluate
explainability with human participation. Representative approaches include:

  * •

Subjective human evaluation. The evaluation methods include interview, self-
report, questionnaire, and case study, which measure e.g., user satisfaction,
mental model, and trust (gunning2019darpa, ; hoffman2018metrics, ;
poursabzi2018manipulating, ).

  * •

Human-AI task performance. In tasks with human-AI collaboration, the
collaborative performance is significantly affected by human understanding of
the AI collaborators, and can be viewed as a reflection of the explanation
quality (mohseni2018multidisciplinary, ). This evaluation has been used in the
development of e.g., recommendation systems (kulesza2012tell, ) and data
analysis (goodall2018situ, ).

In addition, if the explainability is achieved by an explainer, the
performance of the explainer, such as the approximation precision (fidelity
(guidotti2018local, ; ribeiro2018anchors, ; ribeiro2016should, )), can be used
to indirectly quantitatively evaluate the explainability (alvarez2018towards,
).

In spite of the above various evaluations, a direct quantitative measurement
of explainability is still a problem far from solved. Some naive measurements
of model complexity like tree depth (blanco2020machine, ) and rule set size
(lakkaraju2017interpretable, ) have been studied as surrogate explainability
metrics in previous works. We believe a unified quantitative metric lies at
the very heart of fundamental AI research. The recent research on ML model
complexity (hu2021model, ) and cognitive functional complexity
(wang2003measurement, ) may inspire more future research on inventing a
unified quantitative evaluation metric.

####  2.3.2. Transparency

Transparency requires the information disclosure of a system and has been a
recognized requirement in the area of software engineering (do2010software, ;
cysneiros2009initial, ). In the AI industry, this requirement of transparency
naturally covers the lifecycle of an AI system and helps stakeholders confirm
that appropriate design principles are reflected. Consider a biometric system
for identification for example. users generally concern the purpose of
collecting their biometric information and how it to be used. Business
operators will concern the accuracy and robustness on attacks to control
risks. Government sectors will concern whether it carefully follows guidelines
and regulations. On the whole, transparency serves as a basic requirement to
build the public’s trust in AI systems (jobin2019global, ;
arnold2019factsheets, ; knowles2021sanction, ).

To achieve lifecycle transparency of an AI system, various information
regarding its creation should be considered to disclose, including design
purposes, data sources, hardware requirements, configurations, working
conditions, expected usage, system performance, etc. A series of research have
studied to disclose this information by appropriate documentation
(arnold2019factsheets, ; gebru2018datasheets, ; holland2018dataset, ;
mitchell2019model, ; piorkowski2020towards, ) and will also be discussed in
Section 4.5.1. The recent trend of open source also significantly contributes
to the algorithmic transparency of AI systems.

Due to the complex dynamic internal procedure of an AI system, the facts
regarding the creation of an AI system are not sufficient in many cases to
fully introspect its mechanism. Hence, transparent introspection of the
runtime process and decision-making should also be considered in various
scenarios. For For an interactive AI system, appropriately designed user
interface serves as an important means to disclose the underlying decision
procedure (aihleg2018assessment, ). In many safety-critical systems such as
autonomous driving vehicles, logging systems (koo2019distributed, ;
ayvaz2019witness, ; perez2010argos, ; yao2020smart, ) are widely adopted to
trace and analyze the system execution.

##### Evaluation

Though a unified quantitative evaluation is not yet available, the qualitative
evaluation of transparency has undergone recent development in the AI
industry. Assessment checklists (aihleg2018assessment, ; schelenz2020applying,
) have been regarded as an effective means to evaluate and enhance the
transparency of a system. From the psychological aspect of users or the
public, user study or A/B test would provide a useful evaluation based on user
satisfaction (mohseni2018multidisciplinary, ).

Quality evaluations of AI documentation have also been explored in recent
studies. Some literature (raji2019ml, ; arnold2019factsheets, ;
gebru2018datasheets, ; holland2018dataset, ; mitchell2019model, ) has proposed
standard practices to guide and evaluate the documentation of an AI system.
(piorkowski2020towards, ) further summarizes the general quality dimension for
more specified evaluations.

###  2.4. Reproducibility

Modern AI research involves both mathematical derivation and computational
experiments. The reproducibility of these computational procedures serves as
an essential step to verify AI research. In terms of AI trustworthiness, this
verification facilitates the detection, analysis, and mitigation of potential
risks in an AI system, such as vulnerability on specific inputs or unintended
bias. With the gradual establishment of the open cooperative ecosystem in the
AI research community, reproducibility is becoming an arising concern among AI
researchers and developers. Besides enabling effective verification of
research, reproducibility allows the community to quickly convert the latest
approaches into practice or conduct follow-up research on improvement.

It has become a new trend in the AI research community to regard
reproducibility as a requirement when publicizing research
(gundersen2018reproducible, ). We have seen top conferences such as NeurIPS,
ICML, and ACMMM to introduce reproducibility policies or programs
(pineau2020improving, ) to encourage the reproducibility of works. To obtain
subdivided assessment, degrees of reproducibility are also studied in works
such as ACM Artifact Review or (gundersen2018state, ;
drummond2009replicability, ). For example, as in (gundersen2018state, ), the
lowest degree requires the exact replication of an experiment with the same
implementation and same data, while the higher degree requires to use
different implementation or different data. Beyond the basic verification of
research, the higher degree of reproducibility promotes the further
understanding of research by distinguishing the key factors of effectiveness.

Some recently developed massive AI models such as GPT-3 or AlphaGo
representatively reflect the challenges to the reproducibility of AI research.
The creation of the models involves specifically designed data collection
strategies, efficient storage of big data, communication and scheduling
between distributed clusters, algorithm implementation, software, and hardware
environment, and various knowhow tricks. It can be seen that factors affecting
reproducibility range over the whole lifecycle of an AI system. This fact
reflects the requirements that the reproducibility problem should be addressed
by considering the whole lifecycle of an AI system. In recent studies on ML
reproducibility, this requirement is decomposed into reproducibility factors
of data, method, and experiment (gundersen2018state, ;
gundersen2018reproducible, ; isdahl2019out, ), where the experiment factor
ranges over a series of lifecycle artifacts such as code, documentation,
software, hardware, and deployment configuration. Based on this methodology,
an increasing number of machine learning platforms are being developed that
assist researchers and developers to better track the lifecycle in a
reproducible manner (isdahl2019out, ; zaharia2018accelerating, ).

##### Evaluation

Reproducibility checklists have been recently widely adopted in ML conferences
to effectively assess the reproducibility of submissions (pineau2020improving,
). Beyond the replication of experiments in publication,
(gundersen2018reproducible, ; gundersen2018state, ) also further specify
checklists to evaluate reproducibility at different degrees. Besides
checklists, mechanisms such as reproducibility challenges and reproducibility
paper tracks are also adopted to evaluate paper reproducibility
(pineau2020improving, ; ferro2018overview, ; ferro2019overview, ). To
quantitatively evaluate reproducibility in challenges, a series of
quantitative metrics have been studied. For example, (ferro2018overview, ;
ferro2019overview, ; breuer2020measure, ) specifically designed metrics to
quantify how close an information retrieval system can be reproduced to its
origins.

###  2.5. Fairness

When AI systems help us in hiring, financial risk assessment, face
identification, etc., systematic unfairness in their decisions could have
significant negative social ramifications (bogen2018help, ;
dressel2018accuracy, ), usually disproportionately impacting underpriviledged
communities (howard2018ugly, ; hao2019making, ; hao2019ai, ). This not only
damages the trust in AI from various stakeholders, but also hampers the
further development and application of AI technology for the greater good.
Therefore, it is crucial that practitioners keep in mind the fairness of the
AI systems to avoid instilling or exacerbating social bias (mehrabi2019survey,
; caton2020fairness, ; drozdowski2020demographic, ).

The main objective of fairness in AI systems is to eliminate or mitigate the
effect of biases. Bias often manifests in the form of unfair _treatment_ in an
application of different _groups_ of people based on their protected
information (e.g., gender, race, ethnicity, etc.), therefore group identity
(sometimes also called _sensitive variables_) and system response (prediction)
are two of the main factors. In some cases, there are also objective _ground
truths_ of a given task that one should consider when evaluating system
fairness, e.g., whether the speech of a person is correctly recognized or
their face correctly identified.

Fairness can be applicable at multiple granularities of system behavior
(mehrabi2019survey, ; caton2020fairness, ; verma2018fairness, ). More
commonly, we are concerned with the aggregated behavior and the bias therein
of an AI system, which is referred to as _statistical fairness_ or _group
fairness_. In certain applications, it is also helpful to consider the
criterion of _individual fairness_ or _counterfactual fairness_ , especially
when the sensitive variable can be more easily decoupled from the other
features that should justifiably determine the system prediction. While the
former is more widely applicable to various machine learning tasks (e.g.,
speech recognition, face identification, etc.), the latter can be critical in
cases like resume reviewing for candidate screening (bertrand2004emily, ).

At the group level, researchers have identified three abstract principles
(caton2020fairness, ). We illustrate these with a simple example of hiring
candidates from a population consisting of 50% male and 50% female candidates,
where gender is the sensitive variable (examples adapted from
(verma2018fairness, ; zhong2018tutorial, )):

  * •

Independence. This requires that the system outcome to be statistically
independent of sensitive variables. In our example, this requires the
admission rate for male and female candidates to be equal (known as
demographic parity (zemel2013learning, ); see also disparate impact
(friedler2014certifying, )).

  * •

Separation. Independence does not account for a justifiable correlation
between the ground truth and the sensitive variable (e.g., fewer female
candidates might be able to lift 100-pound goods than male candidates).
Separation therefore requires that the independence principle holds
conditioned on the underlying ground truth. That is, if the job requires
strength qualifications, the admission rate for qualified male and female
candidates to be equal (known as equal opportunity (hardt2016equality, ); see
also equal odds (berk2017fairness, )).

  * •

Sufficiency. Sufficiency similarly considers the ground truth but requires
that the true outcome and the sensitive variable are independent when
conditioned on the same system prediction. That is, given the same hiring
decision predicted by the model, we want the same ratio of qualified
candidates among male and female candidates (known as test fairness
(chouldechova2017fair, ; hardt2016equality, ); see also well calibration
(kleinberg2016inherent, )). This is closely related to model calibration
(pleiss2017fairness, ).

Note that these principles are mutually exclusive under certain circumstances
(e.g., independence and separation cannot both hold when the sensitive
variable is correlated with the ground truth), and (kleinberg2016inherent, )
discusses the trade-off between various fairness metrics.

##### Evaluation

Despite the simplicity of the abstract criteria of fairness outlined in the
previous section, fairness can manifest as many different forms following
these principles (see (wilson2021building, ; caton2020fairness, ) for more
comprehensive surveys, and (madaio2020co, ) for AI Ethics Checklists). We
categorize fairness metrics by the properties of models and tasks they concern
to help the reader choose appropriate ones for their application:

Discrete vs continuous variables. Task output, model prediction, and sensitive
variables can all be discrete (e.g., classification, nationality, etc.),
ranked (e.g., search engines, recommendation systems), or continuous (e.g.,
regression, classifier scores, age, etc.) in nature. For discrete variables,
empirical correlation can be evaluated with standard statistical tools such as
correlation coefficients (Pearson/Kendall/Spearman) and ANOVA, while
continuous variables often further require binning, quantization, or loss
functions to evaluate fairness (caton2020fairness, ).

Loss function. Fairness criteria often cannot be exactly satisfied given
limitations of empirical data (e.g., demographic parity between two groups
when only hiring three candidates), where loss functions are useful to gauge
how far we are from empirical fairness. The choice of loss functions can be
informed by the nature of the variables of concern: if the variables represent
probabilities, likelihood ratios are more meaningful (e.g., disparate impact
(friedler2014certifying, )); for real-valued regression, the difference
between mean distances to the true value aggregated over each group might be
used instead to indicate whether we model one group significantly better than
the other (calders2013controlling, ).

Multiple sensitive variables. In many applications, the desirable AI system
should be fair to more than one sensitive variables (e.g., loan risk
prediction should be fair to both gender and ethnicity, inter alia; a
recommendation system should ideally be fair to both users and recommendees).
One could either tradeoff between the “marginal fairness” between these
variables when considered one at a time, i.e., evaluate fairness on each
variable separately and combine the loss functions for final evaluation, or
explore the full cartesian product (sixta2020fairface, ) of all variable
values to achieve joint fairness, which typically requires more empirical
observations available but tends to satisfy stronger ethical requirements.

###  2.6. Privacy protection

Privacy protection mainly refers to protecting against unauthorized use of the
data that can be used to directly or indirectly identify a person or
household. These personal data covers a wide range of information, including
but not limited to name, age, gender, face image, and fingerprints. Privacy
has been traditionally viewed as part of fundamental human rights. Commitment
on privacy protection is regarded as a core factor determining the
trustworthiness of an AI system. The recently released AI ethics guidelines
also highlight privacy as one of the consensus AI ethical principles
(jobin2019global, ; aihleg2018ethics, ).

To regulate data privacy protection, government agencies are taking increasing
efforts on privacy protection legislation. The General Data Protection
Regulation (GDPR) being enforceable since 2018 is one of the representative
legal frameworks. The coercive forces from legislation push enterprises to
take effective measures for user privacy protection.

Besides internal privacy protection within each enterprise, recent development
in data exchange across AI stakeholders is bringing new scenarios and
challenges for privacy protection. For example, when training a medical AI
model, each healthcare institution typically only has data from local
residents, which might be insufficient. This leads to a practical demand to
collaborate with other institutions and jointly train a model
(sheller2020federated, ) without leaking privacy information across
institutions.

Existing protection techniques penetrate in the whole life cycle of AI systems
to address the rising concern of privacy. In Section 4.2.5 we briefly review
the data privacy protection techniques that are used in data collection and
processing (data anonymization, differential privacy in Section 4.1.2), model
training (secure multi-party computing, federated learning, in Section 4.2.5),
and model deployment (hardware security in Section 4.4.4). The implementation
of privacy protection is also closely related to some other aspects of
trustworthy AI. For example, the transparency principle is widely adopted by
AI systems. They inform users for personal data collection and enabling
privacy settings. In the development of privacy-preserving ML software such as
federated learning222such as FATE and PySyft, open source is becoming a common
practice to increase transparency and certify its protectiveness.

##### Evaluation

Data privacy protection laws like GDPR require data protection impact
assessment (DPIA) if any data processing is possible to bring about data
privacy risk. Measures have to be taken to address the risk concerns and
demonstrate compliance with the law (aihleg2018assessment, ). Data privacy
protection professionals and other stakeholders are required to be involved in
the assessment to evaluate it.

In academic research, quantitative principles are studied to evaluate data
privacy preservation. Taking data anonymization as an example, $k$-anonymity
is one popular principle (samarati1998protecting, ). It means in a data
release the information for each person cannot be distinguished from at least
$k-1$ other people whose information is also contained in the release. Other
principles include $(c,k)$-safety (martin2007worst, ), $\delta$-presence
(nergiz2007hiding, ), etc.

The protectiveness of privacy-preserving AI algorithms is usually verified by
mathematical proof. The verification is usually conducted under assumptions
such as semi-honest security, which implies all participated parties follow
the protocol to perform the computation task but they may try to infer the
data of other parties from the received intermediate computation results. One
example is (lindell2009proof, ), in which a proof is provided for Yao’s two-
party functionality computing protocol (yao1986generate, ), under semi-honest
security assumption. A stricter assumption is the malicious attack assumption,
where each participated party may not follow the protocol and take any
possible measures to infer the data (lindell2021secure, ). Covert adversaries
assumption is proposed in (aumann2007security, ), where the involved parties
may be malicious but they do not wish to be “caught” attacking the
computation.

###  2.7. Value Alignment

AI systems should be designed to enhance human society’s welfare without
violating principles of human values. With the widespread application of AI
products, the abuse of AI technology without human values consideration has
drawn wide concern in the last decade. Negative examples such as offensive
remarks by conversational assistants (chai2020keep, ) and DeepFake fraud cause
losses to society and even harm the development of the AI industry and
research. To alleviate this concern and build AI trustworthiness, it is
necessary to incorporate alignment with human values over the whole lifecycle
of an AI system, which requires efforts on not only product design but also AI
technology development.

Product design. Based on the application scenario and functionality
requirements of an AI system, relevant ethical principles should be considered
when designing every stage of its lifecycle. Consider a video recommendation
system as an example. A good understanding of the user groups during
requirement design provides important guidance to avoid ethical risks such as
making offensive recommendations to particular groups. To enhance social
responsibility, some companies have studied to design a ranking strategy which
avoids addiction and punishes clickbaits (stray2020aligning, ). To this end, a
suitable user interface with anti-addict design will also be of important
value.

Technology development. It has been a long-lasting problem on whether and how
to impose human moral and values on machines (wallach2008moral, ).
Conventional optimization-based ML training methodology relies on designed
objective losses or rewards to supervise a model to learn specific tasks.
However, it is admitted to be difficult to turn the various principles of
human values into computable objectives. Though being premature, some works
have pioneered in the research on imposing moral principles in AI models
(gabriel2020artificial, ). Representative studies include apprenticeship
learning (abbeel2004apprenticeship, ) and evolutionary strategies
(salimans2017evolution, ) in reinforcement learning.

##### Evaluation

Beyond maximizing performance or profit as the objective, the AI community is
focusing on defining more various measurements to optimize the AI systems to
increase human wellbeing and social good. Specifically designed checklists
(aihleg2018assessment, ) and guidelines (jobin2019global, ; aihleg2018ethics,
) have been an intuitive manner to evaluate if AI systems follow principles of
human values. Another intuitive means is user survey, which is widely used in
modern product design. Due to the difficulty of conducting sufficient user
survey in various scenarios, proxy measurements also draws attention in AI
industry. (stray2020aligning, ) introduces an example of increasing user
wellbeing in social network by increasing active interaction with friends. A
metric of meaningful social interaction is selected as a proxy measurement to
survey users’ rates on different social media posts. A model is then trained
from the survey data to assess posts and further to optimize the news feed
ranking system.

###  2.8. Accountability: a Holistic Evaluation throughout the Above
Requirements

We have described a series of requirements to build trustworthy AI.
Accountability addresses the regulation on the AI systems to follow these
requirements. With the gradually improving legal and institutional norms on AI
governance, accountability becomes a crucial factor for AI to sustainably
benefit society with trustworthiness (doshi2017accountability, ).

Accountability runs through the whole lifecycle of an AI system and requires
stakeholders of an AI system to be obligated to justify their design,
implementation, and operation to align with human values. On the executive
level, this justification is realized by considerate product design, reliable
technique architecture, responsible assessment on the potential impact, and
the information disclosure of these aspects (leslie2019understanding, ). Note
that in terms of information disclosure, transparency contributes the basic
mechanism to facilitates accountability of an AI system
(doshi2017accountability, ; diakopoulos2016accountability, ).

Accountability also derives the concept of auditability, which requires the
justification of a system to be reviewed, assessed, and audited
(leslie2019understanding, ). Algorithmic auditing has become a recognized
approach to ensure the accountability of an AI system and assess its impact on
multiple dimensions of human values (raji2020closing, ). See also Section
4.5.2.

##### Evaluation

Checklist-based assessments have been studied to qualitatively evaluate
accountability and auditability (aihleg2018assessment, ; sai2020audting, ). As
mentioned in this section, we consider accountability as the comprehensive
justification of each concrete requirement of trustworthy AI. Its realization
is composed of evaluations of these requirements over the lifecycle of an AI
system (raji2020closing, ). Hence, the evaluation of accountability is
actually reflected by to what extent these requirements of trustworthiness and
their impact could be evaluated.

##  3\. Recent Practices

Diverse AI stakeholders have made efforts to enhance AI trustworthiness in the
last decade. We review their recent practices in this section. We begin by
introducing how various aspects of trustworthiness are being implemented in
case studies of different AI applications. We then briefly review the recent
progress towards trustworthy AI in multidisciplinary areas including research,
engineering, and regulation. We then show that a systematic framework is
necessary to organize all these efforts.

Applications | Requirements | Literature
---|---|---
Face Recognition | Robustness |  (ramachandra2017presentation, ) (chakraborty2014overview, ) (xiao2021improving, ; guo2021meaningful, ; komkov2021advhat, ; sharif2019general, )
Generalization |  (yang2019learning, ) (wang2019racial, )
Fairness |  (drozdowski2020demographic, ) (gong2021mitigating, ) (wang2019decorrelated, )
Explainability | (yin2019towards, ; williford2020explainable, )
Privacy Protection | (aggarwal2021fedface, ; bai2021federated, )
Autonomous Driving | Robustness |  (huang2016autonomous, ; horwick2010strategy, ) (eykholt2018robust, ; cao2019adversarial, ) (wood2019safety, ; horwick2010strategy, )
Generalization | (zhang2018deeproad, ; sakaridis2019guided, )
Transparency | (cysneiros2018software, )
Explainability | (zablocki2021explainability, ; likmeta2020combining, )
Accountability | (perez2010argos, ; yao2020smart, )
NLP | Robustness | (jia2020building, )
Explainability | (wiegreffe2021teach, )
Fairness | (google2018reducing, ; tay2014stereotypes, ; unesco2019blush, )
Value Alignment | (chkroun2018did, ; park2021use, ; chai2020keep, ; elgan2018kidsalexa, ; rudgard2018alexachildren, ; xinhua2020savelife, )
Table 1. Trustworthiness is raising concern in various scenarios of AI
applications. We summarize representative papers addressing different aspects
of trustworthy AI in applications of face recognition, autonomous driving, and
NLP.

###  3.1. Industrial Case Study

####  3.1.1. Face Recognition

Face recognition has been one of the earliest AI techniques widely adopted in
real-world scenarios. Compared to other biometric measurements, facial
features provides a much more convenient interface for users to leverage for
identification. However, the recent widespread applications of face
recognition bring new challenges and risks such as malicious attacks and
privacy breaches. From the perspective of building trustworthy AI, we show
recent representatives of these problems and their solutions.

Face recognition has been widely adopted in security-critical scenarios such
as border control and e-bank authentication. These applications raise concerns
about the robustness of face recognition against malicious attacks. Naive face
recognition can be easily spoofed by a photo of a person targeted for identity
theft. This is known as the presentation attack on the face recognition system
(ramachandra2017presentation, ). Liveness detection (chakraborty2014overview,
) is a typical approach to defend against presentation attacks by photos or
displays. However, liveness detection is still vulnerable against new types of
attacks by fake videos generated by AI techniques like DeepFake. To
comprehensively defend against presentation attacks, a number of new research
directions have aroused attention from the AI community, including liveness
detection, face anti-spoofing, and synthetic image detection
(rossler2019faceforensics, ). Besides presentation attacks, adversarial
attacks against face recognition have recently been proved as a new type of
threat. Research has shown that face recognition systems can be easily spoofed
by impersonation attackers wearing a small printed adversarial patch
(xiao2021improving, ; guo2021meaningful, ; komkov2021advhat, ;
sharif2019general, ). Concerns on this new type of risk have made the
adversarial attack and defense in face recognition an important application
research direction of adversarial ML.

Modern face recognition systems are known to be sensitive to domain shift of
input data such as the change of acquisition sensors or racial groups.
Conventionally the mitigation of this performance degeneracy requires a large
of annotated data from different domains. To overcome the problem of data
scarcity, domain generalization techniques have been adopted in face
recognition scenarios to enable unsupervised training. For example, some
approaches cluster unlabeled face images to generate pseudo labels for
training (yang2019learning, ). Domain adaption techniques are also studied to
enhance the generalizability of face recognition models between different data
domains (wang2019racial, ).

Fairness issue is another challenge which modern face recognition systems are
confronted with. It is known that conventional face recognition algorithms are
biased in performance on different groups of genders, races or ages
(drozdowski2020demographic, ). This problem not only harms the user experience
of specific groups, but also harms the societal trust on AI. To mitigate bias,
debiasing algorithms have recently aroused interest in face recognition
research. Representatives either develop training strategies to balance
performance between groups (gong2021mitigating, ) or extract features which
are invariant across groups (wang2019decorrelated, ).

Compared to the conventional image classification problem, the explainability
of face recognition is less studied in the past years. However, the value of
explainable face recognition should not be neglected in practical high-stake
applications where the explanation of model decision helps build trust.
Research has explored some potential directions to interpret face recognition
procedures (yin2019towards, ; williford2020explainable, ) by enforcing
internal features to express explainable structures.

Last but not the least, the sensitivity of face data arouses the concern of
privacy leakage for modern face recognition systems. A typical case is that
deep learning models are known to leak training face data when attacked by
“deep leakage” (zhu2018deep, ). Privacy-preserving techniques like federated
learning provide a solution to extract knowledge from distributed user data
while preserving the identification-sensitive information
(aggarwal2021fedface, ; bai2021federated, ).

####  3.1.2. Autonomous Driving

Autonomous driving is one of the most representative AI systems applied in
safety crucial scenarios. Its significant impact on social life raises global
concern on ensuring its trustworthiness. Various efforts have been made in
different aspects to address this problem, which actually provides valuable
practice of building a trustworthy AI system over its full lifecycle.

Robustness is one of the most critical requirements for an autonomous driving
vehicle. Poor model performance as well as external attacks
(eykholt2018robust, ; cao2019adversarial, ) both threaten its safety. To
address this issue, the combination of multiple sensors including cameras,
lidars, etc, is widely adopted when designing a state-of-the-art autonomous
driving vehicle (chen2017multi, ). At the development stage, to validate the
system robustness in a large number of traffic scenarios, simulation-based
testing is widely used. Such simulation not only includes software simulation
like ParSim, CARLA, etc, but also includes hybrid realization with hardware-
in-the-loop or vehicle-in-the-loop design (rosique2019systematic, ). In
addition, a fallback plan (wood2019safety, ; horwick2010strategy, ) is also
required as an essential component on autonomous driving products. Inherited
from the strict safety requirement in the automotive industry, standards like
ISO 26262 also provide effective guidelines on ensuring the robustness and
safety of the autonomous driving system. The safety-critical character of
autonomous driving further raises the requirement of auditability. Inspired by
the success of flight data recorders in aviation, event data recorders for
autonomous driving (perez2010argos, ; yao2020smart, ) are proposed to collect
audit trails.

Transparency and explainability are also important requirements for autonomous
driving systems. At the product design stage, a clear specification of the
functionality, e.g., traffic jam pilot (TJP), and autonomy level, i.e., L1–L5,
provide the basic transparency of the functionality boundary of an autonomous
driving system. At the algorithm level, though DNN has achieved impressive
performance in autonomous driving tasks like perception and planning, its lack
of explainability still places a demand to require explicit explanations or
impose rules to ensure that the vehicle is behaving in a regulated manner
(zablocki2021explainability, ; likmeta2020combining, ).

Last but not least, ethics and value alignment have always been concerns along
with the evolving autonomous driving technology. For example, besides human
drivers, the AI driver is now also often challenged by the public with the
well-known trolley dilemma (piano2020ethical, ). Based on different
objectives, autonomous driving system might make different decisions to
minimize the overall physical harm or protect those who obey traffic rules.
Debates on different decision objectives reflect the ambiguity of human
morality, especially fairness in this case and lead to new challenges to build
the trustworthiness of autonomous driving systems to the society.

####  3.1.3. Natural Language Processing

Unlike autonomous driving, natural language processing (NLP) applications are
often less safety-critical. However, its widely available applications have
challenged trustworthiness from different aspects, especially the alignment of
human values and ethics. As more research and media attention is directed to
these issues, industry practitioners have also introduced methods to address
these concerns recently.

Society has noticed a series of negative events caused by the lack of ethical
consideration in conversational AI products. A notable example of the breach
in trust is the prevalent gender bias in machine translation. Specifically,
when translating from languages that do not have gender distinction for third-
person pronouns (e.g., Turkish) into one that does (e.g., English),
translations usually exhibit bias that might stem from the data (e.g., “He is
a doctor. She is a nurse.” even if gender cannot be inferred from the original
text). Providers such as Google have taken steps to provide alternative
gendered translations in these cases to help mitigate such issues
(google2018reducing, ).

On the other hand, systems that are designed by taking human values into
consideration can also help avoid unforeseen risks. For instance,
conversational AI systems that help serve customers in e-businesses could be
equipped with flagging mechanisms for risky inquiries or unstable mental
conditions of its users, and help us intervene in suicide attempts and work to
improve social well-being (xinhua2020savelife, ).

Last but not least, voice assistants are becoming more and more prevalent in
our daily lives, helping us operate internet of things devices in smart homes,
as well as conveniently automating our daily workflows. However, most
commercially available voice assistants (Apple Siri, Amazon Alexa, Google
Assistant, etc.) were made exclusively available in female voices, or used
female personas by default (tay2014stereotypes, ; unesco2019blush, ). This
could potentially strengthen gender stereotypes, and potentially associate
aggressive user behaviors with feminine service personnel in real life, which
is especially concerning when a new generation of children grow up around
these devices (elgan2018kidsalexa, ; rudgard2018alexachildren, ). More
recently, most major voice assistant service providers have made them
available in more voice options, which helps these systems better align with
human values by reducing the implicit gender inequity.

###  3.2. Recent Multidisciplinary Progresses

As can be seen from the above recent practices in different applications,
establishing AI trustworthiness requires consideration, design, development,
and governance throughout different stages of the system lifecycle to fulfill
the requirements as mentioned in Section 2. This fact has motivated efforts
from various fields to building trustworthy AI. In this section, we present
the recent representative progresses and demonstrate that a framework to
organize these fields along the AI system lifecycle is a necessity.

####  3.2.1. Multidisciplinary Research

Trustworthy AI is more than a subdiscipline of computer science or ML. Its
profound impact on human society has derived various research topics demanding
prompt solutions.

Societal understanding of trustworthy AI. As a problem emerging with the rise
of AI, our understanding about trustworthy AI is also developing.
Trustworthiness is a reflection of both social welfare and human psychology.
With the deepening impact of AI on human life and society, it becomes an
imperative area in social science to research the trustworthy relationship
between humans and AI.

The impact of AI on human society is profound and double-edged. It becomes an
important problem to understand and forecast both the negative and positive
impact of AI. Literature such as (kaloudi2020ai, ; king2020artificial, ;
yampolskiy2019predicting, ) has collected and studied representative AI misuse
cases and summarized high-stake patterns which AI failures tend to follow
(yampolskiy2019predicting, ), such as vulnerability to attacks,
underperformance on noises, and biased prediction. Understanding these
patterns naturally derives a number of specific requirements of AI
trustworthiness, such as robustness, generalization, and fairness. Besides
lessons from the AI failures, potential areas for AI to make positive impact
are also important to study (tomavsev2020ai, ; hager2019artificial, ). For
example, as noted in (tomavsev2020ai, ; qi2021conversational, ), various
scenarios have been studied to apply AI to advance the United Nations’
Sustainable Development Goals (SDGs) for social good.

In addition to the social impact of AI application, the societal and
psychological outline of trustworthiness should be a prerequisite for the
establishment of trustworthy AI. A main aspect of this study starts by
analyzing the risk and threat of AI and derives the requirements of
trustworthiness (thiebes2020trustworthy, ). Beyond empirical analysis, the
formalization of AI trustworthiness is also studied. For example,
(jacovi2021formalizing, ) formalizes human-AI trust by considering and
extending the concepts of interpersonal trust. In terms of specific dimensions
of trustworthy AI, previous works such as (gunning2019darpa, ;
mohseni2018multidisciplinary, ; miller2019explanation, ) have studied the
problem of explainability and transparency from philosophy, cognitive
psychology, and social psychology. Analysis of ethical values regarding AI
application is also an important problem in the aspect of social science and
can be found in recent literature such as (piano2020ethical, ;
greene2019better, ; bankins2021ethical, ; mittelstadt2019principles, ).

Technical approaches. As is shown in Section 2, AI trustworthiness is
decomposed as the fulfillment of a series of requirements. It is recognized
that technology improvement serves as a core solution to many of these
requirements, such as robustness, generalization, explainability, fairness,
and privacy. In the last decade, these requirements have derived a variety of
active directions in AI research. Recently there has been some literature
which briefly surveys the related technology to facilitate the establishment
of trustworthy AI. For example, (toreini2020relationship, ;
liu2021trustworthy, ) list example trustworthy research directions in the area
of ML algorithms. (wickramasinghe2020trustworthy, ) surveys trustworthy AI
development techniques from the aspect of human-system interaction.
(kumar2020trustworthy, ) comprehensively decomposes requirements of
trustworthy AI to executable techniques that are of the current interest in
the community of AI research. Due to the wide coverage of trustworthy AI
related techniques, it is difficult to exhaustively survey its every dimension
in a single paper. Fortunately, there have been surveys available which covers
several subdivisions of research mentioned in this paper. We refer to these
surveys in Sections 2 and 4.

The establishment of trustworthy AI systems involves the development of both
AI algorithms and engineering techniques. Take the requirement of robustness
as an example. Adversarial attack and defense are becoming popular research
directions. They have been studied in various high-stake tasks such as face
recognition (komkov2021advhat, ), autonomous driving (cao2019adversarial, ),
and recommendation systems (li2016data, ). Besides algorithms to enhance model
robustness against attacks, engineering techniques are increasingly raising
concern to enhance system robustness. At the development stage of the system,
comprehensive testing on malicious inputs serves as an essential guarantee to
avoid malfunction on attacks. At the deployment stage, a fail-safe mechanism
prevents the system to cause loss even if malfunction happens.

Non-technical approaches. The achievement of trustworthy AI requires more than
just technical approaches. The creation and operation of AI systems involve
the participation of multiple roles including researchers, software
developers, business operators, and regulators. The system lifecycle ranges
over steps of product design, algorithm design, system development,
deployment, and business operation. It becomes a prerequisite of trustworthy
AI to align the understanding and efforts of these various roles and steps,
which raises the demand for appropriate management strategies.

Literature such as (shneiderman2020bridging, ; baker2021trustworthy, ;
rakova2021responsible, ) has pointed a series of reforms to the organization
of participants and procedures to build trustworthy AI. These reforms include
responsible product design aligned with trustworthiness principles
(amershi2019guidelines, ), new human resource strategies
(shneiderman2020bridging, ; rakova2021responsible, ), and justification of
trustworthiness via continuous auditing (shneiderman2020bridging, ). We
continue to consider the aforementioned example of enhancing AI robustness.
Algorithm researchers and software engineers should be aligned with the same
requirement of robustness so that they apply appropriate techniques in their
area to prevent the risk of attacks. In high-stake applications, safety
engineers (siddique2020safetyops, ) should be involved to oversight the
lifecycle of a system and assess potential risks. Further, business operators
should be aware of the working condition and limitation of the system to avoid
exceptional use cases. When a new type of attacks happen, feedback from
business operators to system manufacturers serves as a prerequisite to upgrade
the system against the problem. Besides direct participants in the system,
third-party auditors provide independent certification of robustness and
security, which serves as an external driving force to the fulfillment of
trustworthiness requirements (zicari2021z, ; raji2019ml, ).

####  3.2.2. Engineering Practices

With the development of the study on trustworthy AI, a number of common
approaches in building AI trustworthiness have been organized as tools to
address various aspects of AI trustworthiness.

Development tools. Various toolboxes have been released in the past years to
assist the algorithm experimentation and software development to enhance AI
trustworthiness in different aspects such as fairness (bellamy2018ai, ;
bird2020fairlearn, ; saleiro2018aequitas, ), XAI (nori2019interpretml, ;
github-xai, ; arya2019one, ), and adversarial robustness
(nicolae2018adversarial, ; github-robustness, ). For the management stage,
tools and references are released to facilitate the improvement of
documentation and thus enhance transparency (arnold2019factsheets, ).
Development platforms such as CodaLab provide workspaces for reproducible AI
experimentation and development. Tools to test and audit the system
trustworthiness in specific requirements such as fairness (wilson2021building,
) provide further verification to the system development.

To manage and evaluate different tools, the OECD Network of Experts on
Implementing Trustworthy AI is working on a generic framework to help
practitioners choose suitable tools for their specific use case (OECD-tools,
). The OECD has conducted a survey and shows that three types of tools are of
the common focus: 1) Technical tools aiming to address technical aspects such
as the aforementioned robustness, explainability, and bias detection; 2)
Procedural tools providing implementation guidance; 3) Educational tools
training stakeholders of the AI systems.

Development frameworks. As can be seen from the above examples and
categorization, existing tools reflect the consideration of trustworthy AI at
different stages in the lifecycle of an AI system, such as algorithm design,
development, and management (aihleg2018ethics, ). Each of these tools mainly
focuses on a specific requirement of trustworthy AI and improves a specific
stage of the lifecycle. To build a complete trustworthy AI system, there is a
necessity to select and organize these tools systematically.

Some institutions have released their collection of tools as a framework to
facilitate the development of trustworthy AI systems, e.g., Amazon SageMaker,
IBM OpenScale. We take Google’s Responsible AI with TensorFlow as a
representative practical collection of modules considering requirements of
trustworthy AI at multiple stages when building AI systems (google-ai, ). The
People+AI Research (PAIR) Guidebook provides guidelines and best practices to
better understand the human factors when designing an AI product. At the model
development stage, libraries and tools are provided within the framework to
enhance the explainability, fairness, and privacy of the model. Tool-chains
are also available to track the meta information regarding the training
process (mitchell2019model, ). Responsible AI reflects consideration of AI
trustworthiness from the perspective of the AI system lifecycle. It provides
well-developed libraries which can be directly adopted at the stages of
problem definition, data preparation, model training, model evaluation, and
model deployment. However, considering the case studies aforementioned in this
section, this collection of tools is still insufficient to guide and support
the industrial practices to address trustworthy AI through the system
lifecycle.

####  3.2.3. Institutionalization

Though the value of trustworthy AI has been widely recognized, the motivation,
interpretation, and consideration on AI trustworthiness vary over different
stakeholders. Such inconsistency is even more evident in the industry due to
the commercial consideration. Even within a single company, different
departments of such as business, technology, and legality are likely to hold
different interpretation on requirements of AI trustworthiness. To address
this issue, guidelines, standards, and regulations become necessary to unify
the understanding of different AI stakeholders.

Guidelines. To address regulate the development of AI, a variety of guidelines
and principles have been proposed by governmental organizations in recent
years. Representatives of these guidelines include the EU Ethics Guidelines
for Trustworthy AI (aihleg2018ethics, ). Besides governmental organizations, a
wealth of companies also release their guidelines or principles to demonstrate
their compliance with AI trustworthiness. Thorough review and discussion of
these guidelines can be found from e.g., (jobin2019global, ; schiff2021ai, ;
hagendorff2020ethics, ). Though divergence exists, it is shown that global
convergence on the requirement of AI trustworthiness is recognized as
summarized in Section 2.

Standardization. With various ethics guidelines released globally as soft
recommendations for AI development and converges to consensus principles,
standards to concretize these principles in specific AI application areas
serve as the further means to regulate the trustworthiness of AI products
(zielke2020artificial, ). Multiple standardization activities are ongoing
parallelly, including IEEE P7000 and ISO/IEC SC 42 (lewis2021ontology, ),
which specifically focuses on AI trustworthiness. There are multiple different
aspects addressed in the current standardization. For example, at the holistic
level, ISO/IEC SC 42 Working Group 3 (WG3) works on establishing
trustworthiness for general ML systems through transparency, robustness,
explainability, etc (lewis2021ontology, ). Besides generic AI-centered
standards, certain applications of AI should also seek compliance with domain-
specific standards to build trustworthy AI products, e.g., ISO 26262 for the
functional safety of intelligent vehicles and ISO 14155 for clinical practice
on medical devices.

Regulation. Governmental legislation is catching up its pace with the rapid
development of AI technology and industrial application. Recently EU releases
its proposal for Artificial Intelligence Act (devulder2021world, ). A notable
point of the act is the requirement to execute risk management approaches
throughout different stages in the lifecycle of an AI system. These approaches
include but not limited to data governance, testing for risks, detailed
documentation, logging, and human oversight. Besides regulation on AI systems,
it should also be noted that a wealth of regulation in the area of general
information technology is also bringing significant impact on the AI industry,
such as GDPR, the Chinese Cybersecurity Law, and the Personal Information
Security Specification.

####  3.2.4. Towards systematic organization

As can be seen from the above recent practices, the establishment of
trustworthy AI requires efforts at different stages in the lifecycle of an AI
system. This fact motivates the need to organize all these fragmented efforts
in a unified framework. A number of works have conducted studies on such
frameworks, mainly on a specific area such as technique or management. For
example, (kumar2020trustworthy, ) briefly reviews executive techniques and
organizes them by the specified requirement dimensions of trustworthy AI.
(wickramasinghe2020trustworthy, ) categorizes the interactions between AI
systems, developers, and users to organize the technical approaches and
responsibility in the lifecycle of trustworthy AI. (thiebes2020trustworthy, )
simplifies the system lifecycle as the flow of data and technically analyzes
threats existing at each flow stage. The aforementioned tool-chains such as
Google Responsible AI also reflect the idea of organizing existing techniques
in the frame of system development lifecycle. In terms of management,
(baker2021trustworthy, ) studies a management workflow to guarantee that
ethics are aligned throughout the creation and update of an AI system.
(raji2020closing, ) specifically organizes auditing approaches for the
management of system lifecycle.

Whilst the above existing literature has studied the organization of technical
and management approaches to build trustworthy AI, we argue that these works
are still insufficient from the industrial perspective. Considering the case
study of e.g., autonomous driving in Section 3.1, this research of frameworks
has not yet covered all aspects which should be addressed to build
trustworthiness. As depicted in Section 1 and Figure 2, the AI industry holds
a connecting position to organize efforts from multidisciplinary fields in the
establishment of trustworthy AI. This position requires industrial
stakeholders to learn and organize the diverse approaches from areas of
academic research, engineering, and management so that trustworthy AI can be
ensured throughout the lifecycle of products. By surveying the above previous
works, we find a framework covering all these areas is still lacking. In the
next section, we push a step forward towards this requirement by a
multidisciplinary survey of approaches to build trustworthy AI. These
approaches cover active research areas, engineering techniques, and management
strategies and are organized in the frame of an industrial system lifecycle. A
workflow is then proposed to integrate these approaches and facilitate dynamic
feedback and upgrade from academia and governmental regulation over these
approaches.

Figure 2. The AI industry holds a connecting position to organize
multidisciplinary practitioners, including users, academia, and government in
the establishment of trustworthy AI. In Section 4, we discuss the current
approaches towards trustworthy AI in the five main stages of the lifecycle of
an AI system, i.e., data preparation, algorithm design, development,
deployment and management.

##  4\. Trustworthy AI: A Systematic Approach

We have introduced the relevant concepts of trustworthy AI in Section 2 and
discussed trustworthy AI-related issues in exemplar industry applications in
Section 3. In this section, we conduct a brief survey of the related
techniques used for building trustworthy AI products, and organize them across
the product development lifecycle in an industrial perspective. As shown by
the solid lined boxes in Figure 2, the lifecycle of the development of a
typical AI product can be partitioned into data preparation, algorithm design,
development-deployment, and management (ashmore2019assuring, ). We review
several critical algorithms, guidelines, and government regulations that are
closely relevant to the AI product trustworthiness in each lifecycle stage,
aiming to provide a systematic approach and an easy-to-follow guide to
practitioners from varying backgrounds to establish trustworthy AI.

###  4.1. Data Preparation

Currently, AI technology is largely data-driven. Appropriate management and
exploitation of data not only improves the performance of an AI system, but
also affects its trustworthiness. In this section, we consider two major
aspects of data preparation, i.e., data collection and data preprocessing. We
discuss the corresponding trustworthy AI requirements.

####  4.1.1. Data Collection

Data collection is a fundamental stage of the lifecycle of AI systems.
Elaborately designed data collection strategy could conduce to the enhancement
of AI trustworthiness such as fairness and explainability.

Bias mitigation. One common source of bias is the training and evaluation data
used to develop AI systems. Many different sources of bias could plague
fairness in data collection, requiring different processes and techniques to
combat it (see (mehrabi2019survey, ) for a comprehensive survey). Bias
mitigation techniques during data collection can be divided into two broad
categories: debias sampling and debias annotation. The former concerns which
data points to use or to annotate, while the latter focuses on choosing the
right annotators. When sampling data points to annotate, we note that a
dataset reflecting the user population does not guarantee fairness, since
statistical methods and metrics tend to favor majority groups and amplify
inequality. This bias can be further amplified if the majority group is more
homogeneous for the task (e.g., recognizing speech in less-spoken accents can
be naturally harder due to data scarcity). System developers should therefore
take task difficulty into consideration when developing and evaluating fair AI
systems. On the other hand, choosing the right annotators is especially
crucial for underrepresented data (e.g., when annotating speech recognition
data, most humans are also bad at recognizing rarely heard accents).
Therefore, care must be taken in selecting the correct experts, especially
when annotating data from underrepresented groups, to prevent human bias from
creeping into annotated data.

Explanation collection. Aside from model design and development, data
collection is also integral in building explainable AI systems. As will be
mentioned in Section 4.2.2, adding an explanation task to the AI model
effectively explains the intermediate features of an AI model. This strategy
is used in tasks like NLP-based reading comprehension by generating supporting
sentences (tu2020select, ; yang2018hotpotqa, ). To train the explanation task,
it is helpful to consider collecting explanations or other side-information
that may not directly be part of the end task either directly from annotators
(wiegreffe2021teach, ) or with the help of automated approaches
(kim2016examples, ).

Data Provenance. Data provenance requires to record the data lineage,
including the source, dependencies, contexts, and processing steps of data
(singh2018decision, ). By tracking data lineage at the highest resolution,
data provenance provides an effective enhancement to the transparency,
reproducibility, and accountability of an AI system (herschel2017survey, ;
janssen2020data, ). Moreover, recent research also shows data provenance can
be used to mitigate data poisoning (baracaldo2017mitigating, ), enhancing the
robustness and security of an AI system. Technical realization of data
provenance can be found in (herschel2017survey, ). Tool chains
(schelter2017automatically, ) and documentation (gebru2018datasheets, ) guides
are also studied regarding the specific scenario of AI systems. To further
ensure provenance to be tamper-proof, blockchain is recently considered as a
promising tool to certify data provenance in AI (dillenberger2019blockchain, ;
sarpatwar2019towards, ; alshamsi2021artificial, ).

Figure 3. A look-up table which organizes the surveyed approaches to realize
AI trustworthiness from different aspects and at different stages of the AI
system lifecycle. Some approaches might be adopted to improve AI
trustworthiness at multiple aspects and exists in multiple columns. We dim
these duplicated blocks by stride filling for better visualization. See the
corresponding paragraphs in Section 4 for the details of approaches.

####  4.1.2. Data Preprocessing

Data preprocessing has been recognized as an essential step in practical ML
systems. Before feeding data into an AI model, data preprocessing helps remove
inconsistent pollution from the data which might harm model behavior and
sensitive information which might leak privacy.

Anomaly Detection. Anomaly detection is also known as outlier detection and
has been a lasting yet active research area in ML (tae2019data, ; chu2016data,
; chalapathy2019deep, ; pang2020deep, ). Due to the sensitivity of ML models
to outlier data, data cleaning by anomaly detection serves as an effective
approach to enhance AI performance. In recent studies, anomaly detection is
also shown to be useful in addressing some requirements of AI trustworthiness.
For example, fraud data have been a concerning problem which challenges the
robustness and security of systems in areas such as banking and insurance.
Various approaches have been proposed to address this issue by anomaly
detection (chalapathy2019deep, ). The detection and mitigation of adversarial
inputs is also adopted as an effective means to defend against evasion attacks
and data poisoning attacks (silva2020opportunities, ; li2020backdoor, ;
akhtar2018threat, ), which is also referred as data sanitization
(chan2018data, ; paudice2018label, ; cretu2008casting, ).

Data Anonymization (DA). DA modifies the data so that the protected data
information can not be recovered. Data format-specific DA approaches have been
studied for decades (yuan2011protecting, ; zhang2013scalable, ;
jakob2020design, ; kushida2012strategies, ). For example, the privacy
information of social network graph data can be potentially contained in
vertex properties and labels, link relationship, weight and labels, or graph
metrics (zhou2008brief, ). How to anonymize such kind of data is introduced in
the literature (liu2008towards, ; majeed2020anonymization, ; beigi2020survey,
). Specific DA methods have also been designed for relational data
(pervaiz2013accuracy, ), set-valued data (terrovitis2008privacy, ;
he2009anonymization, ) and image data (maximov2020ciagan, ; dirik2014analysis,
), etc. Guidelines and standards have been made for data anonymization, such
as the US HIPAA and the UK ISB1523. Data pseudonymization
(mourby2018pseudonymised, ) is a relevant technique, which is promoted by
GDPR. It specifically replaces the privacy information with nonidentifying
references.

Desirable data anonymization is expected to be immune from the data de-
anonymization or re-identification attacks that try to recover the privacy
information from the anonymized data (el2011systematic, ; ji2014structural, ).
Taking the graph data as an example, there has been seed-based and seed-free
algorithms (ji2016graph, ). Seed-based algorithms first identify several users
as seeds then more users are de-anonymized from these seeds. Seed-free
algorithms de-anonymize users without finding seeds first. In (ji2015secgraph,
), an open-source platform is provided to evaluate the privacy protection
performance of graph data anonymization algorithms against de-anonymization
attacks.

Differential privacy (DP). DP is a theoretically rigorous data anonymization
method (dwork2006calibrating, ; dwork2014algorithmic, ; dwork2016calibrating,
). An algorithm satisfying DP definition roughly suggests one cannot tell from
the algorithm output whether any individual sample is used in the computing.
Utilizing DP technique has been explored in various data publishing tasks such
as log data (hong2014collaborative, ; zhang2016anonymizing, ), set-valued data
(chen2011publishing, ), correlated network data (chen2014correlated, ),
crowdsourced data (wang2016real, ; ren2018textsf, ), etc. It has also been
applied in single and multi-machine computing environments, integrated with
machine learning models for protecting model privacy or improving model
robustness (fletcher2019decision, ; wang2019differentially, ; xu2019achieving,
; abadi2016deep, ; lecuyer2019certified, ; han2016differentially, ;
wei2020federated, ). Research studies on attacking the DP algorithm are seen
in the literature (cormode2011personal, ; haeberlen2011differential, ).

The local DP algorithm has been adopted by Apple to transform the user data
into the form that the true data cannot be reproduced. In
(erlingsson2014rappor, ), researchers propose RAPPOR algorithm and prove it
satisfies DP definition. The algorithm is used for crowdsourcing statistical
user software analysis. DP is also used to improve AI model robustness toward
adversarial samples (lecuyer2019certified, ).

Lifecycle | Approaches | Literature
---|---|---
Data Preparation | Data Collection | Bias Mitigation (mehrabi2019survey, ; caton2020fairness, ; verma2018fairness, )
Data Provenance (herschel2017survey, ; janssen2020data, )
Data Preprocessing | Anomaly Detection (tae2019data, ; chu2016data, ; chalapathy2019deep, ; pang2020deep, )
Data Anonymozation (al2015literature, ; majeed2020anonymization, )
Differential Privacy (dwork2008differential, ; dwork2014algorithmic, ;
ji2014differential, )
Algorithm Design |  Adversarial Robustness | Adversarial Robustness (silva2020opportunities, ; akhtar2018threat, ; chakraborty2018adversarial, ; amodei2016concrete, ; yuan2019adversarial, ; bhambri2019survey, ; wang2019convergence, )
Poisoning defense (li2020backdoor, )
Explainability ML | Explainability ML (arrieta2020explainable, ; guidotti2018survey, ; bodria2021benchmarking, ; dovsilovic2018explainable, ; molnar2020interpretable, ; dovsilovic2018explainable, )
Model Generalization | Model Generalization (friedman2001elements, ; goodfellow2016machine, ; zhang2016understanding, ; arpit2017closer, ; krueger2017deep, ; kawaguchi2017generalization, ; belkin2018reconciling, )
Domain Generalization (zhou2021domain, ; wang2021generalizing, )
Algorithmic Fairness | Fairness and bias mitigation (mehrabi2019survey, ; caton2020fairness, ; verma2018fairness, )
Privacy Computing | SMPC (zhao2019secure, ; evans2017pragmatic, )
Federated Learning (yang2019federated, ; MAL-083, )
Development | Functional Testing | (huang2020survey, ; zhang2020machine, ; marijan2020software, )
| Performance
---
Benchmarking
(croce2020robustbench, ; hendrycks2019benchmarking, ; brun2018software, ;
galhotra2017fairness, ; tramer2017fairtest, ; bellamy2018ai, ;
friedler2019comparative, ; sixta2020fairface, ; mathew2020hatexplain, ;
deyoung2019eraser, ; mohseni2018human, )
Simulation | (koenig2004design, ; todorov2012mujoco, ; dreossi2019verifai, ; wymann2000torcs, ; benekohal1988carsim, ; tideman2010scenario, ; lee2019convlab, ; bokc2007validation, ; brogle2019hardware, )
Formal Verification | (urban2021review, ; huang2020survey, ; seshia2016towards, )
Deployment | Anomaly Monitoring | (chalapathy2019deep, ; pang2020deep, ; vzliobaite2016overview, )
Human-AI Interaction | (shneiderman2020bridging, ; chatzimparmpas2020state, ; kafle2020artificial, ; torres2018accessibility, )
Fail-Safe Mechanism | (fu2014monocular, ; muller2016increased, ; horwick2010strategy, ; magdici2016fail, )
Hardware Security | (xu2021security, ; sabt2015trusted, ; pinto2019demystifying, )
Management | Documentation | (aimar1998introduction, ; mitchell2019model, ; gebru2018datasheets, ; bender2018data, ; holland2018dataset, ; arnold2019factsheets, )
Auditing | (raji2020closing, ; buolamwini2018gender, ; madaio2020co, ; schelenz2020applying, ; reisman2018algorithmic, ; sandvig2014auditing, ; wilson2021building, )
Cooperation | (askell2019role, ; dafoe2020open, )
Workflow | MLOps | (makinen2021needs, ; hummer2019modelops, ; siddique2020safetyops, )
Table 2. Representative papers of approaches or research directions mentioned
in Section 4. For research directions which are widely studied, we provides
the corresponding survey papers for readers to further learn the directions.
For approaches or research directions without survey available, we provides
representative technical papers.

###  4.2. Algorithms Design

A number of aspects of trustworthy AI have been addressed as algorithmic
problems in the community of AI research and have attracted widespread
interest. We organize recent technical approaches by their corresponding AI
trustworthiness aspects, including robustness, explainability, fairness,
generalization and privacy protection, for quick reference by practitioners.

####  4.2.1. Adversarial Robustness

It is recognized that the robustness of an AI model is majorly affected by the
training data and the algorithm (sambasivan2021everyone, ). We describe
several representative and active directions in this section. More
comprehensive survey can be found in specific literature such as
(silva2020opportunities, ; akhtar2018threat, ; chakraborty2018adversarial, ;
amodei2016concrete, ; li2020backdoor, ; yuan2019adversarial, ;
bhambri2019survey, ).

##### Adversarial training

Since the early discovery of adversarial attacks, it is recognized that
augmentation of training data with adversarial samples provides an intuitive
approach for defense, which is typically referred as adversarial training
(goodfellow2014explaining, ; moosavi2016deepfool, ; wang2019convergence, ;
li2019improving, ). The augmentation can be done in a brute-force manner by
feeding both original data and adversarial samples during training
(kurakin2016adversarial, ), or by denoting a regularization term to implicitly
represent adversarial samples (goodfellow2014explaining, ). Conventional
adversarial training augments data with a specific attack approach and is
majorly defensive to this corresponding attack approach but vulnerable to
others. Various improvements have been studied to achieve further defense
(bhambri2019survey, ). (tramer2017ensemble, ) augments training data with
adversarial perturbations transferred from other models. It is shown to
further provide defense against black-box attacks which do not require
knowledge of the model parameters and thus help mitigate greater risks.
(maini2020adversarial, ) combines multiple types of perturbations into
adversarial training and improves model robustness against multiple types of
attacks.

##### Adversarial regularization

Besides the regularization term which implicitly represents adversarial
samples, recent research has explored more network structures or
regularization to overcome the vulnerabilities of DNN on adversarial attacks.
An intuitive motivation of this regularization is to avoid the network outcome
to change dramatically at small perturbations. For example, (gu2014towards, )
penalizes large partial derivatives of each layer to improve its output
stability by using additional penalty on the magnitude of model Jacobian.
Similar regularization on gradients is also adopted by (ross2018improving, ).
Parseval networks (cisse2017parseval, ) train the network by imposing a
regularization on the Lipschitz constant at each layer.

Certified bound of model prediction is a special type of adversarial
regularization. As mentioned in Section 2.1, certified robustness provides a
formal evaluation on the robustness of a model. This study of evaluation also
inspires new regularization terms for defensive training. For example,
(hein2017formal, ) derives a lower bound on the norm of the input manipulation
required to change the classifier decision and uses it as a regularization
term for robust training. To address the issue that exact computation of such
lower bound is intractable for large networks, various relaxation or
approximation such as (zhang2019towards, ; weng2018evaluating, ) has been
proposed as alternative regularization.

##### Poisoning defense

Typical poisoning or backdoor attacks contaminate training data to mislead the
model behavior. Besides avoiding suspicious data at the data sanitization
stage, defensive algorithms against poisoning data are also an active area of
recent ML research (li2020backdoor, ). The defense is studied at different
stages of a DNN model. For example, based on the observation that the
backdoor-related neurons are usually inactivated for benign samples,
(liu2018fine, ) proposed to prune those neurons from a network to remove the
hidden backdoor. Neural Cleanse (wang2019neural, ) proactively discovers
backdoor patterns in a model. The backdoor can then be avoided by early
detection of backdoor patterns from data or retraining the model to mitigate
backdoor. Diagnosis of a model to detect backdoor attacks also serves as an
effective approach to avoid potential risks and vulnerabilities
(kolouri2020universal, ).

####  4.2.2. Explainable ML

In this section we review representative approaches in the two aspects of ML
explainability as mentioned in Section 2.3.1 and their applications on
different modalities of tasks.

##### Explainable ML model design

Although recognized to be disadvantageous in performance, explainable models
have been an active research area in the past years and a variety of fully or
partially explainable ML models have been studied to push the performance
limit.

Self-explainable ML models. A number of self-explainable models have been
studied in machine learning over the years. The representatives include
k-nearest neighbors (KNN), linear/logistic regression, decision trees/rules,
probabilistic graphical models, etc (arrieta2020explainable, ;
guidotti2018survey, ; bodria2021benchmarking, ; dovsilovic2018explainable, ;
molnar2020interpretable, ; dovsilovic2018explainable, ). Note that the self-
explainability of these models is sometimes undermined by complexity. For
example, very complex tree or rule structures might sometimes be considered
incomprehensible or unexplainable.

Besides conventional models, some other learning paradigms are also considered
explainable, such as causal inference (kuang2020causal, ; pearl2009causal, )
and knowledge graph (wang2019explainable, ). These methods are also expected
to provide valuable inspiration to solve the problem of explainability for ML.

Beyond self-explainable ML models. Compared with blackbox models such as DNN,
conventional self-explainable models are recognized to have disadvantageous
performance on complex tasks such as image classification and text
understanding. To compromise between explainability and performance, hybrid
models which combine self-explainable models and blackbox models are
attracting growing attention. A typical design fashion is to embed an
explainable bottleneck model in a DNN. For example, previous research has
embedded linear models and prototype selection in DNN (alvarez2018towards, ;
vaughan2018explainable, ; chen2018looks, ; angelov2020towards, ). In the well-
known work Class Activation Mapping (CAM) (zhou2016learning, ), an average
pooling layer at the end of a DNN can also be regarded as an explainable
linear bottleneck. Attention mechanisms (bahdanau2014neural, ; xu2015show, )
recently also attract interest of the explainability research. In some
literature, it is regarded as an explainable bottleneck in a DNN
(choi2016retain, ; martins2016softmax, ). However, this argument is still
under debate since attention weights representing different explanations can
produce similar final predictions (jain2019attention, ;
wiegreffe2019attention, ).

##### Post-hoc model explanation

Besides designing self-explainable models, it is also an important problem to
understand how a specific decision is made by existing blackbox models. A
major part of the research on this problem addresses the methodology of post-
hoc model explanation and have proposed various approaches.

Explainer approximation aims to mimic the behavior of a given model with
explainable models. This is also referred as the global explanation of a
model. Various approaches have been proposed to approximate ML models such as
random forests (schetinin2007confident, ; zhou2016interpreting, ; tan2020tree,
) and neural networks (craven1995extracting, ; boz2002extracting, ;
augasta2012reverse, ; zhou2003extracting, ; craven1994using, ). With the rise
of deep learning in the past decade, explainer approximation on DNN is further
developed as the knowledge distillation problem on explainers such as trees
(zhang2019interpreting, ; frosst2017distilling, ).

Feature importance has been a continuously active area in explainability
research. One representative aspect uses local linear approximation to model
the contribution of each feature to the prediction. LIME (ribeiro2016should, )
and SHAP (lundberg2017unified, ) are two influential approaches which are
adopted on tabular data prediction, computer vision, and NLP. Gradients are
also shown to reflect how features contribute to the predicted outcome and
have drawn great interest in the explainability of DNN (simonyan2013deep, ;
selvaraju2017grad, ). In NLP or CV, gradients or their variants are used to
back-trace the model decision to the most related input location, which
measures importance in the form of saliency maps and sentence highlights
(sundararajan2017axiomatic, ; shrikumar2017learning, ; zeiler2014visualizing,
; molnar2020interpretable, ).

Feature introspection aims to provide a semantic explanation on intermediate
features. One representative aspect attaches an extra branch to a model to
generate a human-interpretable explanation outcome. For example, in NLP-based
reading comprehension, supporting sentence generation serves as an explanation
task in addition to the answer generation (tu2020select, ; yang2018hotpotqa,
). In image recognition, part template masks can be used to regularize feature
maps to focus on local semantic parts (zhang2018interpretable, ). Concept
attribution (bodria2021benchmarking, ) is another aspect which maps given
feature space to human-defined concepts. Similar ideas are also adopted in
generative networks to gain attribute control, such as the gender, age, and
ethnics in a face generator (liu2019stgan, ).

Example-based explanation also provides a means to explain AI model outcomes.
For example, an influential function is borrowed from robust statistics in
(koh2017understanding, ) to find the most influential data instance for a
given outcome. Counterfactual explanation (wachter2017counterfactual, ;
kim2016examples, ) works in a contrary way by finding the boundary case to
flip the outcome. This helps users better understand the decision surface of
the model.

####  4.2.3. Model Generalization

Model generalization techniques not only aim to improve model performance but
also explore to train AI model with limited data and cost. We review
representative approaches of model generalization categorized as classic
generalization and domain generalization as mentioned in Section 2.2.

##### Classic generalization mechanisms

As a fundamental principle of model generalization theory, the bias-variance
trade-off indicates that a generalized model should keep a balance between
underfitting and overfitting (belkin2019reconciling, ; friedman2001elements,
). For an overfitted model, decreasing the model’s complexity/capacity will
lead to better generalization. Taking the neural network as an example, adding
a bottleneck layer, which has fewer neurons than both below and above layers,
can help reduce the model complexity and reduce overfitting.

Other than adjusting the model architecture, one can further mitigate
overfitting for better generalization via various explicit or implicit
regularizations such as early stopping (yao2007early, ), batch normalization
(ioffe2015batch, ), dropout (srivastava2014dropout, ), data augmentation, and
weight decay (krogh1992simple, ). These regularizations are standard
techniques to improve model generalization when the training data is much less
than the number of model parameters (vapnik2013nature, ). Basically, they aim
at pushing the learning to a subspace of hypothesis with manageable
complexity, and thus reduce the model complexity (zhang2016understanding, ).
However, (zhang2016understanding, ) also come up with a new observation that
explicit regularization may improve generalization performance, but is
insufficient to reduce generalization error. The generalization of the deep
neural network is still an open problem.

##### Domain generalization

One of the challenges for modern deep neural networks is the generalization on
out-of-distribution data. This challenge arises from various practical AI
tasks and has been addressed as an active research area in the last decade
(zhou2021domain, ; wang2021generalizing, ). Model pre-training has become a
widely adopted mechanism to overcome this challenge by leveraging knowledge
learned from other domain data. For example, in the field of computer vision,
an established successful paradigm is to pre-train models on a large-scale
dataset such as ImageNet and then fine-tune them on target tasks with less
training data (girshick2014rich, ; zeiler2014visualizing, ; long2015fully, ).
It is because the pre-trained feature representation being able to transfer
information to target tasks (zeiler2014visualizing, ). Recently, unsupervised
pre-training has been very successful for language processing (e.g., BERT
(devlin2018bert, ) and GPT (radford2018improving, )) and computer vision tasks
(e.g., MoCo (he2020momentum, ) and SeCo (yao2020seco, )). In addition, self-
supervised learning provides a good mechanism to learn a cross-modality
feature representation, such as the vision and language models including VL-
BERT (su2019vl, ) and Auto-CapTIONs (pan2020auto, ). To explain the
effectiveness of unsupervised pre-training, (erhan2010does, ) conducted a
series of experiments to illustrate that unsupervised pre-training is able to
drive the learning to basins of minima that have better generalization.

Going to the extreme, the few shots learning paradigm aims at generalizing the
learning from a few labeled examples (wang2020generalizing, ). To this end,
the meta-learning framework (vanschoren2018meta, ) that learns how to learn
has been widely adopted, which basically attempt to learn prior knowledge for
generalization from a large number of similar tasks. Feature similarity
(koch2015siamese, ; snell2017prototypical, ) has been used as a representative
type of knowledge prior. Learning rules form another type of prior and have
been used in works such as MAML (finn2017model, ), reinforcement learning
method (li2016learning, ) and Memory-augmented NN (santoro2016meta, ).

####  4.2.4. Algorithmic Fairness

Methods to reduce bias in AI models during algorithm development can intervene
before data is fed into the model (pre-processing), when the model is being
trained (in-processing), or on model predictions after a model has already
been trained (post-processing).

##### Pre-processing methods

Aside from debiasing the data collection process, we can further debias data
before model training. Common approaches include:

Adjusting sample importance. This is helpful especially if debiasing data
collection is not sufficient or no longer possible. Common approaches
including resampling (adler2018auditing, ), which selects a subset of the
existing data, reweighting (calders2010three, ), which assigns different
importance to different data examples, and adversarial learning
(madry2017towards, ), which could be achieved through resampling or
reweighting with the help of a trained model to find offending cases.

Adjusting feature importance. Inadvertent correlation between features and
sensitive variables can lead to unfairness. Common debiasing approaches
include representation transformation (calmon2017optimized, ), which helps
adjust the relative importance of features, and blinding (chen2018my, ), which
omits features that are directly related to the sensitive variables.

Data augmentation. Besides making direct use of existing data samples, it is
also possible to introduce additional data samples that typically involve
making changes to existing samples, including perturbations and relabeling
(calders2010three, ; cowgill2017algorithmic, ).

##### In-processing methods

Pre-processing techniques are not guaranteed to take the desired effect during
model training, since different models could leverage features and examples in
different ways. This is where in-processing techniques could be helpful:

Adjusting sample importance. Similar reweighting (krasanakis2018adaptive, )
and adversarial learning (celis2019improved, ) techniques could apply in-
processing, with the potential of either making use of model parameters or
predictions that are not yet fully optimized to more directly debias model
optimization.

Optimization-related techniques. Alternatively, model fairness can also be
enforced more directly via optimization techniques. For instance, quantitative
fairness metrics can potentially be used to serve as regularization
(aghaei2019learning, ) or optimization constraints on model parameters
(celis2019classification, ).

##### Post-processing methods

Even if all precautions have been taken with regard to data curation and model
training, the resulting models might still exhibit unforeseen biases. Post-
processing techniques can be applied for debiasing, often with the help of
auxiliary models or hyperparameters to adjust model output. For instance,
optimization techniques (e.g., constraint optimization) could be applied to
train a smaller model to transform model outputs or calibrate model confidence
(kim2018fairness, ). Reweighting predictions of multiple models could also
help reduce bias (iosifidis2019fae, ).

####  4.2.5. Privacy Computing

Apart from the privacy-preserving data processing methods such as what we
introduce in Section 4.1.2, there is another line of methods that preserve
data privacy during the model learning process. In this part, we briefly
review the two popular algorithm categories, secure multi-party computing, and
federated learning.

Secure Multi-party Computing (SMPC) deals with the task that multiple data
owners compute a function, with data privacy protected and no trusted third
party serving as coordinator. A typical SMPC protocol satisfies the properties
including privacy, correctness, independence of inputs, guaranteed output
delivery, and fairness (zhao2019secure, ; evans2017pragmatic, ). Garbled
circuit is a representative paradigm for secure two-party computation
(yao1982protocols, ; micali1987play, ). Oblivious transfer is among the key
techniques. It guarantees the sender not to know what information the receiver
obtains from the transferred message. For multi-party condition, secret
sharing is one of the generic frameworks (karnin1983secret, ). Each data
instance is treated as a secret and split into several shares. Those shares
are then distributed to the multiple participated parties. The function value
computation is decomposed into basic operations and those basic operations are
computed following the protocol.

The use of SMPC in machine learning tasks has been studied in both model-
specific learning tasks e.g., linear regression (gascon2017privacy, ),
logistic regression (shi2016secure, ) and generic model learning tasks
(mohassel2017secureml, ). Secure inference is the most emerging topic that
tailors SMPC for machine learning use recently. Its application is machine
learning as a service, in which the server holds model and clients hold
private data. To alleviate the huge computation and communication cost of
SMPC, parameter quantization and function approximation are adopted together
with cryptographic protocols in (ball2019garbled, ; agrawal2019quotient, ).
Several tools have been open-sourced, such as MP2ML (boemer2020mp2ml, ),
CryptoSPN (treiber2020cryptospn, ), CrypTFlow (kumar2020cryptflow, ;
rathee2020cryptflow2, ), and CrypTen (knott2020crypten, ).

Federated Learning (FL) was initially proposed by Google as a secure scheme to
collaboratively train a machine learning model from the data of user
interaction with their devices (mcmahan2017communication, ). It quickly gains
extensive interest from academia and industry as a solution to collaborative
model training tasks using data from multiple parties. It aims to address data
privacy concerns that hinder machine learning algorithms from properly using
multiple data sources. Its application has been seen in numerous domains such
as healthcare (rieke2020future, ; sheller2020federated, ) and finance
(long2020federated, ).

Existing FL algorithms can be categorized into horizontal FL, vertical FL, and
federated transfer learning algorithms (yang2019federated, ). Horizontal FL
refers to the scenario that each party has different samples but the samples
share the same feature space. One research issue that characterizes horizontal
FL is that the datasets across the machines are not independently identically
distributed. Recent research studies the algorithm convergence property e.g.,
(li2019convergence, ) and novel algorithms that have more favorable
convergence property e.g., (reddi2020adaptive, ). Vertical FL refers to the
setting that all parties share the same sample ID space but own different
features. (hardy2017private, ) presents using Homomorphic Encryption for
vertical logistic regression model learning. In (gu2020federated, ), an
efficient kernel learning method is proposed. Federated transfer learning is
applicable to the condition that all parties have non-overlap in both sample
and feature space (liu2020secure, ). The connection between FL and other
research topics, such as multi-task learning, meta-learning, and fairness
learning, is discussed in (MAL-083, ). To expedite the FL-related research and
development, many open-source libraries are released for public use, such as
FATE333https://github.com/FederatedAI/FATE, FedML (he2020fedml, ) and
Fedlearn-Algo (liu2021fedlearn, ).

###  4.3. Development

The manufacture of reliable products requires considerable efforts in software
engineering, which is sometimes overlooked by AI developers. This lack of
diligence, such as insufficient testing and monitoring, may incur long-term
cost in the subsequent lifecycle of AI products (a.k.a. technical debt
(sculley2015hidden, )). The software engineering problem in the stage of
development and deployment has recently aroused wide concern as an essential
condition for reliable AI systems (amershi2019software, ; lavin2021technology,
). Moreover, various techniques researched in this stage are shown to further
contribute to the trustworthiness of an AI system (amershi2019software, ). In
this section, we survey the representative techniques.

####  4.3.1. Functional Testing

Inherited from the canonical software engineering workflow, testing
methodology has drawn arising attention in the development of an AI system. In
terms of AI trustworthiness, testing serves as an effective approach to
certify that the system is fulfilling specific requirements. Recent research
explores new approaches to adapt functional testing on AI systems and are
reviewed in literature such as (huang2020survey, ; zhang2020machine, ;
marijan2020software, ). We describe two aspects of adaption from the
literature which is useful to enhance the trustworthiness of an AI system.

Test criteria. Different from canonical software engineering where exact
equity is tested between the actual and expected outputs of a system, an AI
system is usually tested by its prediction accuracy on a specific testing
dataset. Beyond accuracy, various test criteria have been studied to further
reflect and test more complex properties of an AI system. The concept of test
coverage criterion in software testing has been transplanted to highly
entangled DNN models (pei2017deepxplore, ; ma2018deepgauge, ). The name of a
representative metric – neuron coverage (pei2017deepxplore, ) – figuratively
illustrates that it measures the coverage of activated neurons in a DNN in
analogy to the code branches in canonical software testing. It has been shown
that such coverage criteria are effective to certificate the robustness of a
DNN against adversarial attack (ma2018deepgauge, ).

Test case generation. To thoroughly test an AI system, human-annotated
datasets are insufficient, and large-scale automatically generated test cases
are widely adopted. Similar to the canonical software testing, the problem to
automatically generate expected ground truth, known as the oracle problem,
also exists in the AI software testing scenario. The hand-crafted test case
template is shown to be an intuitive but effective approach in applications
such as NLP (ribeiro2020beyond, ). Metamorphic testing is also a practical
approach whose basic idea is to convert existing input/output pairs to new
test cases. For example, (zhang2018deeproad, ) transfers daylight road scene
images to rainy style by GAN as new test cases and re-uses the original
invariant annotation to test an autonomous driving system. These testing cases
are shown to be useful in evaluating the generalization performance of an AI
model. Similar methodology is also adopted by adding adversarial patterns to
normal images in testing the adversarial robustness (ma2018deepgauge, ).
Simulated environments are also widely used to test applications such as
computer vision and reinforcement learning. We will further review this topic
in Section 4.3.3.

####  4.3.2. Performance Benchmarking

Unlike conventional software, the functionality of AI systems is often not
easily captured in unit test cases or functional tests alone. To ensure that
systems are trustworthy on different aspects of interest, benchmarking (a.k.a.
performance test in software engineering) is often applied to ensure system
performance and stability when these characteristics can be automatically
measured.

Robustness is one important aspect to trustworthiness, and is relatively
amenable to automatic evaluation. (croce2020robustbench, ) and
(hendrycks2019benchmarking, ) introduce a suite of black-box and white-box
attacks to automatically evaluate the robustness of AI systems, which can
potentially be performed as a sanity check before they are deployed to affect
millions of users.

Software fairness has also been a concern ever from the conventional software
testing (brun2018software, ; galhotra2017fairness, ). Criteria on AI systems
are also studied to check the fairness by investigating the correlation
between the sensitive attributes and the system outcomes (tramer2017fairtest,
). Well-curated datasets and metrics have been proposed in previous literature
to evaluate the performance on fairness metrics that are of interest to
different tasks (bellamy2018ai, ; friedler2019comparative, ;
sixta2020fairface, ).

More recently, there has also been growing interest in benchmarking
explainability when models output explanations in NLP applications. For
instance, (mathew2020hatexplain, ) asks crowd workers to annotate salient
pieces of text that lead to their belief that the text is hateful or
offensive, and examines how well model-predicted importance fits human
annotations; (deyoung2019eraser, ) instead introduces partial perturbations to
the text for human annotators, and observes if the system explanations match
perturbations that changed human decisions. In the meantime,
(mohseni2018human, ) reports that explainability benchmarking remains
relatively more difficult since visual stimuli is higher-dimensional and
continuous.

####  4.3.3. Development by Simulation

While benchmarks serve to evaluate AI systems on their predictive behavior
given static data, the behavior of many systems is deeply rooted in their
extended interaction with the world. For example, benchmarking autonomous
vehicle systems on static scenarios has very limited value in helping us
evaluate their performance on dynamic roadways. For these systems, simulation
often plays an important role in ensuring their trustworthiness before
deployment.

Robotics is one of the subfields in AI where simulation is most commonly used.
Control systems for robots can be compared and benchmarked in simulated
environments such as Gazebo (koenig2004design, ), MuJoCo (todorov2012mujoco,
), VerifAI (dreossi2019verifai, ), etc. Similarly, simulators for self-driving
cars have also been widely used, including TORCS (wymann2000torcs, ), CarSim
(benekohal1988carsim, ), PRESCAN (tideman2010scenario, ), etc. These software
platforms simulate the environment in which robots and vehicles will operate,
as well as the actuation of controls on simulated robots or cars. In NLP,
especially Conversational AI, simulators are also widely used to simulate user
behavior to test systems’ ability to fulfill user needs by engaging in a
dialogue (lee2019convlab, ). These simulators can help automatically assure AI
systems in an interactive environment, as well as diagnose issues before they
are deployed.

Despite the efficiency, flexibility, and replicability afforded by software
simulators, they often still fall short at perfectly simulating constraints
the AI system faces when deployed, as well as environmental properties or
variations. For AI systems that are deployed on embedded or otherwise boxed
hardware, it is important to understand system behavior when they are run on
the hardware used in real-world scenarios. Hardware-in-the-loop simulations
can help developers understand system performance when it is actually run on
the chips, sensors, and actuators in a simulated environment, which is
particularly helpful for latency- and power-critical systems like autonomous
driving systems (bokc2007validation, ; brogle2019hardware, ). Taking real-
world simulations one step further, one can also construct controlled real-
world environments for fully integrated AI systems to roam around in (e.g.,
test tracks for self-driving cars with road signs and dummy obstacles),
further providing more realistic measurements and assurance of performance
before releasing such systems to actual users.

####  4.3.4. Formal Verification

The aforementioned software testing has established a series of best practices
to eliminate system bugs. However, it is still unable to guarantee that the
systems work correctly and reliably, which is significantly important for
safety-critical applications. Formal verification aims at developing rigorous
mathematical specification and verification techniques for the assurances of
software correctness (clarke1996formal, ). Various formal methods, including
semantic static analysis (cousot1977abstract, ), model checking
(clarke2018model, ), and theorem proving(bertot2013interactive, ), have been
developed for traditional software verification.

Due to the different development processes, the formal methods developed for
traditional software may not be suitable for modern machine learning systems.
The lifecycle of an AI system includes problem specification, data
preparation, model training, model deployment, and feedback. Unlike
traditional software, the model training is non-deterministic, which brings
challenges to the AI system verification. To develop a verified AI system,
(seshia2016towards, ) itemizes four challenges (e.g., environment modeling,
formal specification, system modeling, and scalability) and reviews the
corresponding solutions.

The research of formal methods for machine learning mainly focuses on robust
training to deal with perturbations. Basically, for each input, the
adversarial training attempts to compute the lower bound of the worst-case
loss (moosavi2016deepfool, ; goodfellow2014explaining, ). Most adversarial
training approaches do not guarantee a full defense to all kinds of
adversarial attacks. To solve this problem, the certified robustness
approaches, such as CNN-Cert (boopathy2019cnn, ), CROWN(zhang2018efficient, ),
Fast-lin, or Fast-lip (weng2018evaluating, ), etc., aim to minimize an upper
bound of the worst-case loss. A recent survey work by (urban2021review, ) has
a thorough review on formal verification for the neural network. They mainly
categorize the formal methods into complete verification and incomplete
verification methods. The former ones can precisely verify if a specification
holds to the network, but generally they do not scale to large networks and
have constraints to network architectures and activation functions. In
contrast, the latter can scale to large networks with fewer constraints.

###  4.4. Deployment

After development, AI systems are deployed on realistic products, and interact
with environment and users. To guarantee systems to be trustworthy, there are
a number of approaches which should be considered at the deployment stage,
such as adding additional components to monitor anomalies, developing specific
human-AI interaction mechanisms for transparency and explain-ability, and more
as discussed below.

####  4.4.1. Anomaly Monitoring

Anomaly monitoring has become a well-established methodology in the realm of
software engineering. In terms of artificial intelligence, the range of
monitoring is further extended to cover the data outliers, data drifts, model
performance, etc. As a keying insurance of the successful operation of an AI
system, monitoring provides means to enhance the system trustworthiness in
multiple aspects. Some representative examples are discussed as follows.

Attack monitoring has been widely adopted in conventional SaaS, such as fraud
detection (abdallah2016fraud, ) in e-commerce systems. In terms of the recent
emerging adversarial attacks, detection and monitoring (metzen2017detecting, )
of such attack inputs is also recognized as an important means to ensure
system robustness. Data drift monitoring (rabanser2018failing, ) provides
important means to maintain the reproducibility and generalization of an AI
system under concept change (vzliobaite2016overview, ) caused by dynamic
environment such as market change (samuylova2020machine, ). Misuse monitoring
is recently also adopted in several cloud AI services (javadi2020monitoring, )
to avoid improper use such as unauthorized population surveillance or
individual tracking by face recognition, which helps ensure the proper
alignment of ethical values.

####  4.4.2. Human-AI Interaction