As an extension of human-computer interaction, human-AI interaction has
aroused wide attention in the AI industry (abdul2018trends, ;
amershi2019guidelines, ). Effective human-AI interaction affects the
trustworthiness of an AI system in multiple aspects, of which we briefly
illustrate two topics as follows:

User interface serves as the most intuitive factor affecting the user
experience. It is one of the major mediums for an AI system to disclose
internal information and decision-making procedure to users and thus is
recognized to have an important effect on the transparency and explainability
of the system (shneiderman2020bridging, ; weld2019challenge, ). Various
interface approaches have been studied to enhance explainability, including
visualization of ML models (chatzimparmpas2020state, ) and interactive
parameter tuning (weld2019challenge, ). In addition to transparency and
explainability, accessibility of the interface is also significantly affecting
users’ experience of trustworthiness. AI-based interaction techniques have
enabled various new forms of human-machine interfaces such as chatbot, audio
speech recognition, and gesture recognition, which might result in
accessibility problems for disabled people. Mitigation of such unfairness has
aroused concerns in recent research (kafle2020artificial, ;
torres2018accessibility, ).

Human intervention, by such as monitoring failure or participating in decision
(schmidt2017intervention, ), has been applied in various AI systems to
compensate the limited performance. For example, ADAS systems can be
considered as a typical example of human intervention system where AI takes
low-level driving work and human makes the high-level decision. In addition to
compensating decision-making, human intervention provides informative
supervision to train or fine-tune AI systems in many scenarios such as the
shadow mode (templaton2019tesla, ) of autonomous driving vehicles. To minimize
and make the best use of human efforts in interaction, efficient design of
human-machine cooperation patterns has been a recently emerging topic in the
interdisciplinary study of HCI and AI, and is referred as human-in-the-loop or
interactive machine learning (holzinger2016interactive, ) in literature.

####  4.4.3. Fail-Safe Mechanisms

Considering the imperfection of current AI systems, it becomes an important
problem to avoid harm when the system fails in exceptional cases. Learned from
the conventional real-time automation systems, the AI community is realizing
that the fail-safe mechanism or fallback plan should be taken as an essential
part in the design of an AI system if its failure can cause harm or loss. This
mechanism is also becoming an important requirement by recent AI guidelines
such as (aihleg2018ethics, ). We have seen this idea of design in multiple
robotic areas in the past years. In the area of UAV, the fail-safe algorithm
has been studied for a long time to avoid the frequent collision of
quadrocopters (fu2014monocular, ) or ensure safe landing upon system failure
(muller2016increased, ). In autonomous driving where safety is much more
critical, a fail-safe mechanism like standing still has become an
indispensable component in ADAS products (horwick2010strategy, ) for years and
is also continuously researched in a higher level of automation
(magdici2016fail, ).

####  4.4.4. Hardware Security

AI systems are currently widely deployed on various hardware platforms to cope
with demands from diverse scenarios, ranging from servers in computing centers
to cell phones and embedded systems. Attacks over OS and hardware cause new
risks such as data tampering or stealing, threatening the robustness,
security, and privacy of AI systems. Various approaches have been studied to
address this new threat (xu2021security, ). In the aspect of hardware
security, the concept of trusted execution environment (TEE) is a recent
representative technique which has been adopted by many hardware manufacturers
(sabt2015trusted, ). The general mechanism of TEE is to provide a secure area
for data and code. This area is not interfered by the standard OS that
potentially can attack the program to be protected. ARM processors support TEE
implementation by the TrustZone design (pinto2019demystifying, ). They run a
secure operating system and a normal one simultaneously on a single core. The
secure part provides a safe environment for sensitive information. The Intel
Software Guard Extensions (SGX) implements TEE by hardware-based memory
encryption (mckeen2016intel, ). The enclave mechanism allows the protected
programs to allocate the memory containing private information. Such security
measures have been applied to protect sensitive information like biometric ID
and financial account passwords, and are shown to be applicable to more AI use
cases.

###  4.5. Management

AI practitioners such as researchers and developers have studied various
techniques to improve AI trustworthiness in the aforementioned data,
algorithm, development, and deployment stages. Beyond these concrete
approaches, appropriate management and governance provide a holistic guarantee
that the consideration of trustworthy AI is consistently aligned throughout
each stage and technique procedure of the AI system lifecycle. In this
section, we introduce several executable approaches which facilitate the AI
community to establish management and governance on trustworthiness.

####  4.5.1. Documentation

Conventional software engineering has accumulated a wealth of experience in
leveraging documentation to assist the development. Representative
documentation types include requirement documents, product design documents,
architecture documents, code documents, test documents, etc
(aimar1998introduction, ). Beyond the conventional software engineering,
multiple new types of documents have been proposed to adapt to the ML training
and testing mechanisms. The scope of these documents may include the purposes
and characteristics of the model (mitchell2019model, ), datasets
(gebru2018datasheets, ; bender2018data, ; holland2018dataset, ), and services
(arnold2019factsheets, ). As mentioned in Section 2.3.2 and 2.8, documentation
is recognized as an effective and important approach to enhance the system
transparency and accountability by tracking, guiding, and auditing its whole
lifecycle (raji2020closing, ), which serves as a cornerstone of building a
trustworthy AI system.

####  4.5.2. Auditing

With lessons learned from safety-critical industries such as finance and
aerospace, auditing is recently recognized and deployed as an effective
mechanism to examine whether an AI system complies with specific principles
(buolamwini2018gender, ; wilson2021building, ). In terms of the position of
auditors, the auditing process can be categorized as internal and external.
Internal auditing enables self-assessment and iterative improvement for
manufacturers to follow the principles of trustworthiness. It could cover the
whole lifecycle of the system without risks of trade secret leakage
(raji2020closing, ). On the other hand, external auditing by independent
parties acts more effectively in gaining concern or trust from the public
(buolamwini2018gender, ).

Auditing might involve the whole or selective parts of the lifecycle of an AI
system. (raji2020closing, ) designs a comprehensive framework to internally
audit the whole development lifecycle. Means of the auditing might include
interviews, documented artifacts, checklists, code review, testing, impact
assessment, etc. For example, documentation like product requirement documents
(PRD), model cards (mitchell2019model, ), datasheets (gebru2018datasheets, )
serve as important references to understand the principle alignment during the
development. Checklists are widely adopted as a straightforward qualitative
approach to evaluate fairness (madaio2020co, ), transparency
(schelenz2020applying, ), reproducibility (pineau2020improving, ), etc.
Quantitative testing also serves as a powerful approach and has been
successfully executed to audit fairness in for example the Gender Shade study
(buolamwini2018gender, ). Inspired by the EU Data Protection Impact
Assessments (DPIA), the concept of Algorithmic Impact Assessment (AIA) is
proposed to evaluate the trustworthiness claims of systems and discover
negative effects (reisman2018algorithmic, ). Besides the above
representatives, more designs of algorithmic auditing approaches can be found
in (sandvig2014auditing, ; wilson2021building, ).

####  4.5.3. Cooperation and Information Sharing

As mentioned in Figure 2, the establishment of trustworthy AI requires
cooperation between different stakeholders. Considering from the perspective
of industry, cooperation with academia enables the fast application of new
technology to enhance the product performance and reduce risk. Cooperation
with regulators certifies the products to appropriately follow principles of
trustworthiness. Moreover, cooperation between industrial enterprises is shown
to play an important role in addressing consensus problems such as data
exchange, standardization, and ecosystem building (askell2019role, ). Recent
practices of AI stakeholders have shown the efficacy of cooperation in various
aspects. We summarize these practices in the following representative aspects.
It is worthwhile to consider appropriate cooperation including but not limited
to the following forms to enhance the trustworthiness throughout the lifecycle
of an AI system.

Collaborative research and development. Collaboration has been a successful
driving force to the development of AI technology. To promote the research on
AI trustworthiness, stakeholders are actively setting up various forms of
collaboration such as research workshops on trustworthy AI and cooperative
projects like DARPA XAI (gunning2019darpa, ).

Trustworthy data exchange. The increasing business value of data raises the
demand to exchange data across companies in various scenarios (e.g., the
medical AI system in Section 2.6). Beyond privacy computing techniques, the
cooperation between data owners, technology providers, and regulators is also
making progress in establishing the data exchange ecosystem and addressing
problems such as data pricing and data authorization.

Cooperative development of regulation. Active participation in the development
of standards and regulations serves as an important means for academia,
industry, and regulators to align their requirements and situations.

Incidents sharing. AI community has recently recognized incident sharing as an
effective approach to highlight and prevent potential risks of AI systems
(brundage2020toward, ). AI Incident Database (DavidDao2020, ) provides an
inspiring example for stakeholders to share negative AI incidents so that
industry can avoid similar problems.

###  4.6. TrustAIOps: A Continuous Workflow towards Trustworthiness

Figure 4. The establishment of trustworthy AI requires AI practitioners to be
organized in a continuous workflow. We extend the MLOps workflow to TrustAIOps
by emphasizing the extra properties of continuous feedback loop, management of
artifacts, aligned trustworthy principles and close multidisciplinary
collaboration. The first two properties (green arrows) emphasize the
executable organization strategy and the latter two focus (blue arrows) on the
ethos which practitioners should hold. See Section 4.6.

The problem of trustworthy AI arises from the fast development of AI
technology and its emerging applications. It should be realized that AI
trustworthiness is not a well-studied static bar to reach by some specific
solutions. The establishment of trustworthiness is a dynamic procedure. We
have witnessed the evolution of different dimensions of trustworthiness over
the past decade (jobin2019global, ). For example, the research of adversarial
attack increases the concern on adversarial robustness. The applications on
safety-critical scenarios arouse the requirement of accountability of an AI
system. The development of AI research, the evolution of AI product form, and
the changing cognition of society will keep reformulating the requirements and
solutions of trustworthiness. Therefore, we argue that beyond the requirements
of an AI product, the AI industry should consider trustworthiness as an ethos
of its operation routine and be prepared to continuously enhance the
trustworthiness of its products.

The continuous enhancement of AI trustworthiness positions requirements on a
new workflow for the AI industry. Recent studies on industrial AI workflow
extend the mechanism of DevOps (bass2015devops, ) to MLOps (makinen2021needs,
), to enable effective improvement of ML products. The concept of DevOps has
been successfully adopted in modern software development to continuously
deploy software features and improve quality. MLOps (makinen2021needs, ) and
its variants such as ModelOps (hummer2019modelops, ) and SafetyOps
(siddique2020safetyops, ) extend DevOps to cover the ML lifecycle of data
preparation, training, validation, and deployment, in its workflow. MLOps
workflow provides a start point to build the workflow for trustworthy AI. By
integrating the ML lifecycle, MLOps connects research experimentation and
product development, which enables the rapid leverage of theoretical
development of trustworthy AI. A wealth of MLOps toolchains have been released
recently to track AI artifacts such as data, model, and metadata, which
increases the accountability and reproducibility of products
(hummer2019modelops, ). We have seen recent research to extend MLOps to
further integrate trustworthiness in the AI workflow. For example,
(siddique2020safetyops, ) extends MLOps with safety engineering as SafetyOps
in autonomous driving.

As we have illustrated in this section, the establishment of trustworthiness
requires the continuous and systematic upgrade of the AI lifecycle. By
extending MLOps, we summarize this upgrade of practices as a new workflow,
TrustAIOps, which focuses on imposing requirements of trustworthiness in the
management of the AI lifecycle. This new workflow contains the following
properties.

  * •

Close collaboration between interdisciplinary roles. Building trustworthy AI
requires organizing different roles such as ML researchers, software
engineers, safety engineers, and legal experts. Close collaboration mitigates
the knowledge gap between different expertise.

  * •

Aligned trustworthy principles. The risk of untrustworthiness exists in every
stage in the lifecycle of an AI system. Mitigating such risks requires all
stakeholders in the AI industry to be aware of and aligned with unified
trustworthy principles.

  * •

Extensive management of artifacts. An industrial AI system is built upon a
variety of artifacts such as data, code, models, configuration, product
design, and operation manuals. Elaborate management of these artifacts helps
assess risk and increases reproducibility and auditability.

  * •

Continuous feedback loops. Classical continuous integration and continuous
development (CI/CD) workflows provide effective mechanisms to improve software
in feedback loops. In a trustworthy AI system, these continuous feedback loops
should connect and iteratively improve the five stages of its lifecycle, i.e.,
data, algorithm, development, deployment, and management.

The evolution of AI industrial workflow is a natural reflection of the dynamic
procedure to establish AI trustworthiness. By systematically organizing AI
lifecycle stages and interdisciplinary practitioners, the AI industry is able
to understand the trustworthy requirements from various aspects including
technology, legality, and society, and then continuously deliver improvement.

##  5\. Conclusion, Challenges and Opportunities

In this survey, we outlined key aspects of trustworthiness that we think are
essential to AI systems that make a difference on people’s lives. We
introduced how AI systems can be evaluated and assessed on each of these
aspects, and reviewed current efforts in this direction in the industry. We
further proposed a systematic approach to take these trustworthiness aspects
into the consideration of the entire lifecycle of real-world AI systems, which
offers recommendations for every step of the development and use of these
systems from data to governance.

We recognize that fully adopting this systematic approach towards building
trustworthy AI systems requires practitioners to embrace the new concepts
underlying the key aspects that we have identified. More importantly, it also
requires a shift of focus from _performance-driven_ AI to _trust-driven_ AI.
In the short run, this shift will inevitably involve side-effects such as
learning time, slowed development, and/or increased cost to build AI systems.
However, we encourage practitioners to focus on the long-term benefit of
gaining the trust of all stakeholders involved in the development and use of
these systems.

In this section, we conclude by discussing some of the open challenges and
potential opportunities in the future development of trustworthy AI.

###  5.1. AI Trustworthiness as Long-Term Research

Our understanding of AI trustworthiness is far from complete or universal,
which inevitably evolves as we develop new AI technologies and understand
their societal impact more deeply. This procedure requires long-term research
in multiple key areas of AI. In this section, we discuss several open
questions that we think are crucial to address for the future development of
AI trustworthiness.

####  5.1.1. Immaturity of Approaches to Trustworthiness.

As mentioned in Section 2, several aspects of AI trustworthiness, such as
robustness, explainability, and generalization, address the limitation of
current AI technologies. These problems have attracted wide interest in AI
research, but satisfactory solutions are still far from reach.

Take explainability as an example. Despite being an active field of AI
research, explainability remains poorly understood. Both current explainable
models and post-hoc model explanation techniques share a few common issues,
e.g., 1) The explanation is fragile to perturbations
(ghorbani2019interpretation, ); 2) The explanation is not always consistent
with human interpretation (bodria2021benchmarking, ); 3) It is difficult to
judge if the explanation is correct or faithful (molnar2020interpretable, ).
These problems pose important questions in the study of explainability and
derive valuable research directions in theoretical research on AI.

Another example is robustness. The arms race between adversarial attack and
defense reflects the immaturity of our understanding of AI robustness. Similar
to other areas of security, attacks evolve along with the development of
defenses. Conventional adversarial training (goodfellow2014explaining, ) has
been shown to be easily fooled by later attacks (tramer2017ensemble, ). The
corresponding defense (tramer2017ensemble, ) is later shown to be vulnerable
against new attacks (dong2019evading, ). This not only requires practitioners
to be agile in adopting defense techniques to mitigate the risk of new attacks
in a long-term continuous development process, but also poses long-term
challenges in theoretical research (raghunathan2018certified, ).

####  5.1.2. Frictional Impact of Trustworthy Aspects.

Trustworthy AI reflects a combination of requirements of ethical values on AI
systems. As we have shown in Section 2, there are a wealth of connections and
supports between different trustworthy aspects. On the other hand, a variety
of research has discovered that there are frictions or trade-offs between
these aspects in some cases. Next, we review some recent examples on these
topics.

Increased transparency improves trust in AI systems through information
disclosure. However, disclosing inappropriate information might increase
potential risks. For example, excessive transparency on datasets and
algorithms might leak private data and commercial intellectual property.
Disclosure of detailed algorithmic mechanisms could also lead to the risk of
targeted hacking (akhtar2018threat, ). On the other hand, an inappropriate
explanation might also cause users to overly rely on the system and follow
wrong decisions of AI (stumpf2016explanations, ). Therefore, the extent of
transparency of an AI system should be specified carefully and differently for
the roles of public users, operators and auditors.

From an algorithmic perspective, the effect of different trustworthy
objectives on model performance remain insufficiently understood. Adversarial
robustness increases model generalizability and reduces overfit, but tends to
negatively impact the overall accuracy (tsipras2018robustness, ;
zhang2019theoretically, ). Similar loss of accuracy happens on explainable
models (bodria2021benchmarking, ). Besides this trust-accuracy trade-off,
algorithmic friction also exists between trustworthy dimensions. For example,
adversarial robustness and fairness can negatively affect each other during
training (xu2020robust, ; roh2020fr, ). Furthermore, studies regarding
fairness and explainability have also shown several explanation approaches to
be unfair (dodge2019explaining, ).

These frictional effects suggest that AI trustworthiness cannot be achieved
through hillclimbing on a set of disjoint criteria. Compatibility should be
carefully considered when integrating multiple requirements on a single
system. Recent emerging studies (zhang2019theoretically, ; xu2020robust, ) on
this area provide a good reference to start.

####  5.1.3. Limitations in Current Trustworthiness Evaluation.

Repeatable and quantitative measurements are the cornerstone of scientific and
engineering progress. However, despite increasing research interest and
efforts, many aspects of AI trustworthiness still remain elusive to
quantification. Among the various aspects we have discussed in this paper, the
explainability, transparency, value alignment, and accountability of AI
systems are still seldom evaluated quantitatively, which makes it difficult to
accurately compare different systems. Developing good quantitative evaluation
methods for these desiderata, we believe, will be an important first step
towards making research on these aspects of AI trustworthiness a scientific
endeavor, rather than a purely philosophical one.

####  5.1.4. An Experimental Case Study towards Trustworthy Face Verification.

As illustrated in this section, the harmonic combination and achievement of
multiple objectives of AI trustworthiness, such as robustness, fairness, and
explainability, still remain an unresolved research problem. To better
illustrates the complexity of this combination, we conduct a demonstrative
experiment by considering a practical scenario of face verification (e.g., for
the user to unlock a smartphone).

Based on a conventional CNN face verification model, We enhance the system
robustness by utilizing adversarial learning algorithms. We want the system to
be fair to different age groups and deploy age invariance and domain
generalization algorithms to improve performance on the disadvantageous group.
We further enhance system explainability by switching out a conventional
system with one that is more explainable.

The experiment studies the interaction between trustworthy approaches on by
combining several existing algorithms on single dimensions. Our experiment
shows that negative interaction exists when integrating some trustworthiness
approaches, and the most common negative impact is the drop of accuracy
performance. However, some previous reported trade-offs, such as that between
robust and fairness, are not significant in our experiment, which we attribute
to the distributions and sizes of datasets. On a whole, we use this
experiments to show that the harmonic combination and achievement of multiple
objectives of AI trustworthiness can be complicated and involved with various
factors.

The experiment and its results are described in the appendix of this paper.
The experiment code will be released soon.

###  5.2. End-User Awareness of the Importance of AI Trustworthiness

Other than developers and providers of AI systems, another important group of
stakeholders that should not be neglected is the end-users. Making sure that
users understand the need for trustworthy AI systems is an important societal
goal. Besides educating the general public about the basic concepts of AI
trustworthiness, developers should also consider how various aspects of
trustworthiness can be presented to users to deliver hands-on experiences of
the benefit of trustworthy AI systems. One positive step towards this
direction involves demonstrating system limitations (e.g., Google Translate
displays translations in multiple gendered pronouns when it cannot determine
the correct ones based on the input text) or explainable factors that are used
in the decision process (e.g., many advertisement recommendation systems now
share user traits used in curating ads, like “female aged 20-29” and
“interested in technology”). Taking it one step further, we believe that it
would empower the user to allow them to directly control these factors (e.g.,
user traits) either to their self-identified persona, or to experiment with
the system counter-factually to observe its behavior, and judge for themselves
whether the system is fair, robust, and trustworthy.

Finally, we would also like to recognize that not all aspects of
trustworthiness can be equally easily conveyed to end-users. For instance, the
impact of privacy preservation or transparency cannot be easily demonstrated
to the end-user when AI systems are deployed. We believe that media coverage
and government regulations regarding these aspects would be very helpful in
raising awareness for the public.

###  5.3. Interdisciplinary and International Cooperation

In-depth understanding of AI trustworthiness involves not only the development
of better and newer AI technologies. More importantly, it requires us to
better understand our human values, morals, ethical considerations, and
limitations. We believe this calls for the collaboration across various
disciplines that reach far beyond computer science. First, AI practitioners
should work closely with domain experts whenever AI technologies are deployed
into the real world with impact on real people, e.g., medicine, finance,
transportation, agriculture, etc. Second, AI practitioners should actively
seek advice from social scientists to better understand the (often unintended)
societal impact and work together to remedy them, e.g., the impact of AI-
automated decisions, job displacement in AI-impacted sectors, and AI systems’
effect in social networks. Third, AI practitioners should also carefully
consider how the technology is presented to the public as well as
interdisciplinary collaborators, and make sure to communicate the known
limitations of AI systems honestly and clearly.

In the meantime, the development of trustworthy AI is by no means a unique
problem of any single country, nor does the potential positive or negative
effect of AI systems respect geopolitical borders. Despite the common
portrayal of AI as a race between countries (see §3.2 of
(hagendorff2020ethics, ) for a detailed discussion), technological advances in
the climate of increased international collaboration are far from a zero-sum
game. It not only allows us to build better technological solutions by
combining diverse ideas from different backgrounds, but also helps us better
serve the world’s population by recognizing our shared humanity as well as our
unique differences. We believe that tight-knit interdisciplinary and
international cooperation will serve as the bedrock for rapid and steady
developments in trustworthy AI technology, which will in turn benefit humanity
in unison.

##  6\. Acknowledgments

The authors would like to thank Yanqing Chen, Jing Huang, Shuguang Zhang, and
Liping Zhang for their valuable suggestions. We also thank Yu He, Wenhan Xu,
Xinyuan Shan, Chenliang Wang, Peng Liu, Jingling Fu, Baicun Zhou, Hongbao Tian
and Qili Wang for their assistance with the experiment in the appendix.

## References

  * (1) https://github.com/EthicalML/xai.
  * (2) https://github.com/Trusted-AI/adversarial-robustness-toolbox.
  * (3) https://oecd.ai/wonk/tools-for-trustworthy-ai.
  * (4) https://blog.tensorflow.org/2020/06/responsible-ai-with-tensorflow.html.
  * (5) Ieee standard glossary of software engineering terminology.  ANSI/ IEEE Std 729-1983, pages 1–40, 1983.
  * (6) Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016.
  * (7) Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
  * (8) Aisha Abdallah, Mohd Aizaini Maarof, and Anazida Zainal.  Fraud detection system: A survey.  Journal of Network and Computer Applications, 68:90–113, 2016.
  * (9) Ashraf Abdul, Jo Vermeulen, Danding Wang, Brian Y Lim, and Mohan Kankanhalli.  Trends and trajectories for explainable, accountable and intelligible systems: An hci research agenda.  In Proceedings of the 2018 CHI conference on human factors in computing systems, pages 1–18, 2018.
  * (10) Amina Adadi and Mohammed Berrada.  Peeking inside the black-box: a survey on explainable artificial intelligence (xai).  IEEE access, 6:52138–52160, 2018.
  * (11) Philip Adler, Casey Falk, Sorelle A Friedler, Tionney Nix, Gabriel Rybeck, Carlos Scheidegger, Brandon Smith, and Suresh Venkatasubramanian.  Auditing black-box models for indirect influence.  Knowledge and Information Systems, 54(1):95–122, 2018.
  * (12) Divyansh Aggarwal, Jiayu Zhou, and Anil K Jain.  Fedface: Collaborative learning of face recognition model.  arXiv:2104.03008, 2021.
  * (13) Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos.  Learning optimal and fair decision trees for non-discriminative decision-making.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1418–1426, 2019.
  * (14) Nitin Agrawal, Ali Shahin Shamsabadi, Matt J Kusner, and Adrià Gascón.  Quotient: two-party secure neural network training and prediction.  In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1231–1247, 2019.
  * (15) AI HLEG.  Ethics guidelines for trustworthy AI, 2018.  Accessed: 2021-02-20.
  * (16) AI HLEG.  Assessment list for trustworthy artificial intelligence (altai) for self-assessment, 2020.
  * (17) Alberto Aimar.  Introduction to software documentation.  1998\.
  * (18) Naveed Akhtar and Ajmal Mian.  Threat of adversarial attacks on deep learning in computer vision: A survey.  Ieee Access, 6:14410–14430, 2018.
  * (19) Dalal Al-Azizy, David Millard, Iraklis Symeonidis, Kieron O’Hara, and Nigel Shadbolt.  A literature survey and classifications on data deanonymisation.  In International Conference on Risks and Security of Internet and Systems, pages 36–51. Springer, 2015.
  * (20) Mohammed AlShamsi, Said A Salloum, Muhammad Alshurideh, and Sherief Abdallah.  Artificial intelligence and blockchain for transparency in governance.  In Artificial Intelligence for Sustainable Development: Theory, Practice and Future Applications, pages 219–230. Springer, 2021.
  * (21) David Alvarez-Melis and Tommi S Jaakkola.  Towards robust interpretability with self-explaining neural networks.  In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7786–7795, 2018.
  * (22) Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi, and Thomas Zimmermann.  Software engineering for machine learning: A case study.  In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 291–300. IEEE, 2019.
  * (23) Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al.  Guidelines for human-ai interaction.  In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1–13, 2019.
  * (24) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané.  Concrete problems in ai safety.  arXiv:1606.06565, 2016.
  * (25) Plamen Angelov and Eduardo Soares.  Towards explainable deep neural networks (xdnn).  Neural Networks, 130:185–194, 2020.
  * (26) Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al.  Factsheets: Increasing trust in ai services through supplier’s declarations of conformity.  IBM Journal of Research and Development, 63(4/5):6–1, 2019.
  * (27) Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al.  A closer look at memorization in deep networks.  In International Conference on Machine Learning, pages 233–242. PMLR, 2017.
  * (28) Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information Fusion, 58:82–115, 2020.
  * (29) Vijay Arya, Rachel KE Bellamy, Pin-Yu Chen, Amit Dhurandhar, Michael Hind, Samuel C Hoffman, Stephanie Houde, Q Vera Liao, Ronny Luss, Aleksandra Mojsilović, et al.  One explanation does not fit all: A toolkit and taxonomy of ai explainability techniques.  arXiv:1909.03012, 2019.
  * (30) Rob Ashmore, Radu Calinescu, and Colin Paterson.  Assuring the machine learning lifecycle: Desiderata, methods, and challenges.  arXiv:1905.04223, 2019.
  * (31) Amanda Askell, Miles Brundage, and Gillian Hadfield.  The role of cooperation in responsible ai development.  arXiv:1907.04534, 2019.
  * (32) M Gethsiyal Augasta and Thangairulappan Kathirvalavakumar.  Reverse engineering the neural networks for rule extraction in classification problems.  Neural processing letters, 35(2):131–150, 2012.
  * (33) Yonatan Aumann and Yehuda Lindell.  Security against covert adversaries: Efficient protocols for realistic adversaries.  In Theory of Cryptography Conference, pages 137–156. Springer, 2007\.
  * (34) Serkan Ayvaz and Salih Cemil Cetin.  Witness of things: Blockchain-based distributed decision record-keeping system for autonomous vehicles.  International Journal of Intelligent Unmanned Systems, 2019.
  * (35) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv:1409.0473, 2014.
  * (36) Fan Bai, Jiaxiang Wu, Pengcheng Shen, Shaoxin Li, and Shuigeng Zhou.  Federated face recognition.  arXiv:2105.02501, 2021.
  * (37) Josef Baker-Brunnbauer.  Trustworthy ai implementation (taii) framework for ai systems.  Available at SSRN 3796799, 2021.
  * (38) Marshall Ball, Brent Carmer, Tal Malkin, Mike Rosulek, and Nichole Schimanski.  Garbled neural networks are practical.  IACR Cryptol. ePrint Arch., 2019:338, 2019.
  * (39) Sarah Bankins and Paul Formosa.  Ethical ai at work: The social contract for artificial intelligence and its implications for the workplace psychological contract.  Redefining the Psychological Contract in the Digital Era: Issues for Research and Practice, pages 55–72, 2021.
  * (40) Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, and Jaehoon Amir Safavi.  Mitigating poisoning attacks on machine learning models: A data provenance based approach.  In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pages 103–110, 2017.
  * (41) Peter L Bartlett and Shahar Mendelson.  Rademacher and gaussian complexities: Risk bounds and structural results.  Journal of Machine Learning Research, 3(Nov):463–482, 2002.
  * (42) Len Bass, Ingo Weber, and Liming Zhu.  DevOps: A software architect’s perspective.  Addison-Wesley Professional, 2015.
  * (43) Ghazaleh Beigi and Huan Liu.  A survey on privacy in social media: Identification, mitigation, and applications.  ACM Transactions on Data Science, 1(1):1–38, 2020.
  * (44) Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine learning practice and the bias-variance trade-off.  arXiv:1812.11118, 2018.
  * (45) Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal.  Reconciling modern machine-learning practice and the classical bias–variance trade-off.  Proceedings of the National Academy of Sciences, 116(32):15849–15854, 2019.
  * (46) Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al.  AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias.  arXiv:1810.01943, 2018.
  * (47) Emily M Bender and Batya Friedman.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  Transactions of the Association for Computational Linguistics, 6:587–604, 2018.
  * (48) Rahim F Benekohal and Joseph Treiterer.  Carsim: Car-following model for simulation of traffic in normal and stop-and-go conditions.  Transportation research record, 1194:99–111, 1988.
  * (49) Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth.  Fairness in criminal justice risk assessments: The state of the art. arxiv, 2017.
  * (50) Yves Bertot and Pierre Castéran.  Interactive theorem proving and program development: Coq’Art: the calculus of inductive constructions.  Springer Science & Business Media, 2013.
  * (51) Marianne Bertrand and Sendhil Mullainathan.  Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination.  American economic review, 94(4):991–1013, 2004.
  * (52) Siddhant Bhambri, Sumanyu Muku, Avinash Tulasi, and Arun Balaji Buduru.  A survey of black-box adversarial attacks on computer vision models.  arXiv:1912.01667, 2019.
  * (53) Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker.  Fairlearn: A toolkit for assessing and improving fairness in ai.  Microsoft, Tech. Rep. MSR-TR-2020-32, 2020.
  * (54) Alberto Blanco-Justicia, Josep Domingo-Ferrer, Sergio Martínez, and David Sánchez.  Machine learning explainability via microaggregation and shallow decision trees.  Knowledge-Based Systems, 194:105532, 2020.
  * (55) Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and Salvatore Rinzivillo.  Benchmarking and survey of explanation methods for black box models.  arXiv:2102.13076, 2021.
  * (56) Fabian Boemer, Rosario Cammarota, Daniel Demmler, Thomas Schneider, and Hossein Yalame.  Mp2ml: a mixed-protocol machine learning framework for private inference.  In Proceedings of the 15th International Conference on Availability, Reliability and Security, pages 1–10, 2020.
  * (57) Miranda Bogen and Aaron Rieke.  Help wanted.  An Examination of Hiring Algorithms, Equity, and Bias, 2018.
  * (58) Thomas Bokc, Markus Maurer, and Georg Farber.  Validation of the vehicle in the loop (vil); a milestone for the simulation of driver assistance systems.  In 2007 IEEE Intelligent vehicles symposium, pages 612–617. IEEE, 2007.
  * (59) Akhilan Boopathy, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel.  Cnn-cert: An efficient framework for certifying robustness of convolutional neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3240–3247, 2019.
  * (60) Olcay Boz.  Extracting decision trees from trained neural networks.  In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 456–461, 2002.
  * (61) Timo Breuer, Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, Philipp Schaer, and Ian Soboroff.  How to measure the reproducibility of system-oriented ir experiments.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 349–358, 2020.
  * (62) Craig Brogle, Chao Zhang, Kai Li Lim, and Thomas Bräunl.  Hardware-in-the-loop autonomous driving simulation without real-time constraints.  IEEE Transactions on Intelligent Vehicles, 4(3):375–384, 2019.
  * (63) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  arXiv:2005.14165, 2020.
  * (64) Yuriy Brun and Alexandra Meliou.  Software fairness.  In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 754–759, 2018.
  * (65) Miles Brundage, Shahar Avin, Jasmine Wang, Haydn Belfield, Gretchen Krueger, Gillian Hadfield, Heidy Khlaaf, Jingying Yang, Helen Toner, Ruth Fong, et al.  Toward trustworthy ai development: mechanisms for supporting verifiable claims.  arXiv:2004.07213, 2020.
  * (66) Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In Conference on fairness, accountability and transparency, pages 77–91. PMLR, 2018.
  * (67) Edited by: Peter Kairouz and H. Brendan McMahan.  Advances and open problems in federated learning.  Foundations and Trends in Machine Learning, 14(1):–, 2021.
  * (68) Toon Calders, Asim Karim, Faisal Kamiran, Wasif Ali, and Xiangliang Zhang.  Controlling attribute effect in linear regression.  In 2013 IEEE 13th international conference on data mining, pages 71–80. IEEE, 2013.
  * (69) Toon Calders and Sicco Verwer.  Three naive bayes approaches for discrimination-free classification.  Data Mining and Knowledge Discovery, 21(2):277–292, 2010.
  * (70) Flavio P Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney.  Optimized pre-processing for discrimination prevention.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 3995–4004, 2017.
  * (71) Rosario Cammarota, Matthias Schunter, Anand Rajan, Fabian Boemer, Ágnes Kiss, Amos Treiber, Christian Weinert, Thomas Schneider, Emmanuel Stapf, Ahmad-Reza Sadeghi, et al.  Trustworthy ai inference systems: An industry research view.  arXiv:2008.04449, 2020.
  * (72) Yulong Cao, Chaowei Xiao, Dawei Yang, Jing Fang, Ruigang Yang, Mingyan Liu, and Bo Li.  Adversarial objects against lidar-based autonomous driving systems.  arXiv:1907.05418, 2019.
  * (73) Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In 2017 ieee symposium on security and privacy (sp), pages 39–57. IEEE, 2017.
  * (74) Simon Caton and Christian Haas.  Fairness in machine learning: A survey.  arXiv:2010.04053, 2020.
  * (75) L Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi.  Classification with fairness constraints: A meta-algorithm with provable guarantees.  In Proceedings of the conference on fairness, accountability, and transparency, pages 319–328, 2019.
  * (76) L Elisa Celis and Vijay Keswani.  Improved adversarial learning for fair classification.  arXiv:1901.10443, 2019.
  * (77) Yixuan Chai, Guohua Liu, Ziwei Jin, and Donghong Sun.  How to keep an online learning chatbot from being corrupted.  In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2020.
  * (78) Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay.  Adversarial attacks and defences: A survey.  arXiv:1810.00069, 2018.
  * (79) Saptarshi Chakraborty and Dhrubajyoti Das.  An overview of face liveness detection.  arXiv:1405.2227, 2014.
  * (80) Raghavendra Chalapathy and Sanjay Chawla.  Deep learning for anomaly detection: A survey.  arXiv:1901.03407, 2019.
  * (81) Patrick PK Chan, Zhi-Min He, Hongjiang Li, and Chien-Chang Hsu.  Data sanitization against adversarial label contamination based on data complexity.  International Journal of Machine Learning and Cybernetics, 9(6):1039–1052, 2018.
  * (82) Angelos Chatzimparmpas, Rafael M Martins, Ilir Jusufi, Kostiantyn Kucher, Fabrice Rossi, and Andreas Kerren.  The state of the art in enhancing trust in machine learning models with the use of visualizations.  In Computer Graphics Forum, volume 39, pages 713–756. Wiley Online Library, 2020.
  * (83) Chaofan Chen, Oscar Li, Chaofan Tao, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin.  This looks like that: deep learning for interpretable image recognition.  arXiv:1806.10574, 2018.
  * (84) Irene Chen, Fredrik D Johansson, and David Sontag.  Why is my classifier discriminatory?  arXiv:1805.12002, 2018.
  * (85) Rui Chen, Benjamin CM Fung, S Yu Philip, and Bipin C Desai.  Correlated network data publication via differential privacy.  The VLDB Journal, 23(4):653–676, 2014.
  * (86) Rui Chen, Noman Mohammed, Benjamin CM Fung, Bipin C Desai, and Li Xiong.  Publishing set-valued data via differential privacy.  Proceedings of the VLDB Endowment, 4(11):1087–1098, 2011.
  * (87) Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017.
  * (88) Yu Cheng, Mo Yu, Xiaoxiao Guo, and Bowen Zhou.  Few-shot learning with meta metric learners.  arXiv:1901.09890, 2019.
  * (89) Merav Chkroun and Amos Azaria.  “ did i say something wrong?”: Towards a safe collaborative chatbot.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * (90) Edward Choi, Mohammad Taha Bahadori, Joshua A Kulas, Andy Schuetz, Walter F Stewart, and Jimeng Sun.  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  arXiv:1608.05745, 2016.
  * (91) Alexandra Chouldechova.  Fair prediction with disparate impact: A study of bias in recidivism prediction instruments.  Big data, 5(2):153–163, 2017.
  * (92) Xu Chu, Ihab F Ilyas, Sanjay Krishnan, and Jiannan Wang.  Data cleaning: Overview and emerging challenges.  In Proceedings of the 2016 international conference on management of data, pages 2201–2206, 2016.
  * (93) Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier.  Parseval networks: Improving robustness to adversarial examples.  In International Conference on Machine Learning, pages 854–863. PMLR, 2017.
  * (94) Edmund M Clarke and Jeannette M Wing.  Formal methods: State of the art and future directions.  ACM Computing Surveys (CSUR), 28(4):626–643, 1996.
  * (95) Edmund M Clarke Jr, Orna Grumberg, Daniel Kroening, Doron Peled, and Helmut Veith.  Model checking.  MIT press, 2018.
  * (96) Graham Cormode.  Personal privacy vs population privacy: learning to attack anonymization.  In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1253–1261, 2011.
  * (97) Patrick Cousot and Radhia Cousot.  Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints.  In Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages, pages 238–252, 1977.
  * (98) Bo Cowgill and Catherine Tucker.  Algorithmic bias: A counterfactual perspective.  NSF Trustworthy Algorithms, 2017.
  * (99) Mark Craven and Jude Shavlik.  Extracting tree-structured representations of trained networks.  Advances in neural information processing systems, 8:24–30, 1995\.
  * (100) Mark W Craven and Jude W Shavlik.  Using sampling and queries to extract rules from trained neural networks.  In Machine learning proceedings 1994, pages 37–45. Elsevier, 1994\.
  * (101) Gabriela F Cretu, Angelos Stavrou, Michael E Locasto, Salvatore J Stolfo, and Angelos D Keromytis.  Casting out demons: Sanitizing training data for anomaly sensors.  In 2008 IEEE Symposium on Security and Privacy (sp 2008), pages 81–95. IEEE, 2008.
  * (102) Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein.  RobustBench: a standardized adversarial robustness benchmark.  arXiv:2010.09670, 2020.
  * (103) Luiz Marcio Cysneiros, Majid Raffi, and Julio Cesar Sampaio do Prado Leite.  Software transparency as a key requirement for self-driving cars.  In 2018 IEEE 26th international requirements engineering conference (RE), pages 382–387. IEEE, 2018.
  * (104) Luiz Marcio Cysneiros and Vera Maria Benjamim Werneck.  An initial analysis on how software transparency and trust influence each other.  In WER. Citeseer, 2009.
  * (105) Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel.  Open problems in cooperative ai.  arXiv:2012.08630, 2020.
  * (106) David Dao.  Awful AI.  https://github.com/daviddao/awful-ai, 2020.
  * (107) Kristof De Vulder, Florian De Rouck, and Gilles Hachez.  The world’s first artificial intelligence act: Europe’s proposal to lead in human-centered AI.  https://www.technologyslegaledge.com/2021/04/the-worlds-first-artificial-intelligence-act-europes-proposal-to-lead-in-human-centered-ai, 2021\.  Accessed: 2021-04-30.
  * (108) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv:1810.04805, 2018.
  * (109) Jay DeYoung, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C Wallace.  Eraser: A benchmark to evaluate rationalized nlp models.  arXiv:1911.03429, 2019.
  * (110) Nicholas Diakopoulos.  Accountability in algorithmic decision making.  Communications of the ACM, 59(2):56–62, 2016.
  * (111) Donna N Dillenberger, Petr Novotny, Qi Zhang, Praveen Jayachandran, Himanshu Gupta, Sandeep Hans, Dinesh Verma, Shreya Chakraborty, JJ Thomas, MM Walli, et al.  Blockchain analytics and artificial intelligence.  IBM Journal of Research and Development, 63(2/3):5–1, 2019.
  * (112) Ahmet Emir Dirik, Hüsrev Taha Sencar, and Nasir Memon.  Analysis of seam-carving-based anonymization of images against prnu noise pattern-based source attribution.  IEEE Transactions on Information Forensics and Security, 9(12):2277–2290, 2014.
  * (113) Julio Cesar Sampaio do Prado Leite and Claudia Cappelli.  Software transparency.  Business & Information Systems Engineering, 2(3):127–139, 2010\.
  * (114) Jonathan Dodge, Q Vera Liao, Yunfeng Zhang, Rachel KE Bellamy, and Casey Dugan.  Explaining models: an empirical study of how explanations impact fairness judgment.  In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 275–285, 2019.
  * (115) Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu.  Evading defenses to transferable adversarial examples by translation-invariant attacks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4312–4321, 2019.
  * (116) Finale Doshi-Velez and Been Kim.  Towards a rigorous science of interpretable machine learning.  arXiv:1702.08608, 2017.
  * (117) Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O’Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, et al.  Accountability of ai under the law: The role of explanation.  arXiv:1711.01134, 2017.
  * (118) Filip Karlo Došilović, Mario Brčić, and Nikica Hlupić.  Explainable artificial intelligence: A survey.  In 2018 41st International convention on information and communication technology, electronics and microelectronics (MIPRO), pages 0210–0215. IEEE, 2018.
  * (119) Tommaso Dreossi, Daniel J Fremont, Shromona Ghosh, Edward Kim, Hadi Ravanbakhsh, Marcell Vazquez-Chanlatte, and Sanjit A Seshia.  Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems.  In International Conference on Computer Aided Verification, pages 432–442. Springer, 2019.
  * (120) Julia Dressel and Hany Farid.  The accuracy, fairness, and limits of predicting recidivism.  Science advances, 4(1):eaao5580, 2018.
  * (121) Pawel Drozdowski, Christian Rathgeb, Antitza Dantcheva, Naser Damer, and Christoph Busch.  Demographic bias in biometrics: A survey on an emerging challenge.  IEEE Transactions on Technology and Society, 1(2):89–103, 2020\.
  * (122) Chris Drummond.  Replicability is not reproducibility: nor is it good science.  2009\.
  * (123) Cynthia Dwork.  Differential privacy: A survey of results.  In International conference on theory and applications of models of computation, pages 1–19. Springer, 2008.
  * (124) Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith.  Calibrating noise to sensitivity in private data analysis.  In Theory of cryptography conference, pages 265–284. Springer, 2006\.
  * (125) Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith.  Calibrating noise to sensitivity in private data analysis.  Journal of Privacy and Confidentiality, 7(3):17–51, 2016.
  * (126) Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  Foundations and Trends in Theoretical Computer Science, 9(3-4):211–407, 2014.
  * (127) Khaled El Emam, Elizabeth Jonker, Luk Arbuckle, and Bradley Malin.  A systematic review of re-identification attacks on health data.  PloS one, 6(12):e28071, 2011.
  * (128) Mike Elgan.  The case against teaching kids to be polite to Alexa.  https://www.fastcompany.com/40588020/the-case-against-teaching-kids-to-be-polite-to-alexa, 2018\.  Accessed: January 22, 2021.
  * (129) Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent.  Why does unsupervised pre-training help deep learning?  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 201–208. JMLR Workshop and Conference Proceedings, 2010.
  * (130) Úlfar Erlingsson, Vasyl Pihur, and Aleksandra Korolova.  Rappor: Randomized aggregatable privacy-preserving ordinal response.  In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, pages 1054–1067, 2014.
  * (131) David Evans, Vladimir Kolesnikov, and Mike Rosulek.  A pragmatic introduction to secure multi-party computation.  Foundations and Trends® in Privacy and Security, 2(2-3), 2017.
  * (132) Exforsys.  What is monkey testing.  http://www.exforsys.com/tutorials/testing-types/monkey-testing.html, 2011\.  Accessed: 2021-07-09.
  * (133) Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song.  Robust physical-world attacks on deep learning visual classification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1625–1634, 2018.
  * (134) Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi.  r.  arXiv:1802.08686, 2018.
  * (135) Nicola Ferro, Norbert Fuhr, Maria Maistro, Tetsuya Sakai, and Ian Soboroff.  Overview of centre@ clef 2019: sequel in the systematic reproducibility realm.  In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 287–300. Springer, 2019.
  * (136) Nicola Ferro, Maria Maistro, Tetsuya Sakai, and Ian Soboroff.  Overview of centre@ clef 2018: a first tale in the systematic reproducibility realm.  In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 239–246. Springer, 2018.
  * (137) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International Conference on Machine Learning, pages 1126–1135. PMLR, 2017.
  * (138) Sam Fletcher and Md Zahidul Islam.  Decision tree classification with differential privacy: A survey.  ACM Computing Surveys (CSUR), 52(4):1–33, 2019.
  * (139) Luciano Floridi.  Establishing the rules for building trustworthy ai.  Nature Machine Intelligence, 1(6):261–262, 2019.
  * (140) Sorelle Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian.  Certifying and removing disparate impact.  arXiv:1412.3756, 2014.
  * (141) Sorelle A Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P Hamilton, and Derek Roth.  A comparative study of fairness-enhancing interventions in machine learning.  In Proceedings of the conference on fairness, accountability, and transparency, pages 329–338, 2019.
  * (142) Jerome Friedman, Trevor Hastie, Robert Tibshirani, et al.  The elements of statistical learning, volume 1.  Springer series in statistics New York, 2001.
  * (143) Nicholas Frosst and Geoffrey Hinton.  Distilling a neural network into a soft decision tree.  arXiv:1711.09784, 2017.
  * (144) Changhong Fu, Miguel A Olivares-Mendez, Ramon Suarez-Fernandez, and Pascual Campoy.  Monocular visual-inertial slam-based collision avoidance strategy for fail-safe uav using fuzzy logic controllers.  Journal of Intelligent & Robotic Systems, 73(1):513–533, 2014\.
  * (145) Iason Gabriel.  Artificial intelligence, values, and alignment.  Minds and Machines, 30(3):411–437, 2020.
  * (146) Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou.  Fairness testing: testing software for discrimination.  In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, pages 498–510, 2017.
  * (147) Adrià Gascón, Phillipp Schoppmann, Borja Balle, Mariana Raykova, Jack Doerner, Samee Zahur, and David Evans.  Privacy-preserving distributed linear regression on high-dimensional data.  Proceedings on Privacy Enhancing Technologies, 2017(4):345–364, 2017.
  * (148) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford.  Datasheets for datasets.  arXiv:1803.09010, 2018.
  * (149) Amirata Ghorbani, Abubakar Abid, and James Zou.  Interpretation of neural networks is fragile.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3681–3688, 2019.
  * (150) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580–587, 2014.
  * (151) Sixue Gong, Xiaoming Liu, and Anil K Jain.  Mitigating face recognition bias via group adaptive classifier.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3414–3424, 2021.
  * (152) John R Goodall, Eric D Ragan, Chad A Steed, Joel W Reed, G David Richardson, Kelly MT Huffer, Robert A Bridges, and Jason A Laska.  Situ: Identifying and explaining suspicious behavior in networks.  IEEE transactions on visualization and computer graphics, 25(1):204–214, 2018.
  * (153) Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Machine learning basics.  Deep learning, 1:98–164, 2016.
  * (154) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  arXiv:1412.6572, 2014.
  * (155) Daniel Greene, Anna Lauren Hoffmann, and Luke Stark.  Better, nicer, clearer, fairer: A critical assessment of the movement for ethical artificial intelligence and machine learning.  In Proceedings of the 52nd Hawaii international conference on system sciences, 2019.
  * (156) Bin Gu, Zhiyuan Dang, Xiang Li, and Heng Huang.  Federated doubly stochastic kernel learning for vertically partitioned data.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2483–2493, 2020.
  * (157) Shixiang Gu and Luca Rigazio.  Towards deep neural network architectures robust to adversarial examples.  arXiv:1412.5068, 2014.
  * (158) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Dino Pedreschi, Franco Turini, and Fosca Giannotti.  Local rule-based explanations of black box decision systems.  arXiv:1805.10820, 2018.
  * (159) Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi.  A survey of methods for explaining black box models.  ACM computing surveys (CSUR), 51(5):1–42, 2018.
  * (160) Odd Erik Gundersen, Yolanda Gil, and David W Aha.  On reproducible ai: Towards reproducible research, open science, and digital scholarship in ai publications.  AI magazine, 39(3):56–68, 2018.
  * (161) Odd Erik Gundersen and Sigbjorn Kjensmo.  State of the art: Reproducibility in artificial intelligence.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * (162) David Gunning and David Aha.  Darpa’s explainable artificial intelligence (xai) program.  AI Magazine, 40(2):44–58, 2019.
  * (163) Ying Guo, Xingxing Wei, Guoqiu Wang, and Bo Zhang.  Meaningful adversarial stickers for face recognition in physical world.  arXiv:2104.06728, 2021.
  * (164) Andreas Haeberlen, Benjamin C Pierce, and Arjun Narayan.  Differential privacy under fire.  In USENIX Security Symposium, volume 33, 2011.
  * (165) Thilo Hagendorff.  The ethics of ai ethics: An evaluation of guidelines.  Minds and Machines, 30(1):99–120, 2020.
  * (166) Gregory D Hager, Ann Drobnis, Fei Fang, Rayid Ghani, Amy Greenwald, Terah Lyons, David C Parkes, Jason Schultz, Suchi Saria, Stephen F Smith, et al.  Artificial intelligence for social good.  arXiv:1901.05406, 2019.
  * (167) Shuo Han, Ufuk Topcu, and George J Pappas.  Differentially private distributed constrained optimization.  IEEE Transactions on Automatic Control, 62(1):50–64, 2016.
  * (168) Karen Hao.  Ai is sending people to jail-and getting it wrong.  MIT Technology Review, https://www.technologyreview.com/2019/01/21/137783/algorithms-criminal-justice-ai/, 2019\.  Accessed: 2021-02-20.
  * (169) Karen Hao.  Making face recognition less biased doesn’t make it less scary.  MIT Technology Review, https://www.technologyreview.com/2019/01/29/137676/making-face-recognition-less-biased-doesnt-make-it-less-scary/, 2019\.  Accessed: 2021-02-20.
  * (170) Moritz Hardt, Eric Price, and Nathan Srebro.  Equality of opportunity in supervised learning.  arXiv:1610.02413, 2016.
  * (171) Stephen Hardy, Wilko Henecka, Hamish Ivey-Law, Richard Nock, Giorgio Patrini, Guillaume Smith, and Brian Thorne.  Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption.  arXiv:1711.10677, 2017.
  * (172) Chaoyang He, Songze Li, Jinhyun So, Xiao Zeng, Mi Zhang, Hongyi Wang, Xiaoyang Wang, Praneeth Vepakomma, Abhishek Singh, Hang Qiu, et al.  Fedml: A research library and benchmark for federated machine learning.  arXiv:2007.13518, 2020.
  * (173) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * (174) Yeye He and Jeffrey F Naughton.  Anonymization of set-valued data via top-down, local generalization.  Proceedings of the VLDB Endowment, 2(1):934–945, 2009.
  * (175) Matthias Hein and Maksym Andriushchenko.  Formal guarantees on the robustness of a classifier against adversarial manipulation.  arXiv:1705.08475, 2017.
  * (176) Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  arXiv:1903.12261, 2019.
  * (177) Melanie Herschel, Ralf Diestelkämper, and Houssem Ben Lahmar.  A survey on provenance: What for? what form? what from?  The VLDB Journal, 26(6):881–906, 2017.
  * (178) Robert R Hoffman, Shane T Mueller, Gary Klein, and Jordan Litman.  Metrics for explainable ai: Challenges and prospects.  arXiv:1812.04608, 2018.
  * (179) Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski.  The dataset nutrition label: A framework to drive higher data quality standards.  arXiv:1805.03677, 2018.
  * (180) Andreas Holzinger.  Interactive machine learning for health informatics: when do we need the human-in-the-loop?  Brain Informatics, 3(2):119–131, 2016.
  * (181) Yuan Hong, Jaideep Vaidya, Haibing Lu, Panagiotis Karras, and Sanjay Goel.  Collaborative search log sanitization: Toward differential privacy and boosted utility.  IEEE Transactions on Dependable and Secure Computing, 12(5):504–518, 2014.
  * (182) Markus Hörwick and Karl-Heinz Siedersberger.  Strategy and architecture of a safety concept for fully automatic and autonomous driving assistance systems.  In 2010 IEEE Intelligent Vehicles Symposium, pages 955–960. IEEE, 2010.
  * (183) Ayanna Howard and Jason Borenstein.  The ugly truth about ourselves and our robot creations: the problem of bias and social inequity.  Science and engineering ethics, 24(5):1521–1536, 2018.
  * (184) Xia Hu, Lingyang Chu, Jian Pei, Weiqing Liu, and Jiang Bian.  Model complexity of deep learning: A survey.  arXiv:2103.05127, 2021.
  * (185) WuLing Huang, Kunfeng Wang, Yisheng Lv, and FengHua Zhu.  Autonomous vehicles testing methods review.  In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pages 163–168. IEEE, 2016.
  * (186) Xiaowei Huang, Daniel Kroening, Wenjie Ruan, James Sharp, Youcheng Sun, Emese Thamo, Min Wu, and Xinping Yi.  A survey of safety and trustworthiness of deep neural networks: Verification, testing, adversarial attack and defence, and interpretability.  Computer Science Review, 37:100270, 2020.
  * (187) Waldemar Hummer, Vinod Muthusamy, Thomas Rausch, Parijat Dube, Kaoutar El Maghraoui, Anupama Murthi, and Punleuk Oum.  Modelops: Cloud-based lifecycle management for reliable and trusted ai.  In 2019 IEEE International Conference on Cloud Engineering (IC2E), pages 113–120. IEEE, 2019.
  * (188) Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International conference on machine learning, pages 448–456. PMLR, 2015.
  * (189) Vasileios Iosifidis, Besnik Fetahu, and Eirini Ntoutsi.  Fae: A fairness-aware ensemble framework.  In 2019 IEEE International Conference on Big Data (Big Data), pages 1375–1380. IEEE, 2019.
  * (190) Richard Isdahl and Odd Erik Gundersen.  Out-of-the-box reproducibility: A survey of machine learning platforms.  In 2019 15th international conference on eScience (eScience), pages 86–95. IEEE, 2019.
  * (191) Alon Jacovi, Ana Marasović, Tim Miller, and Yoav Goldberg.  Formalizing trust in artificial intelligence: Prerequisites, causes and goals of human trust in ai.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 624–635, 2021.
  * (192) Sarthak Jain and Byron C Wallace.  Attention is not explanation.  arXiv:1902.10186, 2019.
  * (193) Carolin EM Jakob, Florian Kohlmayer, Thierry Meurers, Jörg Janne Vehreschild, and Fabian Prasser.  Design and evaluation of a data anonymization pipeline to promote open science on covid-19.  Scientific data, 7(1):1–10, 2020.
  * (194) Marijn Janssen, Paul Brous, Elsa Estevez, Luis S Barbosa, and Tomasz Janowski.  Data governance: Organizing data for trustworthy artificial intelligence.  Government Information Quarterly, 37(3):101493, 2020.
  * (195) Seyyed Ahmad Javadi, Richard Cloete, Jennifer Cobbe, Michelle Seng Ah Lee, and Jatinder Singh.  Monitoring misuse for accountable’artificial intelligence as a service’.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 300–306, 2020.
  * (196) Shouling Ji, Weiqing Li, Prateek Mittal, Xin Hu, and Raheem Beyah.  Secgraph: A uniform and open-source evaluation system for graph data anonymization and de-anonymization.  In 24th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 15), pages 303–318, 2015.
  * (197) Shouling Ji, Weiqing Li, Mudhakar Srivatsa, and Raheem Beyah.  Structural data de-anonymization: Quantification, practice, and implications.  In Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, pages 1040–1053, 2014.
  * (198) Shouling Ji, Prateek Mittal, and Raheem Beyah.  Graph data anonymization, de-anonymization attacks, and de-anonymizability quantification: A survey.  IEEE Communications Surveys & Tutorials, 19(2):1305–1326, 2016\.
  * (199) Zhanglong Ji, Zachary C Lipton, and Charles Elkan.  Differential privacy and machine learning: a survey and review.  arXiv:1412.7584, 2014.
  * (200) Robin Jia.  Building Robust Natural Language Processing Systems.  Stanford University, 2020.
  * (201) Anna Jobin, Marcello Ienca, and Effy Vayena.  The global landscape of ai ethics guidelines.  Nature Machine Intelligence, 1(9):389–399, 2019.
  * (202) Sushant Kafle, Abraham Glasser, Sedeeq Al-khazraji, Larwan Berke, Matthew Seita, and Matt Huenerfauth.  Artificial intelligence fairness in the context of accessibility research on intelligent systems for people who are deaf or hard of hearing.  ACM SIGACCESS Accessibility and Computing, (125):1–1, 2020.
  * (203) Nektaria Kaloudi and Jingyue Li.  The ai-based cyber threat landscape: A survey.  ACM Computing Surveys (CSUR), 53(1):1–34, 2020.
  * (204) Ehud Karnin, Jonathan Greene, and Martin Hellman.  On secret sharing systems.  IEEE Transactions on Information Theory, 29(1):35–41, 1983.
  * (205) Kenji Kawaguchi, Leslie Pack Kaelbling, and Yoshua Bengio.  Generalization in deep learning.  arXiv:1710.05468, 2017.
  * (206) Been Kim and F Doshi-Velez.  Introduction to interpretable machine learning.  Proceedings of the CVPR 2018 Tutorial on Interpretable Machine Learning for Computer Vision, Salt Lake City, UT, USA, 18, 2018.
  * (207) Been Kim, Oluwasanmi Koyejo, Rajiv Khanna, et al.  Examples are not enough, learn to criticize! criticism for interpretability.  In NIPS, pages 2280–2288, 2016.
  * (208) Michael P Kim, Omer Reingold, and Guy N Rothblum.  Fairness through computationally-bounded awareness.  arXiv:1803.03239, 2018.
  * (209) Thomas C King, Nikita Aggarwal, Mariarosaria Taddeo, and Luciano Floridi.  Artificial intelligence crime: An interdisciplinary analysis of foreseeable threats and solutions.  Science and engineering ethics, 26(1):89–120, 2020.
  * (210) Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan.  Inherent trade-offs in the fair determination of risk scores.  arXiv:1609.05807, 2016.
  * (211) B Knott, S Venkataraman, AY Hannun, S Sengupta, M Ibrahim, and LJP van der Maaten.  Crypten: Secure multi-party computation meets machine learning.  In Proceedings of the NeurIPS Workshop on Privacy-Preserving Machine Learning, 2020.
  * (212) Bran Knowles and John T Richards.  The sanction of authority: Promoting public trust in ai.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 262–271, 2021.
  * (213) Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov.  Siamese neural networks for one-shot image recognition.  In ICML deep learning workshop, volume 2. Lille, 2015.
  * (214) Nathan Koenig and Andrew Howard.  Design and use paradigms for gazebo, an open-source multi-robot simulator.  In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)(IEEE Cat. No. 04CH37566), volume 3, pages 2149–2154. IEEE.
  * (215) Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In International Conference on Machine Learning, pages 1885–1894. PMLR, 2017.
  * (216) Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann.  Universal litmus patterns: Revealing backdoor attacks in cnns.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301–310, 2020.
  * (217) Stepan Komkov and Aleksandr Petiushko.  Advhat: Real-world adversarial attack on arcface face id system.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 819–826. IEEE, 2021.
  * (218) Yongbon Koo and SungHoon Kim.  Distributed logging system for ros-based systems.  In 2019 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 1–3. IEEE, 2019.
  * (219) Emmanouil Krasanakis, Eleftherios Spyromitros-Xioufis, Symeon Papadopoulos, and Yiannis Kompatsiaris.  Adaptive sensitive reweighting to mitigate bias in fairness-aware classification.  In Proceedings of the 2018 World Wide Web Conference, pages 853–862, 2018.
  * (220) Anders Krogh and John A Hertz.  A simple weight decay can improve generalization.  In Advances in neural information processing systems, pages 950–957, 1992.
  * (221) David Krueger, Nicolas Ballas, Stanislaw Jastrzebski, Devansh Arpit, Maxinder S Kanwal, Tegan Maharaj, Emmanuel Bengio, Asja Fischer, and Aaron Courville.  Deep nets don’t learn via memorization.  2017\.
  * (222) Kun Kuang, Lian Li, Zhi Geng, Lei Xu, Kun Zhang, Beishui Liao, Huaxin Huang, Peng Ding, Wang Miao, and Zhichao Jiang.  Causal inference.  Engineering, 6(3):253–263, 2020.
  * (223) James Kuczmarski.  Reducing gender bias in google translate.  https://blog.google/products/translate/reducing-gender-bias-google-translate/, 2018\.  Accessed: April 28, 2021.
  * (224) Todd Kulesza, Simone Stumpf, Margaret Burnett, and Irwin Kwan.  Tell me more? the effects of mental model soundness on personalizing an intelligent agent.  In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 1–10, 2012.
  * (225) Abhishek Kumar, Tristan Braud, Sasu Tarkoma, and Pan Hui.  Trustworthy ai in the age of pervasive computing and big data.  In 2020 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pages 1–6. IEEE, 2020.
  * (226) Nishant Kumar, Mayank Rathee, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma.  Cryptflow: Secure tensorflow inference.  In 2020 IEEE Symposium on Security and Privacy (SP), pages 336–353. IEEE, 2020.
  * (227) Alexey Kurakin, Ian Goodfellow, Samy Bengio, et al.  Adversarial examples in the physical world, 2016.
  * (228) Clete A Kushida, Deborah A Nichols, Rik Jadrnicek, Ric Miller, James K Walsh, and Kara Griffin.  Strategies for de-identification and anonymization of electronic health record data for use in multicenter research studies.  Medical care, 50(Suppl):S82, 2012.
  * (229) Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec.  Interpretable & explorable approximations of black box models.  arXiv:1707.01154, 2017.
  * (230) Alexander Lavin, Ciaràn M Gilligan-Lee, Alessya Visnjic, Siddha Ganju, Dava Newman, Sujoy Ganguly, Danny Lange, Atılım Güneş Baydin, Amit Sharma, Adam Gibson, et al.  Technology readiness levels for machine learning systems.  arXiv:2101.03989, 2021.
  * (231) Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana.  Certified robustness to adversarial examples with differential privacy.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 656–672. IEEE, 2019.
  * (232) Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Zheng Zhang, Yaoqin Zhang, Xiang Li, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao.  ConvLab: Multi-domain end-to-end dialog system platform.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 64–69, Florence, Italy, July 2019. Association for Computational Linguistics.
  * (233) David Leslie.  Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of ai systems in the public sector.  Available at SSRN 3403301, 2019.
  * (234) Dave Lewis, David Filip, and Harshvardhan J Pandit.  An ontology for standardising trustworthy ai.  2021\.
  * (235) Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik.  Data poisoning attacks on factorization-based collaborative filtering.  Advances in neural information processing systems, 29:1885–1893, 2016.
  * (236) Ke Li and Jitendra Malik.  Learning to optimize.  arXiv:1606.01885, 2016.
  * (237) Pengcheng Li, Jinfeng Yi, Bowen Zhou, and Lijun Zhang.  Improving the robustness of deep neural networks via adversarial training with triplet loss.  arXiv:1905.11713, 2019.
  * (238) Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang.  On the convergence of fedavg on non-iid data.  International Conference on Learning Representation, 2019.
  * (239) Yiming Li, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shu-Tao Xia.  Backdoor learning: A survey.  arXiv:2007.08745, 2020.
  * (240) Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, Riccardo Giol, Marcello Restelli, and Danilo Romano.  Combining reinforcement learning with rule-based controllers for transparent and general decision-making in autonomous driving.  Robotics and Autonomous Systems, 131:103568, 2020.
  * (241) Yehuda Lindell.  Secure multiparty computation (mpc).  Communications of the ACM, 64(2):300, 2021.
  * (242) Yehuda Lindell and Benny Pinkas.  A proof of security of yao’s protocol for two-party computation.  Journal of cryptology, 22(2):161–188, 2009.
  * (243) Zachary C Lipton.  The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery.  Queue, 16(3):31–57, 2018.
  * (244) Bo Liu, Chaowei Tan, Jiazhou Wang, Tao Zeng, Huasong Shan, Houpu Yao, Huang Heng, Peng Dai, Liefeng Bo, and Yanqing Chen.  Fedlearn-algo: A flexible open-source privacy-preserving machine learning platform.  arXiv:2107.04129, 2021.
  * (245) Haochen Liu, Yiqi Wang, Wenqi Fan, Xiaorui Liu, Yaxin Li, Shaili Jain, Anil K Jain, and Jiliang Tang.  Trustworthy ai: A computational perspective.  arXiv:2107.06641, 2021.
  * (246) Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg.  Fine-pruning: Defending against backdooring attacks on deep neural networks.  In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273–294. Springer, 2018.
  * (247) Kun Liu and Evimaria Terzi.  Towards identity anonymization on graphs.  In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 93–106, 2008.
  * (248) Ming Liu, Yukang Ding, Min Xia, Xiao Liu, Errui Ding, Wangmeng Zuo, and Shilei Wen.  Stgan: A unified selective transfer network for arbitrary image attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3673–3682, 2019.
  * (249) Yang Liu, Yan Kang, Chaoping Xing, Tianjian Chen, and Qiang Yang.  A secure federated transfer learning framework.  IEEE Intelligent Systems, 35(4):70–82, 2020.
  * (250) Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang.  Federated learning for open banking.  In Federated Learning, pages 240–254. Springer, 2020.
  * (251) Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * (252) Scott M Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  Advances in Neural Information Processing Systems, 30:4765–4774, 2017.
  * (253) Lei Ma, Felix Juefei-Xu, Fuyuan Zhang, Jiyuan Sun, Minhui Xue, Bo Li, Chunyang Chen, Ting Su, Li Li, Yang Liu, et al.  Deepgauge: Multi-granularity testing criteria for deep learning systems.  In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, pages 120–131, 2018.
  * (254) Gabriel Resende Machado, Eugênio Silva, and Ronaldo Ribeiro Goldschmidt.  Adversarial machine learning in image classification: A survey towards the defender’s perspective.  arXiv:2009.03728, 2020.
  * (255) Michael A Madaio, Luke Stark, Jennifer Wortman Vaughan, and Hanna Wallach.  Co-designing checklists to understand organizational challenges and opportunities around fairness in AI.  In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–14, 2020.
  * (256) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  arXiv:1706.06083, 2017.
  * (257) Silvia Magdici and Matthias Althoff.  Fail-safe motion planning of autonomous vehicles.  In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pages 452–458. IEEE, 2016.
  * (258) Pratyush Maini, Eric Wong, and Zico Kolter.  Adversarial robustness against the union of multiple perturbation models.  In International Conference on Machine Learning, pages 6640–6650. PMLR, 2020.
  * (259) Abdul Majeed and Sungchang Lee.  Anonymization techniques for privacy preserving data publishing: A comprehensive survey.  IEEE Access, 2020.
  * (260) Sasu Mäkinen, Henrik Skogström, Eero Laaksonen, and Tommi Mikkonen.  Who needs mlops: What data scientists seek to accomplish and how can mlops help?  arXiv:2103.08942, 2021.
  * (261) Dusica Marijan and Arnaud Gotlieb.  Software testing for machine learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13576–13582, 2020.
  * (262) David J Martin, Daniel Kifer, Ashwin Machanavajjhala, Johannes Gehrke, and Joseph Y Halpern.  Worst-case background knowledge for privacy-preserving data publishing.  In 2007 IEEE 23rd International Conference on Data Engineering, pages 126–135. IEEE, 2007.
  * (263) Andre Martins and Ramon Astudillo.  From softmax to sparsemax: A sparse model of attention and multi-label classification.  In International Conference on Machine Learning, pages 1614–1623. PMLR, 2016.
  * (264) Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee.  Hatexplain: A benchmark dataset for explainable hate speech detection.  arXiv:2012.10289, 2020.
  * (265) Maxim Maximov, Ismail Elezi, and Laura Leal-Taixé.  Ciagan: Conditional identity anonymization generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5447–5456, 2020.
  * (266) Frank McKeen, Ilya Alexandrovich, Ittai Anati, Dror Caspi, Simon Johnson, Rebekah Leslie-Hurd, and Carlos Rozas.  Intel® software guard extensions (intel® sgx) support for dynamic memory management inside an enclave.  In Proceedings of the Hardware and Architectural Support for Security and Privacy 2016, pages 1–9. 2016.
  * (267) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In Artificial Intelligence and Statistics, pages 1273–1282. PMLR, 2017.
  * (268) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  arXiv:1908.09635, 2019.
  * (269) Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff.  On detecting adversarial perturbations.  arXiv:1702.04267, 2017.
  * (270) Silvio Micali, Oded Goldreich, and Avi Wigderson.  How to play any mental game.  In Proceedings of the Nineteenth ACM Symp. on Theory of Computing, STOC, pages 218–229. ACM, 1987.
  * (271) Tim Miller.  Explanation in artificial intelligence: Insights from the social sciences.  Artificial intelligence, 267:1–38, 2019.
  * (272) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru.  Model cards for model reporting.  In Proceedings of the conference on fairness, accountability, and transparency, pages 220–229, 2019.
  * (273) Brent Mittelstadt.  Principles alone cannot guarantee ethical ai.  Nature Machine Intelligence, 1(11):501–507, 2019.
  * (274) Payman Mohassel and Yupeng Zhang.  Secureml: A system for scalable privacy-preserving machine learning.  In 2017 IEEE Symposium on Security and Privacy (SP), pages 19–38. IEEE, 2017.
  * (275) Sina Mohseni, Jeremy E Block, and Eric D Ragan.  A human-grounded evaluation benchmark for local explanations of machine learning.  arXiv:1801.05075, 2018.
  * (276) Sina Mohseni, Niloofar Zarei, and Eric D Ragan.  A multidisciplinary survey and framework for design and evaluation of explainable ai systems.  arXiv:1811.11839, 2018.
  * (277) Christoph Molnar.  Interpretable machine learning.  Lulu. com, 2020.
  * (278) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574–2582, 2016.
  * (279) Miranda Mourby, Elaine Mackey, Mark Elliot, Heather Gowans, Susan E Wallace, Jessica Bell, Hannah Smith, Stergios Aidinlis, and Jane Kaye.  Are ‘pseudonymised’ data always personal data? implications of the gdpr for administrative data research in the uk.  Computer Law & Security Review, 34(2):222–233, 2018.
  * (280) Mark Wilfried Müller.  Increased autonomy for quadrocopter systems: trajectory generation, fail-safe strategies and state estimation.  PhD thesis, ETH Zurich, 2016.
  * (281) Mehmet Ercan Nergiz, Maurizio Atzori, and Chris Clifton.  Hiding the presence of individuals from shared databases.  In Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pages 665–676, 2007.
  * (282) Maria-Irina Nicolae, Mathieu Sinn, Minh Ngoc Tran, Beat Buesser, Ambrish Rawat, Martin Wistuba, Valentina Zantedeschi, Nathalie Baracaldo, Bryant Chen, Heiko Ludwig, et al.  Adversarial robustness toolbox v1. 0.0.  arXiv:1807.01069, 2018.
  * (283) Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana.  Interpretml: A unified framework for machine learning interpretability.  arXiv:1909.09223, 2019.
  * (284) Yingwei Pan, Yehao Li, Jianjie Luo, Jun Xu, Ting Yao, and Tao Mei.  Auto-captions on gif: A large-scale video-sentence dataset for vision-language pre-training.  arXiv:2007.02375, 2020.
  * (285) Guansong Pang, Chunhua Shen, Longbing Cao, and Anton van den Hengel.  Deep learning for anomaly detection: A review.  arXiv:2007.02500, 2020.
  * (286) Namkee Park, Kyungeun Jang, Seonggyeol Cho, and Jinyoung Choi.  Use of offensive language in human-artificial intelligence chatbot interaction: The effects of ethical ideology, social competence, and perceived humanlikeness.  Computers in Human Behavior, 121:106795, 2021.
  * (287) Andrea Paudice, Luis Muñoz-González, and Emil C Lupu.  Label sanitization against label flipping poisoning attacks.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 5–15. Springer, 2018.
  * (288) Judea Pearl et al.  Causal inference in statistics: An overview.  Statistics surveys, 3:96–146, 2009.
  * (289) Kexin Pei, Yinzhi Cao, Junfeng Yang, and Suman Jana.  Deepxplore: Automated whitebox testing of deep learning systems.  In proceedings of the 26th Symposium on Operating Systems Principles, pages 1–18, 2017.
  * (290) Antonio Pérez, M Isabel García, Manuel Nieto, José L Pedraza, Santiago Rodríguez, and Juan Zamorano.  Argos: An advanced in-vehicle data recorder on a massively sensorized vehicle for car driver behavior experimentation.  IEEE Transactions on Intelligent Transportation Systems, 11(2):463–473, 2010.
  * (291) Zahid Pervaiz, Walid G Aref, Arif Ghafoor, and Nagabhushana Prabhu.  Accuracy-constrained privacy-preserving access control mechanism for relational data.  IEEE Transactions on Knowledge and Data Engineering, 26(4):795–807, 2013.
  * (292) Samuele Lo Piano.  Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward.  Humanities and Social Sciences Communications, 7(1):1–7, 2020.
  * (293) Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle.  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  arXiv:2003.12206, 2020.
  * (294) Sandro Pinto and Nuno Santos.  Demystifying arm trustzone: A comprehensive survey.  ACM Computing Surveys (CSUR), 51(6):1–36, 2019.
  * (295) David Piorkowski, Daniel González, John Richards, and Stephanie Houde.  Towards evaluating and eliciting high-quality documentation for intelligent systems.  arXiv:2011.08774, 2020.
  * (296) Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger.  On fairness and calibration.  arXiv:1709.02012, 2017.
  * (297) Forough Poursabzi-Sangdeh, Daniel G Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach.  Manipulating and measuring model interpretability.  arXiv:1802.07810, 2018.
  * (298) Peng Qi, Jing Huang, Youzheng Wu, Xiaodong He, and Bowen Zhou.  Conversational ai systems for social good: Opportunities and challenges.  arXiv:2105.06457, 2021.
  * (299) Stephan Rabanser, Stephan Günnemann, and Zachary C Lipton.  Failing loudly: An empirical study of methods for detecting dataset shift.  arXiv:1810.11953, 2018.
  * (300) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * (301) Aditi Raghunathan, Jacob Steinhardt, and Percy Liang.  Certified defenses against adversarial examples.  arXiv:1801.09344, 2018.
  * (302) Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang.  Adversarial training can hurt generalization.  arXiv:1906.06032, 2019.
  * (303) Inioluwa Deborah Raji, Andrew Smart, Rebecca N White, Margaret Mitchell, Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes.  Closing the ai accountability gap: defining an end-to-end framework for internal algorithmic auditing.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 33–44, 2020.
  * (304) Inioluwa Deborah Raji and Jingying Yang.  About ml: Annotation and benchmarking on understanding and transparency of machine learning lifecycles.  arXiv:1912.06166, 2019.
  * (305) Bogdana Rakova, Jingying Yang, Henriette Cramer, and Rumman Chowdhury.  Where responsible ai meets reality: Practitioner perspectives on enablers for shifting organizational practices.  Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1–23, 2021.
  * (306) Raghavendra Ramachandra and Christoph Busch.  Presentation attack detection methods for face recognition systems: A comprehensive survey.  ACM Computing Surveys (CSUR), 50(1):1–37, 2017.
  * (307) Deevashwer Rathee, Mayank Rathee, Nishant Kumar, Nishanth Chandran, Divya Gupta, Aseem Rastogi, and Rahul Sharma.  Cryptflow2: Practical 2-party secure inference.  In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security, pages 325–342, 2020.
  * (308) Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Konečnỳ, Sanjiv Kumar, and H Brendan McMahan.  Adaptive federated optimization.  International Conference on Learning Representation, 2021.
  * (309) Dillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker.  Algorithmic impact assessments: A practical framework for public agency accountability.  AI Now Institute, pages 1–22, 2018.
  * (310) Xuebin Ren, Chia-Mu Yu, Weiren Yu, Shusen Yang, Xinyu Yang, Julie A McCann, and S Yu Philip.  $\\{$LoPub$\\}$: High-dimensional crowdsourced data publication with local differential privacy.  IEEE Transactions on Information Forensics and Security, 13(9):2151–2166, 2018.
  * (311) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  “why should i trust you?” explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135–1144, 2016.
  * (312) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  Anchors: High-precision model-agnostic explanations.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * (313) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh.  Beyond accuracy: Behavioral testing of nlp models with checklist.  arXiv:2005.04118, 2020.
  * (314) Nicola Rieke, Jonny Hancox, Wenqi Li, Fausto Milletari, Holger R Roth, Shadi Albarqouni, Spyridon Bakas, Mathieu N Galtier, Bennett A Landman, Klaus Maier-Hein, et al.  The future of digital health with federated learning.  NPJ digital medicine, 3(1):1–7, 2020.
  * (315) Yuji Roh, Kangwook Lee, Steven Whang, and Changho Suh.  Fr-train: A mutual information-based approach to fair and robust training.  In International Conference on Machine Learning, pages 8147–8157. PMLR, 2020.
  * (316) Avi Rosenfeld and Ariella Richardson.  Explainability in human–agent systems.  Autonomous Agents and Multi-Agent Systems, 33(6):673–705, 2019\.
  * (317) Francisca Rosique, Pedro J Navarro, Carlos Fernández, and Antonio Padilla.  A systematic review of perception system and simulators for autonomous vehicles research.  Sensors, 19(3):648, 2019.
  * (318) Andrew Ross and Finale Doshi-Velez.  Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * (319) Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  Faceforensics++: Learning to detect manipulated facial images.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1–11, 2019.
  * (320) Olivia Rudgard.  ‘alexa generation’ may be learning bad manners from talking to digital assistants, report warns.  https://www.telegraph.co.uk/news/2018/01/31/alexa-generation-could-learning-bad-manners-talking-digital/, 2018\.  Accessed: January 10, 2021.
  * (321) Mohamed Sabt, Mohammed Achemlal, and Abdelmadjid Bouabdallah.  Trusted execution environment: what it is, and what it is not.  In 2015 IEEE Trustcom/BigDataSE/ISPA, volume 1, pages 57–64. IEEE, 2015.
  * (322) Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7374–7383, 2019.
  * (323) Pedro Saleiro, Benedict Kuester, Loren Hinkson, Jesse London, Abby Stevens, Ari Anisfeld, Kit T Rodolfa, and Rayid Ghani.  Aequitas: A bias and fairness audit toolkit.  arXiv:1811.05577, 2018.
  * (324) Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever.  Evolution strategies as a scalable alternative to reinforcement learning.  arXiv:1703.03864, 2017.
  * (325) Pierangela Samarati and Latanya Sweeney.  Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression.  1998\.
  * (326) Nithya Sambasivan, Shivani Kapania, Hannah Highfill, Diana Akrong, Praveen Paritosh, and Lora M Aroyo.  “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai.  In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2021.
  * (327) Elena Samuylova.  Machine learning monitoring: What it is, and what we are missing.  https://towardsdatascience.com/machine-learning-monitoring-what-it-is-and-what-we-are-missing-e644268023ba, 2020\.  Accessed: 2021-05-18.
  * (328) Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort.  Auditing algorithms: Research methods for detecting discrimination on internet platforms.  Data and discrimination: converting critical concerns into productive inquiry, 22:4349–4357, 2014.
  * (329) Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap.  Meta-learning with memory-augmented neural networks.  In International conference on machine learning, pages 1842–1850. PMLR, 2016.
  * (330) Kanthi Sarpatwar, Roman Vaculin, Hong Min, Gong Su, Terry Heath, Giridhar Ganapavarapu, and Donna Dillenberger.  Towards enabling trusted artificial intelligence via blockchain.  In Policy-based autonomic data governance, pages 137–153. Springer, 2019.
  * (331) Laura Schelenz, Avi Segal, and Kobi Gal.  Applying transparency in artificial intelligence based personalization systems.  arXiv e-prints, pages arXiv–2004, 2020.
  * (332) Sebastian Schelter, Joos-Hendrik Boese, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert.  Automatically tracking metadata and provenance of machine learning experiments.  In Machine Learning Systems Workshop at NIPS, pages 27–29, 2017\.
  * (333) Vitaly Schetinin, Jonathan E Fieldsend, Derek Partridge, Timothy J Coats, Wojtek J Krzanowski, Richard M Everson, Trevor C Bailey, and Adolfo Hernandez.  Confident interpretation of bayesian decision tree ensembles for clinical applications.  IEEE Transactions on Information Technology in Biomedicine, 11(3):312–319, 2007.
  * (334) Daniel Schiff, Jason Borenstein, Justin Biddle, and Kelly Laas.  Ai ethics in the public, private, and ngo sectors: A review of a global document collection.  IEEE Transactions on Technology and Society, 2021.
  * (335) Albrecht Schmidt and Thomas Herrmann.  Intervention user interfaces: a new interaction paradigm for automated systems.  Interactions, 24(5):40–45, 2017.
  * (336) David Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo, and Dan Dennison.  Hidden technical debt in machine learning systems.  Advances in neural information processing systems, 28:2503–2511, 2015.
  * (337) Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In Proceedings of the IEEE international conference on computer vision, pages 618–626, 2017.
  * (338) Sanjit A Seshia, Dorsa Sadigh, and S Shankar Sastry.  Towards verified artificial intelligence.  arXiv:1606.08514, 2016.
  * (339) Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter.  A general framework for adversarial examples with objectives.  ACM Transactions on Privacy and Security (TOPS), 22(3):1–30, 2019\.
  * (340) Micah J Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al.  Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data.  Scientific reports, 10(1):1–12, 2020.
  * (341) Haoyi Shi, Chao Jiang, Wenrui Dai, Xiaoqian Jiang, Yuzhe Tang, Lucila Ohno-Machado, and Shuang Wang.  Secure multi-party computation grid logistic regression (smac-glore).  BMC medical informatics and decision making, 16(3):175–187, 2016\.
  * (342) Ben Shneiderman.  Bridging the gap between ethics and practice: Guidelines for reliable, safe, and trustworthy human-centered ai systems.  ACM Transactions on Interactive Intelligent Systems (TiiS), 10(4):1–31, 2020.
  * (343) Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning important features through propagating activation differences.  In International Conference on Machine Learning, pages 3145–3153. PMLR, 2017.
  * (344) Umair Siddique.  Safetyops.  arXiv:2008.04461, 2020.
  * (345) Samuel Henrique Silva and Peyman Najafirad.  Opportunities and challenges in deep learning adversarial robustness: A survey.  arXiv:2007.00753, 2020.
  * (346) Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  arXiv:1312.6034, 2013.
  * (347) Jatinder Singh, Jennifer Cobbe, and Chris Norval.  Decision provenance: Harnessing data flow for accountable systems.  IEEE Access, 7:6562–6574, 2018.
  * (348) Richa Singh, Mayank Vatsa, and Nalini Ratha.  Trustworthy ai.  In 8th ACM IKDD CODS and 26th COMAD, pages 449–453. 2021.
  * (349) Tomáš Sixta, Julio CS Jacques Junior, Pau Buch-Cardona, Eduard Vazquez, and Sergio Escalera.  Fairface challenge at eccv 2020: analyzing bias in face recognition.  In European Conference on Computer Vision, pages 463–481. Springer, 2020.
  * (350) Jake Snell, Kevin Swersky, and Richard S Zemel.  Prototypical networks for few-shot learning.  arXiv:1703.05175, 2017.
  * (351) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * (352) Jonathan Stray.  Aligning ai optimization to community well-being.  International Journal of Community Well-Being, 3(4):443–463, 2020\.
  * (353) Simone Stumpf, Adrian Bussone, and Dympna O’sullivan.  Explanations considered harmful? user interactions with machine learning systems.  In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), 2016.
  * (354) Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao.  Is robustness the cost of accuracy?–a comprehensive study on the robustness of 18 deep image classification models.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 631–648, 2018.
  * (355) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  Vl-bert: Pre-training of generic visual-linguistic representations.  arXiv:1908.08530, 2019.
  * (356) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In International Conference on Machine Learning, pages 3319–3328. PMLR, 2017.
  * (357) Supreme Audit Institutions of Finland, Germany, the Netherlands, Norway and the UK.  Auditing machine learning algorithms.  https://www.auditingalgorithms.net/index.html, 2020.
  * (358) Ki Hyun Tae, Yuji Roh, Young Hun Oh, Hyunsu Kim, and Steven Euijong Whang.  Data cleaning for accurate, fair, and robust models: A big data-ai integration approach.  In Proceedings of the 3rd International Workshop on Data Management for End-to-End Machine Learning, pages 1–4, 2019.
  * (359) Sarah Tan, Matvey Soloviev, Giles Hooker, and Martin T Wells.  Tree space prototypes: Another look at making tree ensembles interpretable.  In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, pages 23–34, 2020.
  * (360) Benedict Tay, Younbo Jung, and Taezoon Park.  When stereotypes meet robots: the double-edge sword of robot gender and personality in human–robot interaction.  Computers in Human Behavior, 38:75–84, 2014.
  * (361) Brad Templaton.  Tesla’s "shadow" testing offers a useful advantage on the biggest problem in robocars.  https://www.forbes.com/sites/bradtempleton/2019/04/29/teslas-shadow-testing-offers-a-useful-advantage-on-the-biggest-problem-in-robocars/?sh=7a960b9e3c06, 2019\.  Accessed: 2021-06-15.
  * (362) Manolis Terrovitis, Nikos Mamoulis, and Panos Kalnis.  Privacy-preserving anonymization of set-valued data.  Proceedings of the VLDB Endowment, 1(1):115–125, 2008.
  * (363) Scott Thiebes, Sebastian Lins, and Ali Sunyaev.  Trustworthy artificial intelligence.  Electronic Markets, pages 1–18, 2020.
  * (364) Martijn Tideman.  Scenario-based simulation environment for assistance systems.  ATZautotechnology, 10(1):28–32, 2010.
  * (365) Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012.
  * (366) Nenad Tomašev, Julien Cornebise, Frank Hutter, Shakir Mohamed, Angela Picciariello, Bec Connelly, Danielle CM Belgrave, Daphne Ezer, Fanny Cachat van der Haert, Frank Mugisha, et al.  Ai for social good: unlocking the opportunity for positive impact.  Nature Communications, 11(1):1–6, 2020.
  * (367) Ehsan Toreini, Mhairi Aitken, Kovila Coopamootoo, Karen Elliott, Carlos Gonzalez Zelaya, and Aad van Moorsel.  The relationship between trust in ai and trustworthy machine learning technologies.  In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 272–283, 2020.
  * (368) Cecília Torres, Walter Franklin, and Laura Martins.  Accessibility in chatbots: The state of the art in favor of users with visual impairment.  In International Conference on Applied Human Factors and Ergonomics, pages 623–635. Springer, 2018.
  * (369) Florian Tramer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, Jean-Pierre Hubaux, Mathias Humbert, Ari Juels, and Huang Lin.  Fairtest: Discovering unwarranted associations in data-driven applications.  In 2017 IEEE European Symposium on Security and Privacy (EuroS&P), pages 401–416. IEEE, 2017.
  * (370) Florian Tramer, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel.  Ensemble adversarial training: Attacks and defenses.  arXiv:1705.07204, 2017.
  * (371) Amos Treiber, Alejandro Molina, Christian Weinert, Thomas Schneider, and Kristian Kersting.  Cryptospn: Privacy-preserving sum-product network inference.  arXiv:2002.00801, 2020.
  * (372) Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry.  Robustness may be at odds with accuracy.  arXiv:1805.12152, 2018.
  * (373) Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang, Xiaodong He, and Bowen Zhou.  Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9073–9080, 2020.
  * (374) Matteo Turilli and Luciano Floridi.  The ethics of information transparency.  Ethics and Information Technology, 11(2):105–112, 2009.
  * (375) UNESCO and EQUALS Skills Coalition.  I’d blush if I could: closing gender divides in digital skills through education.  https://unesdoc.unesco.org/ark:/48223/pf0000367416.page=1, 2019\.  Accessed: January 21, 2021.
  * (376) Caterina Urban and Antoine Miné.  A review of formal methods applied to machine learning.  arXiv:2104.02466, 2021.
  * (377) Joaquin Vanschoren.  Meta-learning: A survey.  arXiv:1810.03548, 2018.
  * (378) Vladimir Vapnik.  The nature of statistical learning theory.  Springer science & business media, 2013.
  * (379) Kush R Varshney.  Trustworthy machine learning and artificial intelligence.  XRDS: Crossroads, The ACM Magazine for Students, 25(3):26–29, 2019\.
  * (380) Joel Vaughan, Agus Sudjianto, Erind Brahimi, Jie Chen, and Vijayan N Nair.  Explainable neural networks based on additive index models.  arXiv:1806.01933, 2018.
  * (381) Sahil Verma and Julia Rubin.  Fairness definitions explained.  In 2018 ieee/acm international workshop on software fairness (fairware), pages 1–7. IEEE, 2018.
  * (382) Sandra Wachter, Brent Mittelstadt, and Chris Russell.  Counterfactual explanations without opening the black box: Automated decisions and the gdpr.  Harv. JL & Tech., 31:841, 2017.
  * (383) Wendell Wallach and Colin Allen.  Moral machines: Teaching robots right from wrong.  Oxford University Press, 2008.
  * (384) Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao.  Neural cleanse: Identifying and mitigating backdoor attacks in neural networks.  In 2019 IEEE Symposium on Security and Privacy (SP), pages 707–723. IEEE, 2019.
  * (385) Hao Wang, Dihong Gong, Zhifeng Li, and Wei Liu.  Decorrelated adversarial learning for age-invariant face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3527–3536, 2019.
  * (386) Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Wenjun Zeng, and Tao Qin.  Generalizing to unseen domains: A survey on domain generalization.  arXiv:2103.03097, 2021.
  * (387) Lingxiao Wang and Quanquan Gu.  Differentially private iterative gradient hard thresholding for sparse learning.  In 28th International Joint Conference on Artificial Intelligence, 2019.
  * (388) Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang.  Racial faces in the wild: Reducing racial bias by information maximization adaptation network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 692–702, 2019.
  * (389) Qian Wang, Yan Zhang, Xiao Lu, Zhibo Wang, Zhan Qin, and Kui Ren.  Real-time and spatio-temporal crowd-sourced social network data publishing with differential privacy.  IEEE Transactions on Dependable and Secure Computing, 15(4):591–606, 2016.
  * (390) Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, Yixin Cao, and Tat-Seng Chua.  Explainable reasoning over knowledge graphs for recommendation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5329–5336, 2019.
  * (391) Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni.  Generalizing from a few examples: A survey on few-shot learning.  ACM Computing Surveys (CSUR), 53(3):1–34, 2020.
  * (392) Yingxu Wang and Jingqiu Shao.  Measurement of the cognitive functional complexity of software.  In The Second IEEE International Conference on Cognitive Informatics, 2003. Proceedings., pages 67–74. IEEE, 2003.
  * (393) Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu.  On the convergence and robustness of adversarial training.  In ICML, volume 1, page 2, 2019.
  * (394) Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor.  Federated learning with differential privacy: Algorithms and performance analysis.  IEEE Transactions on Information Forensics and Security, 15:3454–3469, 2020.
  * (395) Daniel S Weld and Gagan Bansal.  The challenge of crafting intelligible intelligence.  Communications of the ACM, 62(6):70–79, 2019.
  * (396) Tsui-Wei Weng, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel.  Evaluating the robustness of neural networks: An extreme value theory approach.  arXiv:1801.10578, 2018.
  * (397) Chathurika S Wickramasinghe, Daniel L Marino, Javier Grandio, and Milos Manic.  Trustworthy ai development guidelines for human system interaction.  In 2020 13th International Conference on Human System Interaction (HSI), pages 130–136. IEEE, 2020.
  * (398) Sarah Wiegreffe and Ana Marasović.  Teach me to explain: A review of datasets for explainable nlp.  arXiv:2102.12060, 2021.
  * (399) Sarah Wiegreffe and Yuval Pinter.  Attention is not not explanation.  arXiv:1908.04626, 2019.
  * (400) Jonathan R Williford, Brandon B May, and Jeffrey Byrne.  Explainable face recognition.  In European Conference on Computer Vision, pages 248–263. Springer, 2020.
  * (401) Christo Wilson, Avijit Ghosh, Shan Jiang, Alan Mislove, Lewis Baker, Janelle Szary, Kelly Trindel, and Frida Polli.  Building and auditing fair algorithms: A case study in candidate screening.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 666–677, 2021.
  * (402) Jeannette M Wing.  Trustworthy ai.  arXiv:2002.06276, 2020.
  * (403) Matthew Wood et al.  Safety first for autonomous driving.  https://www.daimler.com/documents/innovation/other/safety-first-for-automated-driving.pdf, 2019\.  Accessed: 2021-02-20.
  * (404) Bernhard Wymann, Eric Espié, Christophe Guionneau, Christos Dimitrakakis, Rémi Coulom, and Andrew Sumner.  Torcs, the open racing car simulator.  Software available at http://torcs. sourceforge. net, 4(6):2, 2000\.
  * (405) Zihao Xiao, Xianfeng Gao, Chilin Fu, Yinpeng Dong, Wei Gao, Xiaolu Zhang, Jun Zhou, and Jun Zhu.  Improving transferability of adversarial patches on face recognition with generative models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11845–11854, 2021.
  * (406) XinhuaNet.  Precisely detecting desperate emotions, this AI customer service saved a life by alerting help.  http://www.xinhuanet.com/tech/2020-06/24/c_1126152953.htm, 2020\.  Accessed: January 19, 2021 (in Chinese).
  * (407) Depeng Xu, Shuhan Yuan, and Xintao Wu.  Achieving differential privacy and fairness in logistic regression.  In Companion Proceedings of The 2019 World Wide Web Conference, pages 594–599, 2019.
  * (408) Han Xu, Xiaorui Liu, Yaxin Li, and Jiliang Tang.  To be robust or to be fair: Towards fairness in adversarial training.  arXiv:2010.06121, 2020.
  * (409) Huan Xu and Shie Mannor.  Robustness and generalization.  Machine learning, 86(3):391–423, 2012.
  * (410) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057. PMLR, 2015.
  * (411) Qian Xu, Md Tanvir Arafin, and Gang Qu.  Security of neural networks from hardware perspective: A survey and beyond.  In 2021 26th Asia and South Pacific Design Automation Conference (ASP-DAC), pages 449–454. IEEE, 2021.
  * (412) Roman V Yampolskiy.  Predicting future ai failures from historic examples.  foresight, 2019.
  * (413) Lei Yang, Xiaohang Zhan, Dapeng Chen, Junjie Yan, Chen Change Loy, and Dahua Lin.  Learning to cluster faces on an affinity graph.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2298–2306, 2019.
  * (414) Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong.  Federated machine learning: Concept and applications.  ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1–19, 2019.
  * (415) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning.  Hotpotqa: A dataset for diverse, explainable multi-hop question answering.  arXiv:1809.09600, 2018.
  * (416) Andrew C Yao.  Protocols for secure computations.  In 23rd annual symposium on foundations of computer science (sfcs 1982), pages 160–164. IEEE, 1982.
  * (417) Andrew Chi-Chih Yao.  How to generate and exchange secrets.  In 27th Annual Symposium on Foundations of Computer Science), pages 162–167. IEEE, 1986.
  * (418) Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei.  Seco: Exploring sequence supervision for unsupervised representation learning.  arXiv:2008.00975, 2020.
  * (419) Yu Yao and Ella Atkins.  The smart black box: A value-driven high-bandwidth automotive event data recorder.  IEEE Transactions on Intelligent Transportation Systems, 2020.
  * (420) Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto.  On early stopping in gradient descent learning.  Constructive Approximation, 26(2):289–315, 2007.
  * (421) Bangjie Yin, Luan Tran, Haoxiang Li, Xiaohui Shen, and Xiaoming Liu.  Towards interpretable face recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9348–9357, 2019.
  * (422) Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou.  Diverse few-shot text classification with multiple metrics.  arXiv:1805.07513, 2018.
  * (423) Mingxuan Yuan, Lei Chen, S Yu Philip, and Ting Yu.  Protecting sensitive labels in social network data anonymization.  IEEE Transactions on Knowledge and Data Engineering, 25(3):633–647, 2011.
  * (424) Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li.  Adversarial examples: Attacks and defenses for deep learning.  IEEE transactions on neural networks and learning systems, 30(9):2805–2824, 2019.
  * (425) Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord.  Explainability of vision-based autonomous driving systems: Review and challenges.  arXiv:2101.05307, 2021.
  * (426) Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al.  Accelerating the machine learning lifecycle with mlflow.  IEEE Data Eng. Bull., 41(4):39–45, 2018.
  * (427) Matthew D Zeiler and Rob Fergus.  Visualizing and understanding convolutional networks.  In European conference on computer vision, pages 818–833. Springer, 2014.
  * (428) Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork.  Learning fair representations.  In International conference on machine learning, pages 325–333. PMLR, 2013.
  * (429) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  arXiv:1611.03530, 2016.
  * (430) Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan.  Theoretically principled trade-off between robustness and accuracy.  In International Conference on Machine Learning, pages 7472–7482. PMLR, 2019.
  * (431) Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh.  Towards stable and efficient training of verifiably robust neural networks.  arXiv:1906.06316, 2019.
  * (432) Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel.  Efficient neural network robustness certification with general activation functions.  arXiv:1811.00866, 2018.
  * (433) Jie M Zhang, Mark Harman, Lei Ma, and Yang Liu.  Machine learning testing: Survey, landscapes and horizons.  IEEE Transactions on Software Engineering, 2020.
  * (434) Mengshi Zhang, Yuqun Zhang, Lingming Zhang, Cong Liu, and Sarfraz Khurshid.  Deeproad: Gan-based metamorphic testing and input validation framework for autonomous driving systems.  In 2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 132–142. IEEE, 2018.
  * (435) Quanshi Zhang, Ying Nian Wu, and Song-Chun Zhu.  Interpretable convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8827–8836, 2018.
  * (436) Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu.  Interpreting cnns via decision trees.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6261–6270, 2019.
  * (437) Sicong Zhang, Hui Yang, and Lisa Singh.  Anonymizing query logs by differential privacy.  In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 753–756, 2016.
  * (438) Xuyun Zhang, Laurence T Yang, Chang Liu, and Jinjun Chen.  A scalable two-phase top-down specialization approach for data anonymization using mapreduce on cloud.  IEEE Transactions on Parallel and Distributed Systems, 25(2):363–373, 2013.
  * (439) Chuan Zhao, Shengnan Zhao, Minghao Zhao, Zhenxiang Chen, Chong-Zhi Gao, Hongwei Li, and Yu-an Tan.  Secure multi-party computation: Theory, practice and applications.  Information Sciences, 476:357–372, 2019.
  * (440) Ziyuan Zhong.  A tutorial on fairness in machine learning.  https://towardsdatascience.com/a-tutorial-on-fairness-in-machine-learning-3ff8ba1040cb.
  * (441) Bin Zhou, Jian Pei, and WoShun Luk.  A brief survey on anonymization techniques for privacy preserving publishing of social network data.  ACM Sigkdd Explorations Newsletter, 10(2):12–22, 2008.
  * (442) Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba.  Learning deep features for discriminative localization.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2921–2929, 2016.
  * (443) Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy.  Domain generalization: A survey.  arXiv:2103.02503, 2021.
  * (444) Yichen Zhou and Giles Hooker.  Interpreting models via single tree approximation.  arXiv:1610.09036, 2016.
  * (445) Zhi-Hua Zhou, Yuan Jiang, and Shi-Fu Chen.  Extracting symbolic rules from trained neural network ensembles.  Ai Communications, 16(1):3–15, 2003.
  * (446) Ligeng Zhu, Zhijian Liu, and Song Han.  Deep leakage from gradients.  In The Conference on Neural Information Processing Systems. 2019\.
  * (447) Roberto V Zicari, John Brodersen, James Brusseau, Boris Düdder, Timo Eichhorn, Todor Ivanov, Georgios Kararigas, Pedro Kringen, Melissa McCullough, Florian Möslein, et al.  Z-inspection®: A process to assess trustworthy ai.  IEEE Transactions on Technology and Society, 2021.
  * (448) Thomas Zielke.  Is artificial intelligence ready for standardization?  In European Conference on Software Process Improvement, pages 259–274. Springer, 2020.
  * (449) Indrė Žliobaitė, Mykola Pechenizkiy, and Joao Gama.  An overview of concept drift applications.  Big data analysis: new algorithms for a new society, pages 91–114, 2016.