# IEEE 7010: A New Standard for Assessing the Well-being Implications of
Artificial Intelligence

1st Daniel Schiff  Georgia Institute of Technology,
Atlanta, USA
ORCID: 0000-0002-4376-7303  2nd Aladdin Ayesh  De Montfort University,
Leicester, UK
ORCID: 0000-0002-5883-6113  3rd Laura Musikanski  Happiness Alliance
Seattle, USA
ORCID: 0000-0002-7477-6683  4th John C. Havens  The IEEE Global Initiative on
Ethics of Autonomous and Intelligent Systems
Piscataway, USA
ORCID: 0000-0002-7226-422X

###### Abstract

Artificial intelligence (AI) enabled products and services are becoming a
staple of everyday life. While governments and businesses are eager to enjoy
the benefits of AI innovations, the mixed impact of these autonomous and
intelligent systems on human well-being has become a pressing issue. This
article introduces one of the first international standards focused on the
social and ethical implications of AI: The Institute of Electrical and
Electronics Engineering’s (IEEE) Standard (Std) 7010-2020 Recommended Practice
for Assessing the Impact of Autonomous and Intelligent Systems on Human Well-
being. Incorporating well-being factors throughout the lifecycle of AI is both
challenging and urgent and IEEE 7010 provides key guidance for those who
design, deploy, and procure these technologies. We begin by articulating the
benefits of an approach for AI centered around well-being and the measurement
of well-being data. Next, we provide an overview of IEEE 7010, including its
key principles and how the standard relates to approaches and perspectives in
place in the AI community. Finally, we indicate where future efforts are
needed.

###### Index Terms:

Well-being, artificial intelligence, autonomous systems, measurement, design

##  I Introduction

Current growth in AI-enabled applications and products is facilitating the
integration of technology into the fabric of modern life [1]. This innovation
is manifesting in a wide range of daily uses—from smart devices and industrial
robotics —and promises to continue as new use cases are developed in
healthcare [2], finance [3], transportation [4], and throughout commercial,
domestic, and the public spaces [5]. Autonomous and intelligent systems111IEEE
7010 uses the term “autonomous and intelligent systems” (A/IS or AIS) to
describe what is termed in this article “artificial intelligence” (AI). are
central to a sea change, as they are increasingly ubiquitous, impacting the
public in visible and invisible ways [6]. We propose that this sea change has
a profound potential to impact human physical and mental well-being in both
positive and negative ways.

Well-being is important to overall quality of life [7, 8, 9], and while some
research has been done on well-being in the digital era [10, 11, 12, 13, 14],
measuring the impact of AI on users as well as those indirectly affected is
still in its early stages [15, 16]. Issues such as algorithmic bias and lack
of transparency in facial recognition, natural language processing, and
criminal justice [17, 18] have captured public interest, along with the role
of AI in targeted advertising and political misinformation [19, 20]. Privacy
[21], inequality [22], social cohesion [23], and labor displacement [24] are
among the many social and ethical risks to human well-being associated with
AI. In response, governments, corporations, and NGOs have proposed new ethical
codes, principles, frameworks, industry standards, and policy strategies [25,
26, 27]. Assessments of AI’s impact on human and societal well-being are
relatively new, and there is not much known about how to incorporate a well-
being orientation and measurement into organizational or public settings.
Thus, we suggest that IEEE 7010 offers initial guidance and sets the stage for
increasingly robust and rigorous development, deployment, and evaluation of AI
in the future.

In this article, we review the first industry standard to our knowledge that
addresses the well-being implications of AI.222The authors of this paper were
members of the working group that formed IEEE 7010 standard as well as a
chapter of IEEE’s Ethically Aligned Design, First Edition focused on well-
being, published in March 2019 (The IEEE Global Initiative on Ethics of
Autonomous and Intelligent Systems 2019). This article does not represent an
official position of IEEE, but rather the opinions of its authors. The
Institute of Electrical and Electronics Engineering (IEEE) Standard 7010-2020
Recommended Practice for Assessing the Impact of Autonomous and Intelligent
Systems on Human Well-being (henceforth IEEE 7010) is part of the IEEE 70xx
series, a series of standards designed to address ethical dimensions of AI
[28, 29]. Projects in the IEEE 70xx series include IEEE P70xx series standards
on ethics in design (IEEE P7000), transparency (IEEE P7001), data privacy
(IEEE P7002), algorithmic bias (IEEE P7003), child and student data (IEEE
P7004), employer data governance (IEEE P7005), data agents (IEEE P7006),
ontologies for ethics (IEEE P7007), nudging (IEEE P7008), fail-safe design
(IEEE P7009), trustworthiness of news (IEEE P7011), machine readable privacy
(IEEE P7012), facial recognition (IEEE P7013), and empathy (IEEE P7014). IEEE
7010 is oriented around a holistic well-being perspective and offers practical
guidance for AI creators seeking to understand and measure direct, indirect,
intended, and unintended impacts to human and societal well-being.

In Section II we argue for the urgency of well-being in the context of
increasing uses of AI and describe the benefits of a well-being perspective.
We suggest that such an approach to AI development and measurement can help
organizations build awareness, provide actionable evidence and insights,
develop infrastructure, manage risks, and improve well-being of individuals
and groups in society.

Section III provides an overview of the core principles and processes that
constitute IEEE 7010, including its well-being impact assessment (WIA). In
this section, we describe how IEEE 7010 complements and differs from other
social and ethical orientations to assess the impacts of AI, such as human
rights; fairness, accountability, and transparency (FAT); law; and
sustainability. We also explain how IEEE 7010 could help governments in the
formation of regulations for AI.

Section IV offers considerations for how IEEE 7010 can be applied within an
organizational context. We propose that implementing a well-being approach in
alignment with traditional software and engineering lifecycle processes could
take a few different forms. We explore the application of well-being as a
principle, incorporated into system design as metrics, and via lifecycle
development stages. We present hypothetical case studies for autonomous
vehicles and healthcare robots to help demonstrate how IEEE 7010 might be
applied fruitfully to prominent AI use cases.

##  II Motivation

###  II-A The urgency of well-being in the autonomous/intelligent systems era

Well-being is a complex, multidimensional, and occasionally contested subject
studied for almost half a century by scholars in psychology, management,
economics, and policy [10, 30, 31]. Some scholars of well-being distinguish
between hedonic, eudaimonic, and social dimensions of well-being. Hedonic
well-being emphasizes immediate subjective experiences, such as positive and
negative affect and life satisfaction (National Research Council 2013), while
the eudaimonic dimension of well-being takes into account a broader definition
of flourishing [32]. Well-being also has both objective and subjective
components, where the former refers to traditional economic metrics like
health and income, and the latter to internal subjective experience [33, 34].
The Organization for Economic Co-operation (OECD) (2011; 2013) conceives of
well-being as incorporating numerous domains of social and economic life such
as education, economy, environment, health, government, community, culture,
work, psychological well-being, and human settlements and encompassing both
subjective and objective components. Well-being incorporates an orientation
towards human rights, one which enjoys wide support and is deeply grounded in
legal systems [35], as well environmental sustainability, now maturely
incorporated into industry standards and processes [36]. Over time, the study
of well-being has matured into an increasingly comprehensive, theoretically
and empirically rigorous, and applied field, that is gaining traction among
researchers as a perspective and in public policy [9, 31].

A focus on well-being can offer an alternative approach to a traditional focus
on economic growth and financially-driven efficiency. While the obligations of
corporations to the public have long been a topic of discussion [37, 38],
concerns about corporate social responsibility have grown in recent years due
to evidence of persistent economic inequality and ecological degradation [22,
39, 40, 41, 42]. Stout [43] as well as Tepper and Hern [44] emphasize recent
trends in corporate governance, arguing that corporations which prioritize
short-term financial gain too often fail to protect the public. These concerns
have given rise to calls for increased corporate social responsibility and
governmental action that protect human and societal well-being. Movements
focused on corporate social responsibility that incorporate environmental
sustainability and human well-being [45, 46, 47] are gaining purchase as a
variety of new conceptual and organizational strategies have surfaced. These
include the Beyond GDP [48], triple bottom line [49], and social enterprise
models [50], now enhanced via formal certification of socially responsible
organizations as B Corps [51] and through legal incorporation as public
benefit corporations [52].

National and intergovernmental bodies have also increasingly broadened their
focus beyond simple economic growth, such as through the United Nations’
Sustainable Development Goals (SDGs), a landmark set of international goals
and strategies to achieve poverty reduction, improved health, sustainability,
and other aspects of sustainable development that include dimensions of well-
being [53, 54, 55]. Meanwhile, the measurement of national well-being, such as
the gross national happiness (GNH) index [56], and the United Kingdom’s Office
of National Statistics Well-being measures 333See:
www.ons.gov.uk/peoplepopulationandcommunity/wellbeing, has now moved from
national to international prominence [31, 57]. Well-being has become a
mainstream topic in international governance bodies like the OECD and United
Nations [58] and the World Happiness Report now evaluates well-being in 156
countries [59]. Engineering and computing have also increased calls for
responsible research and innovation (RRI) [60] in design processes,
incorporating new tools, methodologies, and perspectives surrounding ethical
design [61].

This evolving understanding of the nature of human and societal well-being is
occurring at the same time that we are experiencing unprecedented
technological transformation. The increased influence of technology companies,
recent concerns over privacy and security [62], propaganda [20], algorithmic
bias , and labor displacement [63] have put the power of autonomous and
intelligent systems into the limelight, adding to prior concerns about
military uses of AI [64] and long-term safety [65]. Recent applications of AI
to manage exposure to Covid-19 have also raised privacy concerns. [66].

Scholars, policymakers, and the public now widely recognize that AI, as a
general purpose technology, has an enormous scope and scale of potential
impact across every sector of society. Schwab [67] has argued that AI is
central to an impending fourth industrial revolution, likely to transform
industry and public life over the next decades. The McKinsey Global Institute
[68] (2018) estimated AI’s financial impact at trillions of dollars in the
near future. We suggest that there is an urgent need to consider the sweeping
implications of AI. We propose that it is critical that the design,
deployment, evaluation, and regulation of AI prioritize the measurement and
safeguarding of well-being in society. We further suggest that a holistic
approach to well-being is useful for understanding and managing the impacts of
AI on human and societal well-being.

###  II-B A holistic approach to AI’s impacts on well-being

A holistic and impact-based approach to well-being, widely defined, differs
from approaches based on technological solutions alone. Tools and
methodologies centered on technical fixes can fail to take account of the full
social context and set of ethical issues surrounding AI [69]. They may
emphasize design processes presumed to promote ethics, while failing to
actually assess impacts. Moreover, technical fixes can also assign
responsibility for anticipating and reacting to well-being considerations too
narrowly, focusing mostly on the role of engineers and computer scientists
[70] while minimizing corporate responsibility.

Technical solutions and strategies do have an important role in ensuring
responsible design and use of AI, and the understanding of how these
methodologies work is growing [71]. However, much more needs to be understood
about the fields of AI and well-being in both industry and other sectors
(governmental, educational, etc.). Moreover, we doubt that technical fixes
alone will ever adequately meet challenges that are inherently both technical
and social in nature and so we propose that incorporating both social and
technical elements of AI into standards is a better approach to understanding,
measuring, and managing the direct, indirect, intended, and unintended impacts
of AI on human and societal well-being.

In short, well-being is a robust and promising orientation for considering the
human and social and ethical implications of AI. We propose that organizations
have much to gain by adopting this perspective. What is needed is a framework
to help organizations go about the process of adoption and adaptation of a
well-being framework for understanding and managing AI impacts on human and
societal well-being.

###  II-C Benefits of a well-being orientation

The IEEE 7010 standard is designed for AI creators, including organizations
that wish to design, deploy, procure, or evaluate these systems for their
impact on the well-being of humans. An orientation to AI that revolves around
a well-being perspective provides five benefits:

  * •

Building awareness: building awareness is a critical first step to
organizational change. Diverse organizations and roles can come to understand
the importance and relevance of well-being to their work.

  * •

Providing evidence: measuring well-being provides concrete data regarding
impacts in place of ignorance or assumptions that certain impacts are
inevitable or unlikely. Measurement allows organizations to evaluate successes
and failures, which can be used in goal setting, prioritization, and
management.

  * •

Developing infrastructure: incorporating well-being helps organizations build
the necessary infrastructure for thinking about, analyzing, and responding to
well-being data. Developing infrastructure includes processes for integrating
well-being considerations into data collection, decision-making, identifying
roles and responsibilities, and business use case development, all areas that
require organizational infrastructure.

  * •

Managing risks: well-being measurement and reporting helps organizations and
society both individually and collectively assess the risks of various AI
products and services. Managing risks safeguards employee, customer, user, and
public well-being, protects against reputational and brand threats, and aids
in anticipating and responding to regulatory and legal requirements.

  * •

Improving well-being: awareness, evidence, infrastructure, and risk management
jointly enable organizations to convert learning about well-being into
positive improvements to AI design. Improvements to well-being enhance brand
quality and trust, safeguard against harms to well-being, and enable the
realization of new opportunities to enhance well-being.

Below, we introduce IEEE 7010.

##  III IEEE 7010: A New Standard for Assessing the Well-being Implications of
AI

###  III-A Introduction to IEEE 7010

Importantly, IEEE standards are developed by working groups of volunteers,
often individuals from many sectors of society from around the world. Ther
working group members of IEEE 7010 had expertise in AI, well-being,
organizational science, policy, psychology, and many other areas. Standards
development often requires several years, including multiple rounds of
external review and voting before a standard is approved. IEEE 7010 is one of
the standards in the IEEE P70xx series of standards that emerged from a focus
on ethical dimensions of AI.

IEEE issues four classifications of standards: standards, recommended
practices, guidance, and best practices. IEEE 7010 is a recommended practice,
which identifies recommended procedures that adherents to the standards should
apply. A recommended practice is stronger than a standard classified as a
guide or trial-use document, which states how adherents can apply a certain
practice, but less stringent than a standard of the classification of
standard, which states how adherents must comply. The status of IEEE 7010 as a
recommended practice reflects the state of the field and the need to develop
an understanding of the impact of AI on human and societal well-being, such as
through an Well-being Impact Assessment (WIA), especially within
organizational settings.

As with all IEEE standards, IEEE 7010 describes its scope, purpose, and key
definitions. IEEE 7010 also contains annexes to support the core of the
standard, including examples of well-being indicators, examples of particular
AI use cases, examples of hypothetical organizations applying parts of the
standard, guidance on managerial adoption, a discussion of the value of IEEE
7010 to a wide range of stakeholders, and other useful resources and
references. The annexes and introduction provide additional guidance around
the core components of IEEE 7010, the WIA. The WIA focuses on five activities,
each broken down into several tasks, as depicted in 1. These activities and
tasks are grounded in stakeholder engagement and an iterative process.

Figure 1: Flowchart of IEEE 7010 WIA (IEEE P7010 - Adapted and reprinted with
permission from IEEE. Copyrights IEEE 2020. All rights reserved.)

#### Activity 1. Internal analysis and user and stakeholder engagement

The first activity of the WIA focuses on internal evaluation of an AI product
or service and information gathering from users and broader stakeholders
regarding its possible impacts. In terms of the internal analysis task,
organizations should answer five questions involving 1) the nature of the AI
system, 2) the needs it meets or problems it solves, 3) who the users
(intended and unintended) are, 4) who the broader stakeholders might be, and
5) the likelihood of possible positive and negative impacts, and how can they
be considered and mitigated. The internal analysis task allows organizations
to have a broad and thoughtful conversation about the purpose and impacts of
its AI. This goes beyond considering user experience, as it requires
considering the full range of possible impacts for users and stakeholders.
(Stakeholders are defined as any individuals or groups who are or might be
affected by the AI.)

The initial assessment of AI’s possible impact is not done only internally.
Organizations engage directly with users and stakeholders, such as through
interviews and focus groups. These tasks include asking users and stakeholders
about how they use or intend to use the AI, and what possible impacts,
benefits, and harms they experience or anticipate. The user and stakeholder
engagement tasks of IEEE 7010 asks that organizations go well beyond typical
user experience and product testing processes, and evaluate the full
implications of the AI across its use and lifecycle. For example,
organizations that develop AI products for K-12 education should expect to
engage with students, teachers, parents, administrators, and potentially other
stakeholders. An organization that sells AI products to government entities
for use in the criminal justice system should be prepared to engage with
government employees and individuals in the criminal justice system.

User and stakeholder engagement and internal analysis are part of a continual
reflective process, where multiple iterations may be performed as the
organization gets a better understanding of its AI and its well-being impacts.
Importantly, this first activity and three associated tasks are centered at
acquiring evidence surrounding the well-being impacts of a specific AI product
or service. As such, organizations should be deliberate about a wide range of
possible impacts. To do so, they look to the well-being indicators described
in IEEE 7010. The role and selection of well-being indicators is described
below.

#### Activity 2. Development and refinement of well-being indicators dashboard

IEEE 7010 identifies indicators of human and societal well-being across
multiple dimensions of well-being, divided into twelve domains. These domains
are affect, community, culture, education, economy, environment, health, human
settlements, government, psychological/mental well-being, and work. IEEE 7010
provides sample indicators associated with each domain as well as additional
resources through which to identify additional indicators. The well-being
indicators within IEEE 7010 are sourced from mainstream evaluative tools for
well-being, and are validated by rigorous research. This is a hallmark feature
of IEEE 7010.

The internal analysis and user and stakeholder engagement tasks center around
providing evidence for identifying possible well-being indicators that measure
the impact of the AI. Users and stakeholders are asked about impacts to their
environment, health, work, community, social support and other dimensions of
well-being. The indicators and domains guide the creation of a well-being
indicators dashboard. The dashboard can be represented visually as depicted in
2 and integrated into engineering processes.

Figure 2: Sample well-being indicators dashboard for autonomous vehicles

During the dashboard creation process, organizations should articulate: 1) the
source for each indicator, 2) why the indicator was selected, 3) the
associated domain from the list of twelve, and 4) if appropriate, any
adaptations made to indicators. IEEE 7010 recognizes that existing indicators
of well-being may not adequately reflect the impact of a given organization’s
AI product or service on human or societal well-being. For example, some
indicators focus on measurement of an aspect of well-being at the national
level, whereas an AI may impact a smaller number of individuals. IEEE 7010
allows for customization and flexibility when appropriate, while aiming to
stay as close as possible to scientifically valid well-being indicators.

#### Activity 3. Data planning and collection

As IEEE 7010 is at heart a WIA, data collection is important. The third
activity follows the development or refinement of the well-being indicators
dashboard. Based on the identified indicators and domains, an organization
articulates a data collection plan. This includes collection of both baseline
data and data over time, allowing changes in well-being indicators to be
assessed over time. The data collection plan task involves organizations
describing what data will be collected and how. This includes identifying
sources (e.g., which users or stakeholders), methods (e.g., surveys, product
metrics), and the frequency and timelines associated with data collection.

After establishing a data collection plan, the next task is to collect the
data. This includes collecting baseline data for users and stakeholders as
well as populations that are statistically similar to users and stakeholders.
Collecting data on the latter group helps to set a baseline and is useful for
comparative analysis. Next, data are collected over time, after users and
stakeholders engage with a given AI product or service. Collecting data from
multiple populations and over time allows for assessment of actual impacts.
IEEE 7010 recognizes that collecting data and associating changes over time
with a particular AI product or service is conceptually and empirically
challenging. However, the challenge of identifying specific causal impacts
should not lead to the abandonment well-being impact assessment, but instead
to refine and improve it.

#### Activity 4. Data analysis and improvement to AI

The fourth activity is analysis and use of collected data. Organizations can
identify trends over time and changes with respect to baseline data or
compared to other populations. Data analysis can also illuminate unexpected
uses, behaviors, and impacts, and to document how a particular AI product or
service was used in real-world settings. Analysis helps determine if an AI
does have negative impacts, or if efforts to mitigate negative impacts or
increase positive impacts are successful. Importantly, analysis then feeds
into improvements to AI design, development, assessment, monitoring, and
management. IEEE 7010 does not specify all of the actions that an organization
should take, as how an organization learns from the WIA and data depends on
the specific product or service, its impacts, and organizational
circumstances. However, there are some actions IEEE 7010 recommends towards
refining well-being impact assessment, described in activity five.

#### Activity 5. Iteration

IEEE 7010 presents activities that should be performed in an iterative
fashion. There are deliberate feedback cycles built in. Assessment of the
well-being impacts of an AI product or service should not be a one-time effort
and should be ongoing. Organizations can use lessons learned from the user and
stakeholder engagement process and data analysis to improve the implementation
of the well-being assessment process. They should also refine the well-being
indicators dashboard as their understanding grows. Improving the assessment
process also means strengthening data collection and analysis plan. Over time,
organizations should develop capacity towards streamlining WIA, learning how
to conceive of impacts more robustly, capture and analyze data more
accurately, and efficiently implement changes into engineering, software, and
business processes. Finally, organizations can report both internally and
externally to users and stakeholders, in order to help assess and communicate
progress and opportunities for improvement.

#### Note on Data Collection and Privacy

Data collection is at the heart of WIA. Collected data may be in the form of
user surveys, publicly available government sources, or usage data, for
example, mouse clicks [72], health/physiological data [73], and facial
expressions [74]. However, these and other forms of data are all potentially
highly sensitive based on how data are governed, secured, and used. In light
of the rising prospect of cognitive profiling through affect state estimation
or personalized analytics [75], privacy becomes of urgent importance.

In response, several policy-making institutions have produced guidelines, code
of conducts, or model laws to address the challenge to individual privacy and
data usage rights arising from the rapid development of data-intensive AI and
other applications [76]. Among the most extensive laws is the European Union’s
(EU) General Data Protection Regulation (GDPR)444EU: See: https://eugdpr.org,
which applies to all EU member countries and is now referenced by other
nations [77]. The EU’s GDPR is often considered a baseline for data protection
laws [78]. There is a potential tension between data collection for well-being
improvement and user privacy, though there are privacy-preserving approaches
[66]. IEEE 7010 is not a standard for data protection or other considerations
such as algorithmic bias or data governance but includes mention of protective
measures such as the EU’s GDPR as well as field- or sector-specific
guidelines.

##  IV Relationship of IEEE 7010 to Other Processes and Approaches

IEEE 7010 complements, extends, and differs from other orientations, tools,
and methodologies. Below we describe its relationship to some prominent
approaches relevant to assessing the social and ethical impacts of AI.

####  IV-1 The IEEE 70xx standards series

The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems is
among the earliest and most robust efforts, resulting in three drafts of
IEEE’s Ethically Aligned Design issued in 2016, 2017, and 2019) [79] and a
proposed a series of 14 standards. The standard series addresses AI ethics
issues including privacy, bias, transparency, trustworthiness of news,
empathy, fail-safe design, ontologies, and ethical design [28]. Because at the
time of publishing of IEEE 7010, other standards in the IEEE 70xx series had
not been published, it does not recommend adherence to other standards.).
However, it can be anticipated that other IEEE 70xx standards will be relevant
to the scope and purpose of Std 7010, and future revisions of IEEE 7010 may
incorporate these standards.

####  IV-2 Sustainability

While IEEE 7010 includes “Assessing the Impact of Autonomous and Intelligent
Systems on Human Well-being” in the title, the standard explicitly includes
sustainability in the definition of well-being, especially with respect to the
environment. IEEE 7010’s definition of well-being is “the continuous and
sustainable physical, mental, and social flourishing of individuals,
communities and populations where their economic needs are cared for within a
thriving ecological environment” (IEEE 7010 2020). The environment domain in
IEEE 7010 considers many possible impacts of AI, such as on pollution, waste,
and ecological diversity. Stakeholders concerned with environmental and
ecological sustainability and well-being should view IEEE 7010 as a supportive
tool.

####  IV-3 Human rights

Human rights is an important aspect of human and social well-being. Human
rights have a long tradition in legal philosophy and international governance.
It has been applied to AI recently by scholars [80, 35] and IEEE’s Ethically
Aligned Design includes human rights as indispensable (The IEEE Global
Initiative on Ethics of Autonomous and Intelligent Systems 2019). IEEE 7010 is
consistent with and embraces a human rights perspective. It incorporates
assessment of human rights indicators within the domain of government.

####  IV-4 Fairness, accountability, and transparency

Fairness, accountability, transparency, and ethics are referred to as FAT,
FATE, FEAT, or similar acronyms. These approaches typically emphasize ethical
issues in algorithm design that are seen as technically tractable, especially
algorithmic bias and transparency [81, 82]. This approach is popular amongst
computer science and AI ethics researchers, such as the Academy of Computing
Machinery’s FAccT conference and community. Recently, these communities are
opening up to broader perspectives, such as social science, law, and policy
[69]. Scholars and practitioners interested in this perspective may find that
IEEE 7010 can serve as an important extension to their toolset, one that
allows them to conceive of AI’s impacts on human and societal well-being more
broadly. For example, practitioners of FAT can apply IEEE 7010’s WIA and then
apply FAT tools and methodologies based on impacts through the WIA and data.

####  IV-5 Algorithmic impact assessments

Algorithmic impact assessments are sometimes focused on the downstream effects
of AI as well as ethical issues during the design stage, as they often heavily
feature aspects of FAT. These assessments are promoted by scholars and
nongovernmental organizations (NGOs) [83, 5], and are being considered as part
of regulatory frameworks. The European Commission proposed elements of
algorithmic assessment within its most recent guidance for the EU [84, 85],
and the United States has proposed the Algorithmic Accountability Act [86].
IEEE 7010 is highly concordant with this perspective, as it is centered on
impact assessment. IEEE 7010’s WIA could serve as a tool for policymakers to
look to in the near future.

####  IV-6 Law and regulation

IEEE 7010 is a voluntary industry standard, to be considered as part of
private sector self-governance or collective governance. Other standards, such
as those created by national standards bodies, may be legally mandatory.
However, voluntary standards can play a role in legal frameworks and have
political legitimacy [87, 88]. For example, adherence to an industry standard
can demonstrate good faith efforts beyond minimal levels of compliance and can
reduce a company’s liability. Moreover, governments and courts of law
sometimes give consideration to companies adhering to industry standards when
contracting or grant making. For example, the 2016 U.S. National Research and
Development (R&D) Plan states that “Industry and academia are the primary
sources for emerging AI technologies” and considers “Updating acquisition
processes across agencies to include specific requirements for AI standards in
requests for proposals” [89]. IEEE 7010 may provide a means for companies to
demonstrate readiness to adhere to future laws and regulations.

####  IV-7 Software and engineering processes

The incorporation of well-being measurement into traditional and newer
software engineering processes, such as agile software development and
engineering lifecycle standards [90] is critical to organizations that develop
AI, and remains a challenge [82]. IEEE 7010 includes a description of its
application to the Plan-Do-Check-Act Cycle, Lifecycle Analysis, and other
processes in its Annexes. IEEE 7010 features checklists throughout each of its
activities and tasks within the well-being assessment, consistent with
checklist-based processes are common in engineering design [91].

##  V Recommendations

IEEE 7010 is the first international standard for assessing and managing the
impact of well-being on humans and society from AI. As such, there are not yet
examples or case studies of its use. We recommend the following eight areas
for development:

  1. 1.

Use of IEEE 7010. AI creators, managers and investors use IEEE 7010. We
recommend the use of IEEE 7010 to assess, manage, mitigate, and improve the
well-being impacts on human and societal well-being, extending from individual
users to the public. This recommendation covers AI projects in their inception
to those fully implemented in the marketplace. This recommendation is equally
important for independent or small projects as for large-scale projects
undertaken by international or transnational organizations.

  2. 2.

Development of organizational structures and processes. AI creators, managers,
and investors develop processes, tools, and resources for implementing and
maintaining IEEE 7010 into their organizational processes and structures. This
includes developing staffing and workflow structures, engineering and
computing practices, hiring and training, internal resources and toolkits,
external communication, and other necessary structures.

  3. 3.

Formation of an Industry Connections group. IEEE and industry stakeholders
invested in well-being and AI form an Industry Connections (IC)
group555https://standards.ieee.org/industry-connections/index.html. An IC
group creates an ecosystem for parties to perform research, create tools and
resources, publish information, and promote its work via conferences and other
events. We recommend that such an IC group be created to help translate the
IEEE 7010 standard into practice.

  4. 4.

Shared language of well-being. Researchers and practitioners in the AI or
well-being field define well-being broadly to encompass the multi-dimensional
aspects of well-being that includes subjective and objective measurements
included in IEEE 7010 across the domains of: “(1) Affect, (2) Community, (3)
Culture, (4) Education, (5) Economy, (6) Environment, (7) Human Settlements,
(8) Health, (9) Government, (10) Psychological Well-Being/Mental well-being,
(11) Satisfaction with life and (12) Work” [27]. To encourage consensus-
building, we further recommend that researchers and practitioners that focus
on a single dimension or subset of dimensions of well-being identify their
scope of study within this multidimensional definition of well-being.

  5. 5.

Expanded understanding of well-being. AI creators, managers and investors
develop an understanding of the definition of human and societal well-being
conceptually and in terms of measurement and data. They should develop a
deeper understanding of the impacts on human and societal well-being from AI
in general and related to their specific AI work. We further recommend that
leading individuals and organizations in the AI field bring conversation about
the impacts on human and societal well-being from AI to the forefront through
publications, conferences, competitions, and other means.

  6. 6.

Increased research on AI and well-being. Researchers and practitioners develop
the field of knowledge about the impacts of AI on human and societal well-
being gathering and analyzing well-being data from users and stakeholders of
specific AIs, as well as the general population. We further recommend that AI
creators, managers, and investors cooperate with researchers in the gathering
and analysis of well-being data and that researchers and practitioners
cooperate with each other through sharing of data. We also recommend that
researchers, practitioners, and AI creators jointly contribute to the
formation of best practices, such as to evolve IEEE 7010 and similar efforts.

  7. 7.

Educational efforts focused on well-being. Universities develop courses,
centers, research programs, and other means of teaching and developing skills
for future AI creators, managers, and investors. We recommend that professors
and instructors currently teaching AI integrate IEEE 7010 into their
curriculum and that university and other educational institutions develop
programs and courses which integrate IEEE 7010, with the aim of developing
future AI creators’, managers’, and investors’ capacity to integrate well-
being metrics and goals into their future work.

  8. 8.

Regulation of AI’s impacts on well-being. Governments formulate regulations
and standards to contribute to securing and protecting the well-being of
humans and society in light of the impacts of AI. We recommend that
governments collaborate in the formulation of such standards or regulations as
well as develop these independently. We also recommend that international non-
governmental organizations facilitate this effort through promulgation of
model codes and regulations, as well as issuance of proclamations and
resolutions reinforcing their importance, and the coordination of efforts at
high-level meetings, summits, and conferences aimed. We recommend that
governments adopt a process similar to or based on IEEE 7010 for any
development of or investment in AI. Finally, we recommend that governments
consider adoption of IEEE 7010 as part of grant making procedures and to
demonstrate regulatory compliance.

##  VI Conclusion

This article introduced IEEE 7010, a new international standard aimed at
addressing the well-being implications of AI. After describing the concept of
well-being, we identify the potential benefits of a well-being orientation in
light of the social and ethical issues of the modern world. Next, we provide
an overview of IEEE 7010, outlining the core activities and tasks that define
its assessment of well-being. We then situated IEEE 7010 in relation to other
processes and approaches currently used to think about social, ethical, and
well-being implications related to AI. Finally, we offered recommendations for
a variety of stakeholders.

## References

  * [1] S. Makridakis, “The forthcoming Artificial Intelligence (AI) revolution: Its impact on society and firms,” _Futures_ , vol. 90, pp. 46–60, Jun. 2017. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0016328717300046
  * [2] K.-H. Yu, A. L. Beam, and I. S. Kohane, “Artificial intelligence in healthcare,” _Nature Biomedical Engineering_ , vol. 2, no. 10, pp. 719–731, 2018, number: 10 Publisher: Nature Publishing Group.
  * [3] A. Bahrammirzaee, “A comparative survey of artificial intelligence applications in finance: artificial neural networks, expert system and hybrid intelligent systems,” _Neural Computing and Applications_ , vol. 19, no. 8, pp. 1165–1195, 2010.
  * [4] D. J. Fagnant and K. Kockelman, “Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations,” _Transportation Research Part A: Policy and Practice_ , vol. 77, pp. 167–181, 2015.
  * [5] D. Reisman, J. Schultz, K. Crawford, and M. Whittaker, “Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability,” AI Now, New York, NY, Tech. Rep., 2018. [Online]. Available: https://ainowinstitute.org/aiareport2018.pdf
  * [6] J. C. Augusto, “Ambient Intelligence: The Confluence of Ubiquitous/Pervasive Computing and Artificial Intelligence,” in _Intelligent Computing Everywhere_ , A. J. Schuster, Ed. London: Springer London, 2007, pp. 213–234.
  * [7] E. Diener, “Subjective well-being,” _Psychological Bulletin_ , vol. 95, no. 3, pp. 542–575, 1984.
  * [8] F. A. Huppert and T. T. C. So, “Flourishing Across Europe: Application of a New Conceptual Framework for Defining Well-Being,” _Social Indicators Research_ , vol. 110, no. 3, pp. 837–1246, 2013, publisher: Springer.
  * [9] G. O’Donnell, A. Deaton, M. Durand, D. Halpern, and R. Layard, “Wellbeing and Policy,” Legatum Institute, Tech. Rep., 2014.
  * [10] E. Diener, E. M. Suh, R. E. Lucas, and H. L. Smith, “Subjective well-being: Three decades of progress,” _Psychological Bulletin_ , vol. 125, no. 2, pp. 276–302, 1999, publisher: American Psychological Association Inc.
  * [11] C. Huang, “Internet Use and Psychological Well-being: A Meta-Analysis,” _Cyberpsychology, Behavior, and Social Networking_ , vol. 13, no. 3, pp. 241–249, 2010, publisher: Mary Ann Liebert, Inc., publishers.
  * [12] D. Liu, R. F. Baumeister, C.-c. Yang, and B. Hu, “Digital Communication Media Use and Psychological Well-Being: A Meta-Analysis,” _Journal of Computer-Mediated Communication_ , vol. 24, no. 5, pp. 259–273, 2019, publisher: Oxford Academic.
  * [13] J. M. Twenge, G. N. Martin, and W. K. Campbell, “Decreases in psychological well-being among American adolescents after 2012 and links to screen time during the rise of smartphone technology.” _Emotion_ , vol. 18, no. 6, pp. 765–780, 2018.
  * [14] R. A. Calvo and D. Peters, _Positive Computing: Technology for Wellbeing and Human Potential_. MIT Press, 2014, google-Books-ID: uI6ZBQAAQBAJ.
  * [15] OECD, _Artificial Intelligence in Society_. Paris: OECD Publishing, 2019\.
  * [16] L. Musikanski, B. Rakova, J. Bradbury, R. Phillips, and M. Manson, “Artificial Intelligence and Community Well-being: A Proposal for an Emerging Area of Research,” _International Journal of Community Well-Being_ , vol. 3, no. 1, pp. 39–55, Mar. 2020.
  * [17] A. Chouldechova, D. Benavides-Prado, O. Fialko, and R. Vaithianathan, “A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions,” in _Conference on Fairness, Accountability and Transparency_ , 2018, pp. 134–148. [Online]. Available: http://proceedings.mlr.press/v81/chouldechova18a.html
  * [18] J. Buolamwini and T. Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification,” in _Conference on Fairness, Accountability and Transparency_ , 2018, pp. 77–91.
  * [19] A. T. Little, “Fake News, Propaganda, and Lies Can Be Pervasive Even If They Aren’t Persuasive,” _Critique_ , vol. 11, no. 1, pp. 21–34, 2018.
  * [20] R. Chesney and D. Citron, “Deepfakes and the New Disinformation War: The Coming Age of Post-Truth Geopolitics,” _Foreign Affairs_ , vol. 98, p. 147, 2019.
  * [21] O. Tene and J. Polonetsky, “Big data for all: Privacy and user control in the age of analytics,” _Nw. J. Tech. & Intell. Prop._, vol. 11, p. xxvii, 2012.
  * [22] T. Piketty and A. Goldhammer, _Capital in the twenty-first century_. Cambridge Massachusetts: The Belknap Press of Harvard University Press, 2014.
  * [23] D. M. West, _The Future of Work: Robots, AI, and Automation_. Brookings Institution Press, 2018.
  * [24] D. H. Autor, “Why Are There Still So Many Jobs? The History and Future of Workplace Automation,” _Journal of Economic Perspectives_ , vol. 29, no. 3, pp. 3–30, 2015.
  * [25] A. Jobin, M. Ienca, and E. Vayena, “The global landscape of AI ethics guidelines,” _Nature Machine Intelligence_ , vol. 1, no. 9, pp. 389–399, Sep. 2019. [Online]. Available: https://www.nature.com/articles/s42256-019-0088-2
  * [26] D. Schiff, J. Biddle, J. Borenstein, and K. Laas, “What’s next for ai ethics, policy, and governance? a global overview,” in _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , ser. AIES’20. New York, NY, USA: Association for Computing Machinery, 2020, pp. 153–158.
  * [27] L. Musikanski, J. Havens, and G. Gunsch, “IEEE P7010 Well-being Metrics Standard for Autonomous and Intelligent Systems™,” IEEE, New York, NY, Tech. Rep., 2018.
  * [28] R. Chatila and J. C. Havens, “The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems,” in _Robotics and Well-Being_ , ser. Intelligent Systems, Control and Automation: Science and Engineering, M. I. Aldinhas Ferreira, J. Silva Sequeira, G. Singh Virk, M. O. Tokhi, and E. E. Kadar, Eds. Cham: Springer International Publishing, 2019, pp. 11–16.
  * [29] A. Koene, A. L. Smith, T. Egawa, S. Mandalh, and Y. Hatada, “IEEE P70xx, Establishing Standards for Ethical Technology,” in _Proceedings of the KDD, ExCel_ , 2019, p. 2.
  * [30] R. A. Easterlin, “Income and Happiness: Towards a Unified Theory,” _The Economic Journal_ , vol. 111, p. 20, 2001.
  * [31] J. E. Stiglitz, J.-P. Fitoussi, and M. Durand, _Measuring what counts: the global movement for well-being_. New York: The New Press, 2019.
  * [32] C. D. Fisher, “Conceptualizing and Measuring Wellbeing at Work,” in _Wellbeing_. American Cancer Society, 2014, pp. 1–25.
  * [33] E. Alatartseva and G. Barysheva, “Well-being: Subjective and Objective Aspects,” _Procedia - Social and Behavioral Sciences_ , vol. 166, pp. 36–42, 2015.
  * [34] B. Dale, “Subjective and objective social indicators in studies of regional social well-being,” _Regional Studies_ , vol. 14, no. 6, pp. 503–515, 1980, publisher: Routledge.
  * [35] M. Latonero, “Governing Artificial Intelligence: Upholding Human Rights & Dignity,” Data&Society, New York, NY, Tech. Rep., 2018.
  * [36] P. Bansal and T. Hunter, “Strategic Explanations for the Early Adoption of ISO 14001,” _Journal of Business Ethics_ , vol. 46, no. 3, pp. 289–299, 2003.
  * [37] J. L. Campbell, “Why Would Corporations Behave in Socially Responsible Ways? An Institutional Theory of Corporate Social Responsibility,” _The Academy of Management Review_ , vol. 32, no. 3, pp. 946–967, 2007.
  * [38] D. S. Ruder, “Public Obligations of Private Corporations,” _University of Pennsylvania Law Review_ , vol. 114, no. 2, pp. 209–229, 1965, publisher: The University of Pennsylvania Law Review.
  * [39] R. Wilkinson, _The Inner Level: How more equal societies reduce stress, restore sanity and improve everyone’s well-being_. S.l.: Penguin Books, 2020, oCLC: 1140462934.
  * [40] R. Wilkinson and K. Pickett, _Spirit Level_. New York: Bloomsbury Press, 2009\.
  * [41] J. E. Stiglitz, A. Sen, and J.-P. Fitoussi, “Report by the Commission on the Measurement of Economic Performance and Social Progress,” ommission on the Measurement of Economic Performance and Social Progress, Paris, Tech. Rep., 2009.
  * [42] P. Hawken, A. B. Lovins, and L. H. Lovins, _Natural Capitalism: The Next Industrial Revolution_. Routledge, 2013.
  * [43] L. A. Stout, _The shareholder value myth: how putting shareholders first harms investors, corporations, and the public_. San Francisco: Berrett-Koehler, 2012, oCLC: ocn760975992.
  * [44] J. Tepper and D. Hearn, _The Myth of Capitalism: Monopolies and the Death of Competition_. Hoboken, New Jersey: Wiley, 2018.
  * [45] M. Miles and J. Covin, “Environmental marketing: A source of reputational, competitive, and financial advantage,” _JOURNAL OF BUSINESS ETHICS_ , vol. 23, no. 3, pp. 299–311, 2000.
  * [46] J. S. Armstrong and K. C. Green, “Effects of corporate social responsibility and irresponsibility policies,” _Journal of Business Research_ , vol. 66, no. 10, pp. 1922–1927, 2013.
  * [47] H. S. Brown, M. d. Jong, and T. Lessidrenska, “The rise of the Global Reporting Initiative: a case of institutional entrepreneurship,” _Environmental Politics_ , vol. 18, no. 2, pp. 182–200, 2009.
  * [48] I. Kubiszewski, R. Costanza, C. Franco, P. Lawn, J. Talberth, T. Jackson, and C. Aylmer, “Beyond GDP: Measuring and achieving global genuine progress,” _Ecological Economics_ , vol. 93, pp. 57–68, 2013.
  * [49] J. Elkington, “Partnerships from cannibals with forks: The triple bottom line of 21st-century business,” _Environmental Quality Management_ , vol. 8, no. 1, pp. 37–51, 1998.
  * [50] R. Dart, “The legitimacy of social enterprise,” _Nonprofit Management and Leadership_ , vol. 14, no. 4, pp. 411–424, 2004, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/nml.43.
  * [51] W. Stubbs, “Sustainable Entrepreneurship and B Corps,” _Business Strategy and the Environment_ , vol. 26, no. 3, pp. 331–344, 2017, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/bse.1920.
  * [52] J. S. Hiller, “The Benefit Corporation and Corporate Social Responsibility,” _Journal of Business Ethics_ , vol. 118, no. 2, pp. 287–301, 2013.
  * [53] United Nations, “Transforming Our World: The 2030 Agenda for Sustainable Development,” United Nations General Assembly, Tech. Rep. A/RES/70/1, 2015.
  * [54] L. Iriarte and L. Musikanski, “Bridging the Gap between the Sustainable Development Goals and Happiness Metrics,” _International Journal of Community Well-Being_ , vol. 1, no. 2, pp. 115–135, Feb. 2019.
  * [55] “World Happiness Report: Chapter 6, Sustainable Development and Human Well-Being,” New york, Tech. Rep.
  * [56] K. Ura, S. Alkire, and T. Zangmo, “Bhutan: Gross National Happiness and the GNH index,” The Centre for Bhutan Studies, Tech. Rep., 2012, accepted: 2016-06-23T14:36:41Z.
  * [57] L. Musikanski, “Happiness In Public Policy,” _Journal of Social Change_ , p. 31, 2014.
  * [58] E. Diener, R. Lucas, U. Schimmack, and J. Helliwell, Eds., _Well-being for public policy_ , ser. Series in positive psychology. Oxford ; New York: Oxford University Press, 2009, oCLC: ocn244566759.
  * [59] J. F. Helliwell, R. Layard, J. Sachs, and J.-E. De Neve, “World Happiness Report 2020,” Sustainable Development Solutions Network, New York, Tech. Rep., 2020. [Online]. Available: https://worldhappiness.report/ed/2020/
  * [60] R. v. Schomberg, “A Vision of Responsible Research and Innovation,” in _Responsible Innovation_. Wiley-Blackwell, 2013, pp. 51–74.
  * [61] V. Dignum, “Ethics in artificial intelligence: introduction to the special issue,” _Ethics and Information Technology_ , vol. 20, no. 1, pp. 1–3, 2018.
  * [62] R. Chesney and D. K. Citron, “Deep Fakes: A Looming Challenge for Privacy, Democracy, and National Security,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3213954, 2018.
  * [63] C. B. Frey and M. A. Osborne, “The future of employment: How susceptible are jobs to computerisation?” _Technological Forecasting and Social Change_ , vol. 114, pp. 254–280, 2013.
  * [64] G. E. Marchant, B. Allenby, R. C. Arkin, J. Borenstein, L. M. Gaudet, O. Kittrie, P. Lin, G. R. Lucas, R. O’Meara, and J. Silberman, “International Governance of Autonomous Military Robots,” in _Handbook of Unmanned Aerial Vehicles_ , K. P. Valavanis and G. J. Vachtsevanos, Eds. Dordrecht: Springer Netherlands, 2015, pp. 2879–2910.
  * [65] V. C. Müller and N. Bostrom, “Future Progress in Artificial Intelligence: A Survey of Expert Opinion,” in _Fundamental Issues of Artificial Intelligence_ , ser. Synthese Library. Springer, Cham, 2016, pp. 555–572.
  * [66] M. Nanni, G. Andrienko, C. Boldrini, F. Bonchi, C. Cattuto, F. Chiaromonte, G. Comandé, M. Conti, M. Coté, F. Dignum, V. Dignum, J. Domingo-Ferrer, F. Giannotti, R. Guidotti, D. Helbing, J. Kertesz, S. Lehmann, B. Lepri, P. Lukowicz, A. Monreale, K. Morik, N. Oliver, A. Passarella, A. Passerini, D. Pedreschi, A. Pentland, F. Pratesi, S. Rinzivillo, S. Ruggieri, A. Siebes, R. Trasarti, J. v. d. Hoven, and A. Vespignani, “Give more data, awareness and control to individual citizens, and they will help COVID-19 containment,” _arXiv:2004.05222 [cs]_ , 2020, arXiv: 2004.05222 version: 1\. [Online]. Available: http://arxiv.org/abs/2004.05222
  * [67] K. Schwab, _The fourth industrial revolution_ , first edition ed. New York: Crown Business, 2016.
  * [68] J. Bughin, J. Seong, J. Manyika, M. Chui, and R. Joshi, “Modeling the global economic impact of AI,” McKinsey Global Institute, Tech. Rep., 2018. [Online]. Available: https://www.mckinsey.com/featured-insights/artificial-intelligence/notes-from-the-frontier-modeling-the-impact-of-ai-on-the-world-economy
  * [69] A. D. Selbst, D. Boyd, S. Friedler, S. Venkatasubramanian, and J. Vertesi, “Fairness and Abstraction in Sociotechnical Systems,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3265913, 2018.
  * [70] D. Greene, A. L. Hoffmann, and L. Stark, “Better, Nicer, Clearer, Fairer: A Critical Assessment of the Movement for Ethical Artificial Intelligence and Machine Learning,” _Critical and Ethical Studies of Digital and Social Media_ , p. 10, 2019.
  * [71] J. Morley, L. Floridi, L. Kinsey, and A. Elhalal, “From What to How: An Initial Review of Publicly Available AI Ethics Tools, Methods and Research to Translate Principles into Practices,” _Science and Engineering Ethics_ , Dec. 2019. [Online]. Available: https://doi.org/10.1007/s11948-019-00165-5
  * [72] K. Renaud and P. Gray, “Making sense of low-level usage data to understand user activities,” in _Proceedings of the 2004 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries_ , ser. SAICSIT ’04. Stellenbosch, Western Cape, South Africa: South African Institute for Computer Scientists and Information Technologists, 2004, pp. 115–124.
  * [73] X. Li, J. Dunn, D. Salins, G. Zhou, W. Zhou, S. M. S.-F. Rose, D. Perelman, E. Colbert, R. Runge, S. Rego, R. Sonecha, S. Datta, T. McLaughlin, and M. P. Snyder, “Digital Health: Tracking Physiomes and Activity Using Wearable Biosensors Reveals Useful Health-Related Information,” _PLOS Biology_ , vol. 15, no. 1, p. e2001402, 2017, publisher: Public Library of Science.
  * [74] M. Saneiro, O. C. Santos, S. Salmeron-Majadas, and J. G. Boticario, “Towards Emotion Detection in Educational Scenarios from Facial Expressions and Body Movements through Multimodal Approaches,” 2014, iSSN: 2356-6140 Library Catalog: www.hindawi.com Pages: e484873 Publisher: Hindawi Volume: 2014.
  * [75] A. Ayesh, M. Arevalillo-Herráez, and F. J. Ferri, “Towards Psychologically based Personalised Modelling of Emotions Using Associative Classifiers:,” _International Journal of Cognitive Informatics and Natural Intelligence_ , vol. 10, no. 2, pp. 52–64, Apr. 2016. [Online]. Available: http://services.igi-global.com/resolvedoi/resolve.aspx?doi=10.4018/IJCINI.2016040103
  * [76] C. J. Bennett and C. D. Raab, _The Governance of Privacy: Policy Instruments in Global Perspective_. The MIT Press, 2017, oCLC: 1012163985.
  * [77] A. Chander, M. E. Kaminski, and W. McGeveran, “Catalyzing Privacy Law,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3433922, 2019.
  * [78] P. Voigt and A. von dem Bussche, _The EU General Data Protection Regulation (GDPR)_. Cham: Springer International Publishing, 2017.
  * [79] The IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems, “Ethically Aligned Design: A Vision for Prioritizing Human Well-being with Autonomous and Intelligent Systems, First Edition,” IEEE, Piscataway, New Jersey, Tech. Rep., 2019.
  * [80] C. Cath, M. Latonero, V. Marda, and R. Pakzad, “Leap of FATE: human rights as a complementary framework for AI policy and practice,” in _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , 2020, pp. 702–702.
  * [81] B. Mittelstadt, C. Russell, and S. Wachter, “Explaining explanations in AI,” in _Proceedings of the conference on fairness, accountability, and transparency_ , 2019, pp. 279–288.
  * [82] B. Rakova, R. Chowdhury, and J. Yang, “Assessing the intersection of organizational structure and FAT* efforts within industry: implications tutorial,” in _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , 2020, pp. 697–697.
  * [83] R. A. Calvo, D. Peters, and S. Cave, “Advancing impact assessment for intelligent systems,” _Nature Machine Intelligence_ , vol. 2, no. 22, p. 89–91, Feb 2020.
  * [84] European Commission, “On Artificial Intelligence - A European approach to excellence and trust,” European Commission, Brussels, Belgium, Tech. Rep. COM(2020) 65, 2020.
  * [85] M. E. Kaminski and G. Malgieri, “Algorithmic Impact Assessments under the GDPR: Producing Multi-layered Explanations,” Social Science Research Network, Rochester, NY, SSRN Scholarly Paper ID 3456224, 2019.
  * [86] K. Quezada Tavárez, “The Algorithmic Accountability Act: is the US about to apply EU standards to algorithmic governance?” 2019, publisher: Centre for IT & IP Law.
  * [87] S. Bernstein and B. Cashore, “Can non-state global governance be legitimate? An analytical framework,” _REGULATION & GOVERNANCE_, vol. 1, no. 4, pp. 347–371, 2007.
  * [88] A. Kolk and R. van Tulder, “The Effectiveness of Self-regulation:: Corporate Codes of Conduct and Child Labour,” _European Management Journal_ , vol. 20, no. 3, pp. 260–271, 2002.
  * [89] National Science and Technology Council, “The National Artificial Intelligence Research and Development Strategic Plan,” National Science and Technology Council (NSTC), Networking and Information Technology Research and Development Subcommittee, United States, Washington, D.C, Tech. Rep., 2016.
  * [90] C. Y. Laporte, S. Alexandre, and R. V. O’Connor, “A Software Engineering Lifecycle Standard for Very Small Enterprises,” in _Software Process Improvement_ , ser. Communications in Computer and Information Science, R. V. O’Connor, N. Baddoo, K. Smolander, and R. Messnarz, Eds. Berlin, Heidelberg: Springer, 2008, pp. 129–141.
  * [91] M. A. Madaio, L. Stark, J. W. Vaughan, and H. Wallach, “Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI,” _CHI_ , p. 20, 2020.