# Designing Fiduciary Artificial Intelligence

Sebastian Benthall  New York University School of LawUSA<EMAIL_ADDRESS>and
David Shekman  Northwestern Pritzker School of LawUSA
<EMAIL_ADDRESS>
###### Abstract.

A fiduciary is a trusted agent that has the legal duty to act with loyalty and
care towards a principal that employs them. When fiduciary organizations
interact with users through a digital interface, or otherwise automate their
operations with artificial intelligence, they will need to design these AI
systems to be compliant with their duties. This article synthesizes recent
work in computer science and law to develop a procedure for designing and
auditing Fiduciary AI. The designer of a Fiduciary AI should understand the
context of the system, identify its principals, and assess the best interests
of those principals. Then the designer must be loyal with respect to those
interests, and careful in an contextually appropriate way. We connect the
steps in this procedure to dimensions of Trustworthy AI, such as privacy and
alignment. Fiduciary AI is a promising means to address the incompleteness of
data subject’s consent when interacting with complex technical systems.

††ccs: Security and privacy Human and societal aspects of security and
privacy††ccs: Social and professional topics Computing / technology
policy††ccs: Computing methodologies Artificial intelligence

##  1\. Overview

Fiduciary duties are some of humanity’s oldest rules (Frankel, 2011), dating
back to Hammurabi’s Code and legally recognized throughout the world today.
Named for _fiducia_ , the Latin word for trust, they concern the duties of an
_agent_ hired to perform a task for a _principal_. Because the agent is often
in a position of expertise or power over the principal, the fiduciary duty
legally bolsters the loyalty and care expected of the agent beyond the
specific terms of their contract. They apply to professional roles such as
health providers, legal and investment advisors, and the directors of trusts
and corporations.

Fiduciary duties already apply to technology companies operating in some
regulated sectors. In the European Union, under the Data Governance Act
111Regulation 2022/868 of the European Parliament and of the Council of 30 May
2022 on European data governance and amending Regulation (EU) 2018/1724 (Data
Governance Act), 2022 O.J. (L 152), 1-44. , data intermediaries will be
fiduciaries. Legal scholars have begun exploring the possibility of expanding
fiduciary duties to information controllers more generally (Balkin, 2015; Khan
and Pozen, 2019; Richards and Hartzog, 2020; Balkin, 2020). These broader
applications of fiduciary principles have appeared in recent legislative
proposals.222Federally — American Data Privacy and Protection Act, H.R. 8152,
117th Cong. (2022); Data Care Act of 2021, S.919, 117th Cong. (2021). For
state — H.B. 1749, 192nd Sess., (Mass. 2021); New York Consumer Privacy Act,
S. 6701B, 2021-22 Sess. (NY 2021).

Today, many companies engage their clients primarily through an automated user
interface. When companies deliver services at scale via a website, an app, or
a physical object in the Internet of Things, they delegate their operations to
a computational system. These systems are increasingly trained on data from
their users using general purpose machine learning algorithms and pretrained
models for tasks like facial recognition and natural language processing, and
hence are artificial intelligence systems333For a standard definition of
artificial intelligence systems, see OECD Recommendation on AI:2019; ISO/IEC
22989:2022. Referring to automated clients as Artificial Intelligences (AI)
can conflate the automated decisions made by at the interface with the ongoing
choices made by engineers, content moderators, designers, and product
managers. However, as the client interfaces become more automated and trained
using machine learning, the more referring to them as AI becomes more
literally correct.

This article is a guide to designing AI systems that are operated by people or
corporations that have fiduciary duties to others that are principals or
beneficiaries. We refer to an artificial intelligence that upholds fiduciary
duties owed by its operator as Fiduciary AI. 444This is also what we mean when
we discuss fiduciary duties for computational systems, or the application of
these duties to AI. Fiduciary AI is one form Trustworthy AI (TAI), a way of
designing intelligent systems that are lawful, ethical, and technically and
socially robust.

The first thing a designer of a Fiduciary AI needs is a working understanding
of what fiduciary duties are and why they have been applied. Section 2
presents the legal arguments for fiduciary duties in general. Fiduciary duties
are invoked when an agent holds power over its principals and when the
principal is unable to completely specify the actions of the agent through
consensual contract. These conditions hold in many applications of AI.

To the extent possible in an area of unsettled law, Section 3 discusses what
measures should be taken by the designer of an artificial intelligence system
in order for it to be compliant with these laws. We connect aspects of
Fiduciary AI to known dimensions of Trustworthy AI (Wing, 2021; Varshney,
2019), including privacy and AI Alignment. We outline a procedure involving a
systematic series of sensitizing questions to be asked when the AI is designed
or audited for compliance. For each step in the procedure, we then identify
how this aspect of Fiduciary AI has been illuminated in prior literature.

We maintain that a Fiduciary AI should be scoped to a particular context of
activity (Section 4) and designed with an identified set of principals in mind
(Section 5). Notably, these two steps of the procedure concern legal
requirements and the prioritization of beneficiaries, which are not technical
questions but nevertheless logically precede questions concerning the system’s
technical design.

With this contextual information in hand, the system can then be designed in
technical detail. The best interest of these principals must be assessed
(Section 6), which may involve machine learnt representations of the
principals’ best interests. If there are many principals, these interests must
be aggregated (Section 7). Fiduciaries are ultimately bound to duties of
loyalty and care. Fiduciary AI loyalty can be understood as alignment with the
assessed best interests of the principals (Section 8). Care refers to the
adherence to other best practices of community norms guiding the design of the
system, such as the determination of the systems’ inductive bias (Section 9).

##  2\. Fiduciary Principles and Regulatory Trends

Fiduciary duties are, foremost, legal constructions that encapsulate and
enforce expectations of trustworthiness. This section discusses fiduciary law
and is organized as follows. First, we will discuss fiduciary duties in
general as they are understood in many sectors that predate current uses of
computation and AI, with specific example of duties that can inform Fiduciary
AI design. Next we will discuss proposed expansions of fiduciary duties to new
computational domains. The application of fiduciary duties to computational
systems is currently an area of unsettled law. We do not in this paper take a
position on which systems ought to be legally held to fiduciary principles.
However, because legal outcomes are inherently normative and discursive, we
present the broad legal debate to best anticipate the engineering and design
principles that would guide a TAI system toward compliance.

We must note that fiduciary laws vary between jurisdictions. This paper will
utilize the language of Anglophone common law language to describe fiduciary
relationships, principles, and actors, and focus on U.S. and European
jurisdictions. The term “fiduciary duties”, as it is understood in the
Anglophone legal context, is difficult to translate directly across all
languages. However, the duties of loyalty and care which comprise the concept
have direct corollaries in the Western European context (Gelter and
Helleringer, 2019). Moreover, while there are significant differences between
common law and civil law variations of fiduciary duties, there has been a
convergence of principles with respect to _new_ fiduciary duties aimed at
actors in the data economy and, by extension, AI. We find that the key
principle of Fiduciary AI is that it must be aligned with the best interests
of the beneficiaries in order to remedy an imbalance of power and necessary
incompleteness in contracting.

###  2.1. Justifications of fiduciary principles

Fiduciary duties arise from a recognition that specialization is an important
factor in society. It would be nonsensical and rather inefficient for every
person to gain the expertise necessary to become a doctor, lawyer, or
financial advisor every time they need one (Frankel, 2011). Instead, society
relies on those individuals with whom that expertise is concentrated. However,
this creates a risk for beneficiaries. Fiduciary duty dictates that
fiduciaries must act with loyalty and care to their beneficiaries and face
stiff penalties if they violate these duties. Fiduciary duties are justified
in legal scholarship according to at least two lines of argument — an argument
based on the vulnerability of the principal in fiduciary relationships and an
argument about economic efficiency under conditions when transaction costs
make contracting necessarily incomplete.

####  2.1.1. Fiduciary relationships

The most common account of fiduciary duties focuses on the relationship
between the principal and the agent, where the asymmetry of knowledge and
power creates a fiduciary relationship (Miller, 2019). That asymmetry comes
from what Frankel (2011) calls “entrustment”. Patients trust doctors with
their bodies, clients trust financial advisors and lawyers with their money
and discretion. This entrustment of power over oneself or on one’s behalf
raises the potential for harm by the trustee. Because of the vulnerability
this creates, additional restrictions beyond commercial transactions’ standard
obligation of good faith have been established to protect the principals.

Certain roles or statuses have been deemed to be fiduciaries by conventional
wisdom (and common law), and fiduciary principles attach to agents by virtue
of membership in such status. Miller (2019) notes trustees, directors, agents,
lawyers, and doctors as well-known examples. Courts may also apply to
fiduciary duties in other cases wherein the facts bear that the agent’s role
bears significant enough resemblance to a fiduciary, for example because they
cultivate a relationship of trust, hold themselves out as an expert in the
field, are relied on by a person for advice, and have that person’s complete
trust (Kelly, 2019).

####  2.1.2. Contractarian justification

Parallel with the fiduciary relation arguments, the Law and Economics (L&E)
branch of legal scholarship has endorsed fiduciary duties on different
grounds. Easterbrook and Fischel (1993) leave morality out of the equation,
aligning more closely with a contractarian approach. Contractarians view the
entire commercial milieu, both agency relationships and the corporation
itself, as composed of contracts (Jensen and Meckling, 1976). By default, the
design burden lies with the contracting parties. Easterbrook and Fischel
(1993) view the courts as playing an active role in specification of the
contract via interim or ex post adjudication. The court functions as the
theoretical completing piece of the incomplete contract between the parties
(Brooks, 2019).

This is especially important in cases where there is a severe imbalance of
transaction costs against the principal party. Monitoring and specification
costs, it is posited, would prevent the negotiation of contracts to
completion. But the duty of loyalty (intention) and duty of care (execution)
establish, in an incomplete contract, the standards that would be arrived at
were full negotiation possible. The expertise of the agent is relied on to
“complete” the scope of the contract according to those duties; the agent is
expected to charge a premium for foregone opportunities that result from the
duties. Thus the principal can engage the transaction trusting to get what
they wanted from it: the agent’s expertise.

Fiduciary duties are a way to solve the principal-agent problems of moral
hazard and hidden action. Rather than disempower the agent, fiduciary duties
allow the principal an ex post review of the agent’s behavior. The duties are
written as broadly as possible in order to cover the territory left open by
incomplete contracts (Sitkoff, 2011). This enables contracts that would have
otherwise failed due to the large transaction costs (Easterbrook and Fischel,
1993).

####  2.1.3. Contextuality and subsidiary duties

In the United States, the primary fiduciary duties are the duty of loyalty and
duty of care. These are broad, open-ended standards used across many sectors.
In practice, these broad duties are interpreted by courts into more granular
_subsidiary duties_ that are “field-specific elaborations” (sitkoffother) of
the primary duties. The definitions of subsidiary duties crystallize the
interpretation of broader duties of loyalty and care into forms that are
easier to comply with and enforce. Table 1 displays subsidiary duties for a
variety of professional statuses to which fiduciary duties apply, as well as
speculative subsidiary duties that might apply to computational systems. In
new fiduciary duties introduced by statute, subsidiary duties may be explicit
legal clauses that accompany expressions of the more general duties. Often
these subsidiary duties refer to the flow of information from the agent to or
about the principal.

Context | Loyalty | Care | Both
---|---|---|---
Trusts |  Incurring only reasonable costs | Prudent investor rule |  Giving account to beneficiaries†
|  |  Duty to not commingle trust property | Record-keeping†
Corporate management |  No usurpation of corporate opportunity |  Need for monitoring and compliance |  Disclosure to shareholders†
|  No impairment of shareholder meetings |  |
|  Boardroom confidentiality† |  |
Investment advice |  Best execution of instructions | Prudent investor rule |  Keeping books and records†
Legal representation |  No conflicts of interest |  Safeguarding the client’s confidences† |  Communication with the client†
Health care | Confidentiality† |  |  Informed consent†
(DGA) Data Intermediaries‡ | Facilitate exercise of rights | Security and confidentiality$\dagger$ | Notice of data uses$\dagger$
| Act in subject’s best interest | Ensure interoperability | Consent management
Data Processors‡ (Hartzog and Richards, 2022) |  |  |
Collection | Data minimization | Proper disposal$\dagger$ | Data records$\dagger$
Personalization |  No conflict with collective best interests | Eliminate disparate impact |
Gatekeeping |  No exposing principals to privacy harms | Counterparty evaluation | Proper security practices$\dagger$
Influence | Eliminate dark patterns |  |
Mediation | Disincentivize user harms | Mediation algorithms oversight |
|  Hierarchical loyalty for heterogeneous roles |  |
AI Assistants‡ (Aguirre et al., 2020) | No conflicts of interest |  Clearly indicate potential conflicts$\dagger$ |
| Transparent objective functions | Adequate requests of user input$\dagger$ |
Table 1. Examples of subsidiary duties by legal field. Includes hypothetical
examples of subsidiary duties for information processors. Subsidiary duties
marked with a dagger ($\dagger$) concern information flows. Contexts marked
with a double dagger (${\ddagger}$) are fiduciary contexts not yet determined
by law but suggested in the literature (Aguirre et al., 2020; Hartzog and
Richards, 2022).

####  2.1.4. Example: the prudent investor rule

An example of a subsidiary duty is the _prudent investor rule_ , a subsidiary
duty of care in the context of trust law. 555Originally, the “prudent man
rule” was a common law standard enshrined in 1830 (_Harvard College v. Amory_)
that a trustee prioritize regular income over speculative value when managing
trust assets. Later, this was updated in the American Law Institute’s
Restatement Third of Trusts and promulgated as the Uniform Prudent Investor
Act (UPIA). (Restatement (Third) of Trusts § 90 (2007), 61 A.L.R.7th Art. 1)
This rule, which has since been adopted by many U.S. states, aligns the
fiduciary duties of a trust with modern portfolio theory (Markowitz, 1968),
which is a scientifically validated standard of asset management. As a
scientific expert standard, it generalizes the idiosyncratic interests of
trustees in a way that is consistent with the purpose of the fiduciary as a
trustee. The trustee is bound by the duty of care to this formulation of the
principals’ interests, while the duty of loyalty requires them to put the
principal’s interests over their own.

####  2.1.5. Example: the duty of impartiality

One example of a subsidiary duty that arises from the duty of loyalty, which
will be relevant to the design of Fiduciary AI, is the _duty of impartiality_
, which applies to trusts and corporate directors. This duty comes into play
when there may be conflicting interests between multiple beneficiaries (Gold,
2019), such as trustees with conflicting preferences (Schwarcz, 2009) or
different classes of shareholders (Nir, 2020). Impartiality does not mean that
fiduciaries must treat each beneficiary equally, merely that their balancing
of beneficiaries’ interests is not influenced by the fiduciary’s self-interest
or favoritism. The corporate directors can change the weights they assign to
the preference sets of differing shareholders according to circumstance (Nir,
2020). We discuss this further in Section 7.

###  2.2. New computational fiduciaries

Recent scholarship, advocacy, and legislation has called for extending
fiduciary duties to new contexts associated with AI and digital services. We
briefly consider some of the arguments and controversies of these proposals
here.

####  2.2.1. As a remedy for incomplete contracting.

One key motivation for extending fiduciary duties to AI and digital services
is the insufficiency of the “notice and consent” framework currently governing
consumer privacy and data protection more broadly. Users of a commercial
digital service sign a contract with the provider that determines the terms of
their interaction, including the use of their data. It is well known that most
users will not read the contracts that they digitally sign (McDonald and
Cranor, 2008; Obar and Oeldorf-Hirsch, 2020). In some cases, the system’s
design can be such that the offered consent can be considered either
uninformed (c.f. (Friedman et al., 2000; Romanosky et al., 2006)) or
manipulated (Calo, 2013; Susser et al., 2019; Bongard-Blanchy et al., 2021).
Such a design need not be intentional. An artificially intelligent system
trained to optimize an objective function (such as total clicks on
advertisements) can _learn_ to present an interface to users that elicits
their uninformed consent. In short, the burden of the transaction costs to
users of asserting their interests through contracting prohibitively high, and
AI operating with conflicted interests can exploit this. Corroborating this
view, Hadfield-Menell and Hadfield (2019) have drawn the connection between AI
Alignment and the legal challenges around incomplete contracting, and conclude
that AI Alignment will depend on a larger legal framework in which a
community’s normative structure is imputed into the terms of the relationship
between the principal and the agent. As discussed in Section 2.1.2, fiduciary
duties are an available legal tool for establishing the alignment of an agent
to a principal when contracting is incomplete.

####  2.2.2. Example: AI assistants

Aguirre et al. (2020) consider specifically the case of AI assistants like
Alexa, Siri, Google, and Cortana which serve as interfaces between consumers
and the web. While these assistants may appear to act in the interests of
their users, they may be designed with embedded conflicts of interest. Duties
of loyalty can correct this.

####  2.2.3. Example: E.U. Data Governance Act (DGA)

The European Union recognizes a fundamental right to data protection (Schwartz
and Peifer, 2017). In 2016, the EU passed a landmark data protection law, the
General Data Protection Regulation (GDPR), which received global attention due
to its strong extraterritorially enforced sanctions (Streinz, 2021). Under the
GDPR, consent is a legal basis for lawful processing of personal data that is
often invoked in commercial applications.666Article 6 Regulation (EU) 2016/679
of the European Parliament and of the Council of 27 April 2016 on the
protection of natural persons with regard to the processing of personal data
and on the free movement of such data, and repealing Directive 95/46/EC
(General Data Protection Regulation), art. 6, OJ 2016 L 119/1. This includes
data collected about internet browsing behavior through cookies, which is a
key data source underpinning targeted advertising. Consent Management
Platforms (CMPs) are software solutions that collect, store, and monitor
users’ consent to uses of this personal data (Santos et al., 2021). Use of
these CMPs has expanded rapidly since the passing of GDPR (Hils et al., 2020).
However, these CMPs have not successfully guaranteed data protection rights as
the GDPR has intended. 777The Transparency and Consent Framework (TCF)
(Europe, 2020), a standardized CMP developed by IAB Europe (an association for
the digital marketing industry), was found to violate GDPR by Belgium’s Data
Protection Authority on several grounds. (Belgian Data Protection Authority,
Decision on the Merits 21/2022 of 2 February 2022, Complaint Relating to
Transparency & Consent Framework (IAB Europe), DOS-2019-01377,
https://www.autoriteprotectiondonnees.be/publications/decision-quant-au-
fond-n-21-2022-english.pdf) Others have frustrated privacy scholars and users
alike by employing dark patterns and choice architecture to push users toward
invalid or uninformed consent (Nouwens et al., 2020). Indeed, researchers have
found that in widely deployed CMPs, data is collected, processed and shared
even when users have not consented to it (lieu2022opted).

European lawmakers have passed the Data Governance Act (DGA), which addresses
the uncertain legal status of CMPs by identifying them as a type of “data
intermediary”. 888 _See_ Recital 30, Regulation 2022/868 of the European
Parliament and of the Council of 30 May 2022 on European data governance and
amending Regulation (EU) 2018/1724 (Data Governance Act), 2022 O.J. (L 152),
11. According to the DGA, “data intermediation services providers seek to
enhance the agency of data subjects, and in particular individuals’ control
over data relating to them”.999Recital 30, Regulation 2022/868 of the European
Parliament and of the Council of 30 May 2022 on European data governance and
amending Regulation (EU) 2018/1724 (Data Governance Act), 2022 O.J. (L 152),
11.  The Act uses the language of fiduciary duties and “act[ing] in the best
interests of the data subjects”.101010“Data intermediation services providers
that intermediate the exchange of data between individuals as data subjects
and legal persons as data users should, in addition, bear fiduciary duty
towards the individuals, to ensure that they act in the best interest of the
data subjects.” Recital 33, Regulation 2022/868 of the European Parliament and
of the Council of 30 May 2022 on European data governance and amending
Regulation (EU) 2018/1724 (Data Governance Act), art. 12(m), 2022 O.J. (L
152), 12-13.

Under this law, data intermediation is its own fiduciary context. The DGA
restricts data intermediaries to using collected data for only the purposes of
its intermediation services. It further prohibits the pricing and other
commercial terms of the service from being dependent on the customer’s degree
of use of any other service. In short, data intermediaries under the DGA are
fiduciaries that assist individuals in exercising their data rights for their
best interests.

####  2.2.4. Information fiduciaries and their discontents

While fiduciary principles are already relevant to computational systems
operated in sectors regulated by existing fiduciary law, and to data
intermediaries by the DGA, recent work has proposed the expansion of fiduciary
principles to data processors more general, creating so-called “information
fiduciaries” (Balkin, 2015) with “data loyalty” (Hartzog and Richards, 2022).
Several bills have been proposed both at the federal and state level aiming to
apply duties of loyalty and care, as well as subsidiary duties, to
platforms.111111Federally — American Data Privacy and Protection Act, H.R.
8152, 117th Cong. (2022); Data Care Act of 2021, S.919, 117th Cong. (2021).
For state — H.B. 1749, 192nd Sess., (Mass. 2021); New York Consumer Privacy
Act, S. 6701B, 2021-22 Sess. (NY 2021). 121212This trend is not limited solely
to the US. In India, the 2021 Data Protection Bill, currently under discussion
by the parliament (Mandhani, 2021), explicitly creates data fiduciaries and
data principals. Though the language of data fiduciaries is used in this bill,
the functional implications appear to align closer with that of the “data
controller” under the GDPR regime. (trilegal)

Broad information fiduciary laws are controversial. They would upset how many
well-established multi-sided platform companies (Cusumano et al., 2019), which
necessarily navigate conflicting interests of multiple stakeholders, do
business – and hence how they would use AI, such as recommendation systems.
This raises legal questions about how an information fiduciary would manage
conflict duties to principals in different roles, such as its users,
advertising partners and shareholders (Khan and Pozen, 2019; Grimmelmann,
2019). However, corporate fiduciaries are still bound by the measures of the
law (Tuch, 2021) . Committing an illegal act would be detrimental to the
financial value of the company (either through impositions of fines or other
penalties, reduced revenues, or loss of brand value/goodwill) and thus the
shareholders’ interests. If fiduciary duties to users of platforms were
enacted into law, then that would be another legal constraint within which the
platforms’ corporate officers would need to act. We consider the
prioritization of multiple beneficiary roles in Sections 5 and 8

##  3\. Engineering Fiduciary AI Systems

For the reasons discussed in Section 2, many technology businesses may find
that they have fiduciary duties. Given that businesses increasingly rely on
automation and AI for their operations, and that they can be accountable to
fiduciary duties for these operations, these businesses should be aware of how
their automated systems can be designed _ex ante_ to avoid violation of their
legal obligations. To this end, some technical interpretation is helpful for
identifying what constitutes compliance with the law and to what extent
compliance can be guaranteed by an auditor. 131313Of course, it may be that no
_ex ante_ design will be sufficient for compliance with fiduciary duties,
given the possible _ex post_ actions of businesses and the variability of
judicial interpretation. Some legal scholars are openly skeptical about the
possibility of engineering compliance (Cohen, 2016; Waldman, 2019, 2021).
However, we will connect fiduciary duties to Trustworthy AI principles and
techniques as a pragmatic exercise. We acknowledge that there are both limits
to (Selbst et al., 2019) and value in (Abebe et al., 2020) the use of
computational abstraction to clarify normative guidelines. But for AI systems
in particular, designers must make ethical choices. For example, Baum (2020)
argues that there are three sets of decisions in a social AI system that
require the imposition of the designer’s values: standing, measurement, and
aggregation. We find that analogous questions arise when designing a fiduciary
AI. We have replaced “standing” with “identification”, and “measurement” with
“assessment”. Standing, in Baum’s context, is “to have one’s ethics included
in a social choice process used to determine the ethics of AI” Baum (2020).
However, this differs from the use of the term “standing” in law to refer to
whether one has the ability to bring suit.

The remainder of this paper assumes that a technologist is interested in
implementing a fiduciary service or artificial agent. What do fiduciary duties
mean for their technical practice and design? We contribute a procedure for a
system architect to consult when attempting to build a new system that is
compliant-by-design with fiduciary duties, or when auditing an existing system
for compliance. It is formulated as a rubric of questions. (See Table 2.) The
questions are presented in a logical sequence such that the answer to early
questions, such as “What is the context of the system?” and “Who are the
principals?” inform the answers to later questions, such as “Is the system
aligned with the best interests of the principal?”.

While many of the designers decisions are quite technical, we find that
subsidiary fiduciary duties may directly inform the design at many stages. For
example, a reasonable person standard from a subsidiary fiduciary duty can
inform the answer to the question, ”What are the best interests of the
principals?”, which would otherwise be derived from data. In this way,
subsidiary duties may guide _ex ante_ expectations of behavior and guide
designs. However, fiduciaries will also be held to the more abstract, primary
fiduciary duties of loyalty and care. These latter duties allow for
flexibility by the courts and regulators in _ex post_ analysis.

Further sections of this paper address facets of our rubric for Fiduciary AI
by outlining corresponding techniques or frameworks from computer science.
Naturally, in an area of unsettled law, using such a rubric will be necessary
at best, and never sufficient, for compliance with regulations.141414In the
future, such sufficient conditions could be defined by law via a “safe harbor”
provision, which is essentially a well-understood floor of expectations above
which a covered entity will be considered to be compliant. And in many cases,
the TAI dimensions implicated by fiduciary duties are active research areas
with many open questions. We see Fiduciary AI as an emerging area of both law
and engineering. What follows is a survey of the state of the art at the
intersection of these fields.

1\. Context | What social context or sector will the system will operate in? | Section 4
---|---|---
| What are the purposes of that context? |
| What roles are defined in that context? |
| What norms or rules does that context imply? |
2\. Identification | Who are the principals? | Section 5
| Is there more than one category of principals? |
| Do the categories have different levels of priority? |
3\. Assessment | How are the interests of the principals being assessed? | Section 6
| Are they being empirically measured? With what data? | Section 6.1
| Are future rewards discounted? By how much? | Section 6.2
| Is there are reasonable person standard of best interest? | Section 6.3
4\. Aggregation | If there is more than one principal, how are their interests aggregated? | Section 7
| Whose interests dominate in a conflict? |
| Are there duties about aggregation, e.g. impartiality? | Section 7.2
5\. Loyalty | Is the system aligned with the principals’ interests? | Section 8
| Have subsidiary duties of loyalty been considered? | Section 8.2
6\. Care | What is the context-appropriate level of prudence? | Section 9
| Is the inductive bias of the system up to contextual standards? | Section 9.1
Table 2. Steps for designing or auditing a Fiduciary AI for compliance, with
corresponding sections of this paper where they are discussed.

We see Fiduciary AI practice as involving a novel combination of Contextual
Integrity (Nissenbaum, 2020), a theory of ethical computing design that
focuses on contextualized social norms, and the problem of AI Alignment. The
AI alignment problem, or how to design an AI that acts in a way that is
aligned with the intentions of principal users, has been most widely discussed
in the growing field of Artificial General Intelligence (AGI) research (Dewey,
2011; Hadfield-Menell et al., 2016; Everitt et al., 2021b). Fiduciary duties
are one form of legally binding, contextualized normative structure that can
address the necessary incompleteness in the relationship between AI and its
users (Hadfield-Menell and Hadfield, 2019). Thus, Fiduciary AI design can be a
precursor to a better understanding of Aligned AI (Aguirre et al., 2020).

##  4\. Context

In the law, there are no absolute fiduciaries who are required under all
circumstances to serve the interests of a principal. Rather, individuals are
fiduciaries to others by virtue of their respective roles in a legally
recognized context. A first design principle for Fiduciary AI is that the
context within which the AI is acting as a fiduciary must be established and
understood.

The study of how context should inform computational system design has deep
roots in ubiquitous computing research (Ackerman et al., 2001; Dey et al.,
2001; Dourish, 2004). We recommend that Fiduciary AI draw on Nissenbaum’s
Contextual Integrity (CI) (Nissenbaum, 2020), a well-developed theory of
social contexts that is used in the design and evaluation of technical
systems151515Contextual Integrity is mainly about norms of information flow,
and defines privacy as appropriate information flow. Each norm is
parameterized in terms of actors (sender, receiver, and data subject);
attributes (meanings of information) and transmission principles (for instance
confidentiality, reciprocity, or “with a warrant”). The analyst is encouraged
to evaluate the introduction of a new technology with a heuristic procedure
that identifies how any new information flows depart from the norms embedded
in the context of interest. One of the steps in the procedure is to evaluate
any changing information flows in terms of the purposes of the context as it
functions in society more broadly. Ultimately, the norms are animated by the
specific purpose of that context, as well as societal values and individual
goals.. It begins from the position that a pluralistic society (Walzer, 2008)
can be understood to consist of multiple social contexts or spheres, each with
its own internal system of roles, rules, and meanings. CI offers a rubric for
understanding contexts in terms of:

  * •

_Purposes_. The purpose of the social context in society, towards which its
laws and norms are instrumental.

  * •

_Roles_. The names and definitions of roles that agents can have in that
context.

  * •

_Norms_. Social or legal rules prescribed within the context about how agents
of the identified roles should interact.

Translating the CI framework into technical requirements is an ongoing area of
computer science research (Benthall et al., 2017) We do not recommend any
particular implementation of CI, but rather recommend using it to frame the
answer to a specific question: in what context is an AI expected to be
performing fiduciary duties? This understanding of context in terms of roles,
purposes, and norms will be used to answer other questions related to
identification (Section 5) and aggregation (Section 7). Furthermore, CI is a
fitting framework for evaluating the context of fiduciary duties because some
subsidiary duties, like duties of disclosure and confidentiality, are
explicitly about information flow. We develop this case as an example of
alignment in Section 8.2.

##  5\. Identification

Designing Fiduciary AI requires an explicit understanding of who a system’s
principals are. The question “who are the beneficiaries of this system?” has
long been one of the first questions to ask when designing any social system
(Ulrich, 1987). In most cases today, AI is designed mainly to benefit those
that deploy the system, using mechanism design techniques to steer the
behavior of users towards those goals (Viljoen et al., 2021). Indeed, some of
the literature on AI alignment assumes that the goal of AI safety is to ensure
that the AI system is aligned with the interests of its operator (Amodei et
al., 2016), while others defer the question of the “preference payload” to
other research (Leike et al., 2018).

Fiduciary AI can be designed with other categories of principals in mind, and
identifying these principals is essential for compliance with fiduciary
duties. This choice cannot be founded on purely technical considerations,
rather, it must be inferred from the previously identified legal or social
context (see Section 4). For example, many companies that build AI products
operate under Delaware corporate law and their corporate directors have only
one pressing set of fiduciary obligations: to their shareholders. But a
business that is based in Delaware but also is operating as a data
intermediary in the European Union would need to prioritize its duties to its
data subjects because legal penalties for violating those duties would be, in
turn, bad for the shareholders (see Section 2.2.4).

To illustrate the significance of the fiduciary relationship, we consider a
typical online system that interacts with human users in a commercial context.
In the absence of a fiduciary relationship, the human user’s engagement with
the system will be regulated by a contractual agreement to which the user has
legally consented. (See Section 2.2.1.) We note that _for those users
identified as principals of a fiduciary system_ , consent is _not sufficient_
for the fulfillment of the fiduciary duty. Rather, other requirements such as
loyalty (see Section 8), must govern the interface design to ensure as much as
possible that consent, when given, is in the interests of the principal.

In a similar vein, consider a user who has consented to a system’s broad terms
and conditions and now operates the system through a user interface. Normally,
a system would be expected to perform whatever actions the user chooses for it
to do. However, for a complex system, the instructions given will never be a
complete description of the operations to be performed, and the choice of
instructions may be manipulated by a system’s design. Indeed, there is a
division in law between two models for the interpretation of the duty of
loyalty — the _obedience_ model, wherein the fiduciary must follow directions
given by the beneficiary, and the _best-interests model_ , wherein the
fiduciary acts with more discretion but in service to the beneficiary. The
latter best-interests model is favored by Richards and Hartzog (2020) for
privacy and information platform cases, as it is precisely the complexity of
the technical systems and inability of consumers to grasp their mechanics
which makes contracts (as well as instructions) incomplete or impossible to
monitor. This requires judicious design choices on the part of the data
controller, e.g., the choice of an email provider to maintain deleted emails
on record for a short period of time in case the user ordered their deletion
by accident.

In summary, while most digital or AI systems will be operated by users on the
basis of consent and obedience, Fiduciary AI imposes a higher standard on the
system’s design with respect to its _principal_ users. For example, a medical
AI might be designed to serve the best interests of its medical patients (the
principals), but have a different interface used by system operators (not
principals) that is more strictly obedient. For the remainder of this article,
we will assume that the principals have been identified and the Fiduciary AI
designer must now determine how to fulfill the system’s duties to those
principals.

##  6\. Assessment

Once the principals of a fiduciary AI system have been identified, system
designers should assess the salient best interests of those principals. This
is not a trivial problem by any means: how many people can judge what is in
another person’s best interest, or even their own? Luckily, this problem is
constrained; the system designer must only assess the best interests of the
principal in their role within the fiduciary context.

One approach to assessing these best interests is to learn an objective
function from data provided by the population of principals. The data may be
observations of behavior, statements of preferences, or direct measurements of
user well-being. We will discuss these data sources and their associated
challenges.

Beyond empirical assessments of best interest, the law also furnishes some
fiduciary contexts with _reasonable person standards_ , which are legal
codifications of the interests of a normal or general person in a given role.
In Fiduciary AI cases, a combination of empirical data collection, machine
learning, and reasonable person standards may be used to determine principal
best interests.161616While some researchers have found improved results using
observed human behavior or feedback to train a policy function directly
instead of training a reward function (Knox and Stone, 2010; Griffith et al.,
2013), we do not recommend this as a way of achieving Fiduciary AI, because
Fiduciary AI is specifically concerned with the best interests of the
principals. We also do not recommend, for this application, inferring reward
functions from observable states of the world (Shah et al., 2019), as
Fiduciary AI has been proposed as a solution to problems with the status quo
due to, for example, market failure. Once the information about each
principal’s best interests are identified, they can be aggregated if
appropriate (Section 7), and then rendered into technical design (Section 8),
perhaps as an objective function or reward model (Leike et al., 2018).

###  6.1. Learning reward models

One strategy for assessing the best interests of principals is to attempt to
learn a proxy model of those interests from observational data. This approach
has been best explored in the context of reinforcement learning. While we do
not limit the scope of this work to reinforcement learning, we nevertheless
point to reward modeling (Leike et al., 2018) or reward design (Hadfield-
Menell et al., 2017) as key elements of Fiduciary AI.

Consider the Bellman equation form of the intertemporal decision problem:

(1) $V(s)=\max_{a}r(s,a)+\beta\mathbb{E}\left[V(s^{\prime})|s,a\right]$

Where $V$ is the value function for a state $s$, given an optimal action $a$.
An implied probability distribution $P(s^{\prime}|s,a)$ governs the state
transitions. The value function is defined recursively as the sum of immediate
rewards $r(s,a)$ and the expected value of future rewards
$\mathbb{E}\left[V(s^{\prime})|s,a\right]$, discounted by a factor of
$0<\beta<1$.

We define $Q$ as the state-action value function:

(2) $Q(s,a)=r(s,a)+\beta\mathbb{E}\left[V(s^{\prime})|s,a\right]$

An agent’s policy $\pi$ maps from states to actions. In the reinforcement
learning context, the policy is trained to optimize the action-value function.

(3) $\pi^{*}(s)=\operatorname*{arg\,max}_{a}Q_{\pi}(s,a)$

Researchers working in AI Alignment have worked on techniques for learning a
proxy reward function $r$ that reflects the true objectives of a system’s
principals. This function can then be used in training the AI system’s policy
through reinforcement learning.

Reward learning comes with many challenges. In all forms of reward design, the
proxy reward function will, at best, be based on observations of the true
reward function, leaving the possibility for misalignment due to
incompleteness (Zhuang and Hadfield-Menell, 2020). It is likely that the best
approach will combine several different, complementary forms of human
feedback, perhaps guided by a common framework (Jeon et al., 2020). On-line
reward learning, wherein the reward function is continuously trained on data
learned through its operation, opens the learning process up to manipulation
by the system during training (Fickinger et al., 2020; Armstrong et al.,
2020), so it is best to be avoided. Furthermore, each data source has
associated threats to its validity as a measure of best interests. And in all
cases, the inference of the proxy function will have inductive bias (Xu et
al., 2019) (see Section 9.1).

####  6.1.1. Observations of behavior

Learning the objective function of an agent based on observations of their
behavior is an enduring research problem. One early formulation of this
problem is the _inverse reinforcement learning_ (IRL) paradigm (Ng et al.,
2000; Abbeel and Ng, 2004; Ziebart et al., 2008) . In the simplest
formulations of inverse reinforcement learning, a subject’s behavior is
observed and this is interpreted as a policy $\pi^{*}$ that corresponds to
their value function $V$. In principle, this allows the analyst to derive the
subject’s reward function $r$.

IRL faces a number of challenges. Ng et al. (2000)’s original analysis showed
that IRL suffers from the problem of _degeneracy_ – that there are many reward
functions for which an observed policy is optimal. Gleave et al. (2020) have
shown that reward functions can be grouped into equivalence classes based on
their effects on policy, which helps quantify distances in the space of reward
functions. But in any case, the outcome of IRL will depend on inductive bias
(Section 9.1) as well as assumptions about temporal discounting (see Section
6.2).

Another set of challenges concerns how well the behavior of users reflects
their best interests. Indeed, one might want a fiduciary service precisely
when one is unable to act (or represent) in one’s best interest. And as a
matter of general fact, people do not always act in their best interest due to
cognitive biases (Kahneman and Krueger, 2006). While there have been some
efforts to adapt IRL to a “two system” cognitive architecture (Peysakhovich,
2019), it has been shown that, in general, reward functions in an IRL context
are unidentifiable if the principal’s behavior is irrational (Armstrong and
Mindermann, 2018).

A promising direction for IRL is Cooperative Inverse Reinforcement Learning,
wherein the human trainer is an expert who deliberately teaches the system its
objectives (Hadfield-Menell et al., 2016). In other words, the system is not
trained on behavioral data from “in the wild”. Behavioral data, and especially
expert training data, may be be most useful in practice when combined with
other sources.

####  6.1.2. Preference judgments

Another way to gather data about principal’s objectives is to solicit their
explicit preference judgments. Noothigattu et al. (2018) design a system for
learning “trolley problem” ethics by collecting data from human users about
their preferences over of different outcomes. In the context of deep
reinforcement learning, objective functions have been trained from human
judgements about the value of fragments of the system’s behavioral
trajectories (Christiano et al., 2017). These fragmentary judgments are then
aggregated and recomposed into a complete objective function. Preference
judgments can be fruitfully complemented with other sources of information
such as expert demonstrations (Ibarz et al., 2018). Researchers have
experimented with grounding natural language commands into the reward function
of agents in an IRL context (Squire et al., 2015; Bahdanau et al., 2018; Fu et
al., 2019).

A challenge associated with using this data source is that human preferences
do not have the normative properties of associated with mathematical decision-
theory, such as transitivity and insensitivity to alternatives (Tversky and
Kahneman, 1989). They are also unstable; well-known cognitive science results
have demonstrated that human preference judgments are susceptible to framing
effects, especially framing of a decision as a chance of gain versus a risk of
loss (Tversky and Kahneman, 1985). Social scientists who survey the
preferences of broad populations have learned that for studies to be
effective, subjects need to have a positive reason for valuing the accuracy of
the result (Zawojska and Czajkowski, 2017). In a rare treatment of these
threats to validity in the AI training setting, Thomaz and Breazeal (2008)
find that outcomes are sensitive to how human trainers understand their
relationship with the learning robots. More research is needed to understand
the human side of learning reward functions from human preferences.

####  6.1.3. Measuring well-being

Another way to discover principal preferences is to measure their well-being
directly. Behavioral economists such as Kahneman and Krueger (2006) have
addressed the problem that actions do not reveal true preferences and have
developed alternative, independent methods of measuring well-being, such as
survey techniques. Well-being measurements like these have been employed in
psychology and human-computer interaction (HCI) studies of the users of social
media platforms, and used to assess user interface designs (Kross et al.,
2013; Panger, 2018; Burr et al., 2020). IEEE (Committee et al., 2020) has
since published scientifically valid well-being metrics and a Well-Being
Impact Assessment (WIA) process.

Well-being metrics are sometimes gathered to inform public policy (Graham et
al., 2018). They can combine both objective elements (such as health or
employment) (Linton et al., 2016) and subjective elements (such as frequency
and duration of positive feelings) (Diener et al., 2009). Well-being
measurement is therefore largely separate from the fiduciary context and may
be a noisy signal with respect to the contextualized objectives of the
principal. Nevertheless, they may be a worthwhile complement to behavioral and
preferential data.

###  6.2. Temporal discounting and time inconsistency of preferences

Temporal discounting presents further complications and opportunities in the
assessment of principal interests. In inverse reinforcement learning, the
discount rate $\beta$ is a free parameter whose value must be either assumed
or fit when deriving a human model’s reward function from their
behavior.171717Mathematically, the discount factor $\beta<1$ is necessary for
the sum of the infinite series of expected future rewards to converge on a
finite value. Rothkopf and Dimitrakakis (2011) explore techniques for
inferring discount factors in IRL through a Bayesian process.

However, time inconsistency is yet another way in which principals may be
irrational, frustrating attempts to learn their “true” objective function.
Empirically, people’s discounting of future utility takes the form of
hyperbolic discounting, as opposed to exponential discounting (Benzion et al.,
1989; Green and Myerson, 2004; Chabris et al., 2008).181818Hyperbolic
discounting means that delays that are closer in time feel more costly than
the same delays further away in time, such that, for example, one prefers to
pay more to prevent a delay from today to tomorrow than they would to prevent
a one-day delay a year from now (from 365 days from today to 366 days from
today). Hyperbolic discounting and closely related ideas of time inconsistency
of preferences and present bias (Laibson, 1997; O’Donoghue and Rabin, 2001)
may help explain the “privacy paradox,” wherein a concern for future privacy
risks can be easily subsumed in the present by a desire for instant
gratification (Acquisti and Grossklags, 2003). We do not use the phrase
“privacy paradox” uncritically. A robust discussion of the privacy paradox is
beyond the scope of this paper. C.f. (Solove, 2021) This means that even if a
discount factor is fit to human behavior or preference judgements, it may an
artifact of the method than a representation of the human subject’s
psychologically held values. Correcting these inconsistencies may be a
positive role for a fiduciary to play as an advisor. Puaschunder (2021) argues
that a data fiduciary may be duty-bound to correct these inconsistencies on
behalf of their principals and be more rational, or even more patient, than
they are.191919The financial context suggests how patience might be a virtue
of fiduciary advice. Differences in discount factor are one macroeconomic
explanation for the varied distribution of wealth in society (Krusell and
Smith, 1998; Carroll et al., 2017). An impatient consumer is more likely to
consume more today and save less for the future. One beneficial role of an
investment advisor fiduciary is to nudge their client to consider the long-
term implications of their decisions, in effect advising the principal on what
they should do if they discount their future less. A Fiduciary AI might learn
a principal’s reward function $R$ assuming one level of patience, and then
provide recommendations to them based on the same $R$, but a greater patience
$\beta$.

###  6.3. Legal constraints

Sometimes a legal rule can define a context-specific sense of ’best interest’
that simplifies assessment a great deal. For example, the prudent investor
rule (see Section 2.1.4) established that for the management of trusts, the
principal’s interests are best served by investing according to modern
portfolio theory. This simplifies the fiduciary’s role to one of maximizing
risk adjusted returns on investment; they do not need to take into account
non-monetary measures of well-being. An information fiduciary statute could
likewise construct a standard of user interests based on a combination of a
strictly defined context and scientific results.

While such standards simplify the assessment process, they may allow system
designers to ignore important differences in user preferences, wheres privacy
preferences have been shown to be quite heterogeneous (Urban and Hoofnagle,
2014; Egelman and Peer, 2015; Marwick et al., 2018). (An alternative approach
would be to assess preferences individually, and then simplify the fiduciary’s
goal function through aggregation, as discussed in Section 7.2).

Translating these legal constraints into a technical specification is a
challenge in its own right. In some cases, a translation into a formal
specification language such as linear temporal logic is possible (Datta et
al., 2011). If the rule is better operationalized as a component of or
constraint on an objective function, it may need to be trained based on
expert-provided data. For example, Noothigattu et al. (2019) have used IRL to
learn ethical constraints for a system that is otherwise trained with
reinforcement learning.

##  7\. Aggregation

As discussed in Section 2.1.5, one of the conceptual challenges raised by
computational fiduciaries is how a system can be loyal to the interests of
multiple, potentially conflicting principals. A natural strategy is to
aggregate the many objective functions assessed for each principle into a
single objective function (Noothigattu et al., 2018). However, this strategy
invites criticism from the field of social choice and voting theory, which has
long raised difficulties with preference aggregation. We survey these
critiques and the proposed remedies for social choice in AI, which involved
partially ordered objective functions.

The available legal logic for managing conflicting fiduciaries (Schwarcz,
2009; Gold, 2019) provides additional constraints on the aggregation function
that may simplify the problem. We consider subsidiary duties of impartiality
and the prudent investor rule as examples of how law can guide aggregation.

###  7.1. Impossibility theorems

Social choice, the field that theorizes how to combine many individual
preferences into a collective decision, has revealed many theorems that show
that it is impossible for such mechanisms to meet all of several theoretically
desirable criteria. A prominent example is the Gibbard-Stratherthwaite
theorem(Gibbard, 1973), which states that for any ordinal voting system with
one winner, the rule either:

  * •

is dictatorial, meaning that there is one voter who can choose the winner, or

  * •

limits the outcomes to only two possibilities, or

  * •

is susceptible to manipulation through insincere ballots.

Manipulation becomes a problem when the assessment of best interests is based
on the principals’ own voluntary actions or expressions. These may be untrue
or misleading (Fickinger et al., 2020). Putting aside the question of
incentive compatible mechanism design, we can note that manipulation may be
avoided by decoupling the assessment process from aggregation. In other words,
this is a reason to avoid doing best interest assessment via on-line learning
(Armstrong et al., 2020). More broadly, this and other impossibility theories
make the selection of an aggregation rule a thorny problem. Baum (2020) argues
that such aggregation rules are necessarily moral choices of the designer of a
social AI. Aggregating the best interests of multiple principals of a
Fiduciary AI is a structurally similar task.

Other solutions to voting paradoxes involve a relaxation of requirement that
preferences or objectives be given as a total ordering. Prasad (2018) proposes
that for the highest hierarchical levels of decision-making in AI design, it
is best to use an approval voting mechanism, which gives each voter a binary
choice of consent for every option, rather than an ordinal preference. This
form of voting is less prone to manipulation. It takes as its goal “consent
maximization”, as opposed to “utility maximization”. This is not to be
confused with the initial consent to use the fiduciary service, but rather is
meant to be a heuristic for aggregating the interests of those already
identified as principals. Eckersley (2018) addresses the impossibility
theorems of Arrhenius (2000) and (Parfit, 1984) in population ethics which
show that there is no way to develop a totally ordered objective function over
outcomes for a population without violating human ethical intuitions in one of
several ways. This problem can be avoided if the aggregate objective function
is a partially ordered, as opposed to totally ordered. Indeed, perhaps the
most straightforward way to aggregate individual objective functions is using
multi-objective optimization techniques that reveal Pareto efficient
indifference curves over individual welfare. (Marler and Arora, 2004;
Gunantara, 2018)

In sum, the design of the aggregation function is an important choice with
ethical implications. The relaxation of total ordering conditions eases some
of the design burden. For the remainder of this article, we assume that
objective and utility functions may be partially ordered.

###  7.2. Legal constraints

The problem of conflicting principal interests has arisen for non-
computational fiduciaries, and in some cases it has been settled by further
legal rules. These tend to be context-specific subsidiary duties (see Section
2.1.3). We consider two examples here which may be guides to how subsidiary
duties could inform Fiduciary AI in the future.

In some contexts where there are potentially conflicting principals, the duty
of loyalty can include a subsidiary _duty of impartiality_. As discussed in
Section 2.1.5, impartiality requires that the system not be unduly influenced
by either self-interest (of the agent) or favoritism (towards one of the
principals). This constrains the agent, but the duty of impartiality also
gives the agent the flexibility to act with discretion within those
constraints. Just as corporate directors can make decisions that unequally
impact different classes of shareholders (Nir, 2020), an information fiduciary
could make decisions that unequally impact different classes of users without
violating their duties. As a further guide to how to balance principal
interests, the agent should look to the specific purpose for which the
relationship with the beneficiary was established.202020This resolution to the
multi-principal system design runs parallel to the way CI theorizes the
formation of information norms: as best fit to a balance of the role-based
ends of actors and the purpose of the social context. So in these cases, the
agent’s aggregation problem is constrained in some ways, but relaxed in
others.212121The Trustworthy AI dimensions of fairness and privacy are both
relevant to aggregation in ways that are not currently considered in fiduciary
law. In privacy scholarship, there is a growing recognition of group privacy
(Taylor et al., 2016; Mittelstadt, 2017), when a group of people have a
collective stake in their data, perhaps because of horizontal data effects
(Viljoen, 2021), meaning when the data collected from one subject has
implications for the interest of other subjects. There is an opportunity when
designing the aggregation function to acknowledge these externalities or
relational concerns and balance the weighting of preferences accordingly.
Particularly well-studied in the Trustworthy AI context are fairness criteria.
These fairness metrics are primarily aimed at preventing group-based
discrimination in machine learning systems. However, a Fiduciary AI designer
concerned with compliance with nondiscriminatory regulations or social
expectations could perhaps adapt these fairness criteria to the aggregation
function as well. We pose this as a problem as one for future work.

Other legal constraints can more directly specify the agent’s aggregation
function. Consider again the prudent investor rule (see Section 2.1.4), which
defines principal’s interests as those identified by modern portfolio theory.
This ruling irons over what might otherwise be wrinkles due to idiosyncratic
principal preferences, because those preferences may not be prudent. So, for
example, if one principal wanted their trustee to divest from certain
businesses for political reasons, it would not necessarily be the trustee’s
duty to do so. An information fiduciary statute could likewise construct a
standard of user interests based on a combination of a strict definition of
context and scientific results.

##  8\. Loyalty and Alignment

Loyalty is one of the two pillars of fiduciary duty. Following Richards and
Hartzog (2020), we focus on a “best interests” interpretation of loyalty for
Fiduciary AI because instructions given to an AI will be incomplete, leaving
open questions of its behavior (Hadfield-Menell and Hadfield, 2019). We now
assume that through the proceeding steps of the Fiduciary AI procedure, the
designer has in hand an objective function (perhaps only partially ordered)
that represents the assessed and aggregated best interests of the principals.
In this section, we discuss guarantees of loyalty in more depth. In
particular, we note the legal idea of the “loyalty two-step”: the combination
of a general duty of alignment, and contextually specific subsidiary rules
with clearer requirements (Hartzog and Richards, 2022). The general duty of
loyalty bears some resemblance to the broader issue of AI Alignment, and can
be characterized by optimization of a proxy objective function. Subsidiary
duties of loyalty can provide firmer constraints on system behavior, and
address problems that are difficult to solve using machine learning.

###  8.1. Best interest and value alignment

The first step of the “loyalty two-step” is a “no-conflict” rule in the system
design (Hartzog and Richards, 2022). This means that the system must not be
designed to conflict with the best interests of the principals. The designer
must use their representation fo the best interests of the principals in good
faith.

When the behavior of the system is an AI, the system can be trained to
optimize this proxy objective function. In the reinforcement learning context,
for example, the objective function can be used as the reward function with
which the agent’s policy is trained (Leike et al., 2018). If there are
hierarchical fiduciary duties between multiple classes of principals,
lexicographic (Stadler, 1988) or hierarchical (Osyczka, 1984) methods in
multi-objective optimization can be used to maintain this prioritization.

In the broader research program of AI Alignment (Dewey, 2011; Hadfield-Menell
et al., 2016; Everitt et al., 2021b), it is understood that alignment through
proxy objective functions can fail due to errors. Leike et al. (2017)
distinguish between the _specification_ problem – when the proxy function is
different from the ’true’ principal objectives – and the _robustness_ problem
– when the proxy function is accurate, but other problems in the
implementation of the system lead to misaligned behavior. The requirements of
specification are uncompromising; Zhuang and Hadfield-Menell (2020)
demonstrate that incompleteness – under-specification – of the proxy objective
function can lead to arbitrarily costly behavior of the agent. Robustness
problems arise not from misalignment but from other forms of error in system
calibration, such as distributional shift. In Section 9, we suggest that
mediating these problems be understood as a requirement of the duty of care.

Aguirre et al. (2020) and others have drawn a connection between the alignment
implied by a duty of loyalty and what is studied more broadly as AGI
Alignment. We maintain that Fiduciary AI is sufficient, but not necessary, for
AI Alignment. Fiduciary AI is a stricter demand than mere alignment because of
the constraints from contextual scope. For example, when _data minimization_ –
the requirement that data be collected and stored only when necessary to
pursue legitimate purposes – is a subsidiary duty of loyalty, this limits the
power of an AI to act beyond its intended design. An AGI that is well designed
as a Fiduciary AI would, as a matter of its own duties, limit the
vulnerability of its principals by performing only within its narrowly defined
role (Hartzog and Richards, 2022).222222Perhaps there is one exception. An AI
that served as a corporate director, with fiduciary duties to its
shareholders, may require general intelligence to operate many aspects of the
corporation, even though their duties are narrowly construed in terms of
protecting the value of the shareholder’s investment in the corporation.
Indeed, this sweeping scope of AI power, combined with the narrowness of its
objective function, makes this form of AI particularly risky (Russell, 2019;
Benthall and Goldenfein, 2021).

###  8.2. Subsidiary duties of loyalty

When a designer schematizes the context of a Fiduciary AI system, they should
enumerate any context-specific norms and rules (Section 4). These include
subsidiary duties of loyalty, the second part of the “loyalty two-step”, and
can include other TAI standards such as: data minimization, nondiscriminatory
targeting, reducing data sharing with third parties, absence of dark patterns,
and content moderation (Hartzog and Richards, 2022). (See Table 1.)

Some subsidiary duties may reference the best interests of the principals, but
in more nuanced ways than broad “alignment”. For example, the _duty of
disclosure_ is a positive rule mandating that the fiduciary reveal relevant
facts to the beneficiary, especially if these are facts pertaining to a
potential conflict of interest. This duty creates an opportunity for the
principal to contest an action that is in potential conflict. 232323Aguirre et
al. (2020) argue that, even in absence of stronger fiduciary duties, for AI
the presence of conflicts in design must be “transparently and saliently
indicate[d] to users”, but not forbidden outright. This amounts to extending
the duty of disclosure to all AI systems with users. Fiduciary AI designers
might engage such a duty when, for example, a system identifies that its
objective function is incomplete with respect to its current decision or
situation (Zhuang and Hadfield-Menell, 2020), and more information from the
principals are needed to guarantee a lack of conflict. Inversely, the duty of
confidentiality is a negative rule prohibiting the disclosure of information
(typically, about the beneficiary) for the benefit of the fiduciary. This
subsidiary duty involves representations of both the principal’s interests _as
well as_ the agent’s “true” interests, such as the interests of its
shareholders.

Notably, these subsidiary duties govern flows of information to and from the
agent. They can be understood as transmission principles — norms governing
flows of information as per Contextual Integrity(Nissenbaum, 2020). Formal
work on the strategic value of information and its import for AI alignment has
been pursued by Everitt et al. (2021a).

##  9\. Care

Some jurisdictions and contexts define a fiduciary duty of care. The duty of
care obliges the agent to prevent any foreseeable harms to the principal. It
specifically holds the agent liable for accidental harms to a higher degree
than they would be held otherwise under tort law. Whereas loyalty requires the
agent to make decisions in the best interest of the principal, care requires
the agent to be _highly informed_ before making decisions, so as not to make a
decision recklessly.

Fiduciary agents are held to a standard of prudence that is defined
contextually. For example, the executor of a trust is held to the prudent
investor rule, which assumes expertise in investing, and is a higher standard
than the more general reasonable person standard used in general tort cases.
This is contextually informed level of responsibility becomes a field-specific
subsidiary duty of care.

AI Safety researchers have identified many potential AI errors that could
cause accidental harms(Amodei et al., 2016), including:

  * •

negative side effects, when an agent disturbs its environment in negative ways
not accounted for in its reward function (Armstrong and Levinstein, 2017;
Krakovna et al., 2018)

  * •

reward hacking, when an agent finds a way to optimize its reward in an
unexpected and unintended way (Yuan et al., 2019; Gilbert et al., 2022)

  * •

distributional shift, when the agent is trained in an environment which does
not generalize to the environment in which it operates (Kirschner et al.,
2020; Krueger et al., 2020)

A full discussion of these errors is beyond the scope of this paper. Here, we
note that the fiduciary duty of care is a legal requirement that may include
remediation of any or all of these problems as the context-appropriate
standard of prudence evolves to take them into account.

For Fiduciary AI, the duty of care may be violated if the designer’s judgment
was in bad faith or not prudently informed of the facts.242424In analogous
duty of care cases, corporate directors are only held to the standard of
making the decision with a good faith business judgement, even if their
judgement was _(_ ex post) incorrect. The burden is on the plaintiff to show
bad faith or impropriety on the part of the director. This is called the
_business judgement rule_. For example, a duty of care for Fiduciary AI might
require that the AI designer has considered all potential harms caused by
their system. For example, consider what happens if a hypothetical social
media service, _Zmeta_ , is tried in court for violation of fiduciary duties
after it is found to be addictive and harmful to its users. The company’s
lawyers argue that engagement is a signal of user interest, and so the company
was acting in good faith. A court might consider a _prudent user rule_
informed by research about addiction and well-being (such as (Tromholt, 2016;
Polites et al., 2018; Kuem et al., 2020)). Under such a rule, using engagement
as a proxy for user best interest would be considered negligent, because it
falls below the standard of care appropriate for the context.

Several reporting or labeling frameworks have been proposed for guiding due
diligence and accountability for machine learning based on the data set they
use for training (Gebru et al., 2021), conformity with testing standards
(Arnold et al., 2019), use of the model only for its intended purposes and
with testing for fairness on varying demographic groups (Mitchell et al.,
2019), and for overall design of the reward system (Gilbert et al., 2022).
This can be framed as an improvement to AI Safety due to the reduction of
epistemic uncertainty (Möller, 2012; Varshney, 2016; Varshney and Alemzadeh,
2017), which is uncertainty about which what can be known in principle but has
not been discovered in practice. These standards are likely to evolve over
time as AI, and society’s understanding of it, matures.

###  9.1. Inductive bias

While Fiduciary AI duties apply to everything about a system, from design to
deployment, some aspects of it are especially prone to negligence. Whereas the
choice of training data and benchmarking of a model are often deliberate
choices, the inductive bias of a machine learning system can be an
afterthought, determined by defaults. This makes attention to inductive bias a
good example of what is addressed by the duty of care.

We illustrate this simply in Bayesian terms. For a particular application of
the system that uses the model trained on a data set $D$ to determine the
outcome for a particular case $h$. For simplicity, let $h$ be a binary
decision, such as whether or not to hire an individual. The closer that the
likelihood ratio $\frac{P(D|h=1)}{P(D|h=0}$ is to 1, the less the available
training data has informed the decision, and the more the inductive bias
$P(h)$ dominates. This can happen when there is insufficient data in the
region that is informative for the case of $h$, such as if the person $h$ is a
rare minority among a larger population. A duty of care for Fiduciary AI would
standardize the inductive bias of systems based on their context and purpose
of use. Notably, choices about inductive bias must be made if reward modeling
is used to assess the best interests of the principal (see Section 6.1.2). Xu
et al. (2019) show how training on other tasks can be used to inform the prior
used in IRL on a new task. Hence, a duty of care might standardize a prior
over rewards directly, but it might also standardize a corpus of analogous
tasks used to train that prior.

##  10\. Discussion

Lawmakers have struggled to find a way to regulate artificial intelligence
directly. At the heart of the problem is the incompleteness with which
principals can come to agreement with an AI system, whether through explicit
direction or consent (Hadfield-Menell and Hadfield, 2019). This means that for
the foreseeable future, AI will only be as ethical as the purposes of the
social actor that operates it (Benthall and Goldenfein, 2021). Fiduciary
duties are a time-tested legal means for establishing the trustworthiness of
an incompletely contracted agent by aligning its purposes with a principal
(Easterbrook and Fischel, 1993). Fiduciary duties for computational systems
are part of the law today, and may become more broadly applied with new
legislation in the future (Balkin, 2015; Richards and Hartzog, 2020)

This article has aimed to inform system designers about the technical
requirements implied by fiduciary duties. To this end, we have outlined the
legal rationale for fiduciary duties, and in particular their application to
computational systems and AI. We have then provided a guide for how a
Fiduciary AI system can be designed or audited for compliance with these
rules. This six-step process involves: understanding the context of the
system; identifying the principals; assessing the best interests of the
principals; aggregating those best interests; ensuring the system is loyal to
those best interests; and observing a contextually appropriate standard of
care with respect to unlearned aspects of the system, such as inductive bias.

We have identified dimensions of Trustworthy AI that inform best practices in
each of these stages. Notably, we see Fiduciary AI practice as drawing on both
Contextual Integrity (Nissenbaum, 2020) and AI Alignment (Yudkowsky, 2016)
research in an original way, while also incorporating other dimensions of
Trustworthy AI. Though Fiduciary AI functions may one day be performed by a
general AI system, its duties are scoped to a particular legal context in
order to prevent conflicts of interest with the principals and risk more
generally. The legal structure of fiduciary duties – abstract primary duties
and more contextually specific subsidiary duties – is an important feature of
Fiduciary AI, as in many cases legal constraints can make complex AI design
challenges more tractable or straightforward.

There are many open research questions raised by the application of fiduciary
principles to computational systems. We leave as an open question for future
work how human-centered and participatory design methods (Martin Jr et al.,
2020; Katell et al., 2020; Birhane et al., 2022; Sloane et al., 2022) might,
or might not, be used to assess the best interests of the principals of an
information fiduciary. We also wonder how the narrowly legal duty of care
discussed in this paper compares with other notions of care in AI (Knowles et
al., 2023) and digital infrastructure (Tseng et al., 2022) design. Our focus
on AI and legal methods, to the exclusion of more humanistic methods, is a
limitation of this article.

Fiduciary AI standards will necessarily evolve with the maturation of policy,
technology, and the understanding of the courts. It is pragmatic to take stock
of the state of the art in legal and technical research at this stage so that
different research communities can find common ground in Fiduciary AI
research, and demonstrate the feasibility of policy positions that anticipate
future iterations of AI research and practice. Such policies may have far-
reaching implications for value alignment of more powerful AI to come (Soares
and Fallenstein, 2014; Aguirre et al., 2020).

###### Acknowledgements.

We gratefully acknowledge the helpful comments of Andrew Critch, Francesca
Episcopo, Katrina Geddes, Jiaying Jiang, Tomer Kenneth, Aniket Kesari, Galen
Panger, Ira Rubenstein, Katherine Strandburg, Thomas Streinz, Emily Tseng,
Kush Varshney, Mark Verstraete, Stav Zeitouni. We also are grateful for the
feedback from participants in the 3rd Annual Symposium for Applications of
Contextual Integrity, Center for Human-Compatible Artificial Intelligence
Beneficial AI Seminar, Northeast Privacy Scholars Workshop, and the
Information Law Institute and Privacy Research Group at New York University
School of Law. This material is based upon work supported by the National
Science Foundation under Grant No. 2105301. Any opinions, findings, and
conclusions or recommendations expressed in this paper are those of the
authors and do not necessarily reflect the views of the sponsors. One of the
authors of this article was supported by the New York University Information
Law Institute’s Fellows program, which is funded in part by Microsoft
Corporation.

## References

  * (1)
  * tri ([n. d.) ]trilegal [n. d.].  Data Protection Bill 2021.  https://www.trilegal.com/index.php/publications/update/data-protection-bill-2021
  * lie ([n. d.) ]lieu2022opted [n. d.].  Opted Out, Yet Tracked: Are Regulations Enough to Protect Your Privacy?
  * Abbeel and Ng (2004) Pieter Abbeel and Andrew Y Ng. 2004.  Apprenticeship learning via inverse reinforcement learning. In _Proceedings of the twenty-first international conference on Machine learning_. 1.
  * Abebe et al. (2020) Rediet Abebe, Solon Barocas, Jon Kleinberg, Karen Levy, Manish Raghavan, and David G Robinson. 2020\.  Roles for computing in social change. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_. 252–260.
  * Ackerman et al. (2001) Mark Ackerman, Trevor Darrell, and Daniel J Weitzner. 2001\.  Privacy in context.  _Human–Computer Interaction_ 16, 2-4 (2001), 167–176.
  * Acquisti and Grossklags (2003) Alessandro Acquisti and Jens Grossklags. 2003.  Losses, gains, and hyperbolic discounting: An experimental approach to information security attitudes and behavior. In _2nd Annual Workshop on Economics and Information Security-WEIS_ , Vol. 3. Citeseer, 1–27.
  * Aguirre et al. (2020) Anthony Aguirre, Gaia Dempsey, Harry Surden, and Peter B Reiner. 2020.  AI loyalty: a new paradigm for aligning stakeholder interests.  _IEEE Transactions on Technology and Society_ 1, 3 (2020), 128–137.
  * Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016\.  Concrete problems in AI safety.  _arXiv preprint arXiv:1606.06565_ (2016).
  * Armstrong et al. (2020) Stuart Armstrong, Jan Leike, Laurent Orseau, and Shane Legg. 2020.  Pitfalls of learning a reward function online.  _arXiv preprint arXiv:2004.13654_ (2020).
  * Armstrong and Levinstein (2017) Stuart Armstrong and Benjamin Levinstein. 2017.  Low impact artificial intelligences.  _arXiv preprint arXiv:1705.10720_ (2017).
  * Armstrong and Mindermann (2018) Stuart Armstrong and Sören Mindermann. 2018.  Occam’s razor is insufficient to infer the preferences of irrational agents.  _Advances in neural information processing systems_ 31 (2018).
  * Arnold et al. (2019) Matthew Arnold, Rachel KE Bellamy, Michael Hind, Stephanie Houde, Sameep Mehta, Aleksandra Mojsilović, Ravi Nair, K Natesan Ramamurthy, Alexandra Olteanu, David Piorkowski, et al. 2019\.  FactSheets: Increasing trust in AI services through supplier’s declarations of conformity.  _IBM Journal of Research and Development_ 63, 4/5 (2019), 6–1.
  * Arrhenius (2000) Gustaf Arrhenius. 2000\.  An impossibility theorem for welfarist axiologies.  _Economics & Philosophy_ 16, 2 (2000), 247–266.
  * Bahdanau et al. (2018) Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. 2018.  Learning to understand goal specifications by modelling reward.  _arXiv preprint arXiv:1806.01946_ (2018).
  * Balkin (2015) Jack M Balkin. 2015\.  Information fiduciaries and the first amendment.  _UCDL Rev._ 49 (2015), 1183.
  * Balkin (2020) Jack M Balkin. 2020\.  The Fiduciary Model of Privacy.  _Harv. L. Rev. F._ 134 (2020), 11.
  * Baum (2020) Seth D Baum. 2020\.  Social choice ethics in artificial intelligence.  _AI & SOCIETY_ 35, 1 (2020), 165–176.
  * Benthall and Goldenfein (2021) Sebastian Benthall and Jake Goldenfein. 2021.  Artificial Intelligence and the Purpose of Social Systems. In _Proceedings of the 2021 AAAI/ACM Conference on AI Ethics and Society (AIES’21)_.
  * Benthall et al. (2017) Sebastian Benthall, Seda Gürses, Helen Nissenbaum, et al. 2017\.  _Contextual integrity through the lens of computer science_.  Now Publishers.
  * Benzion et al. (1989) Uri Benzion, Amnon Rapoport, and Joseph Yagil. 1989\.  Discount rates inferred from decisions: An experimental study.  _Management science_ 35, 3 (1989), 270–284.
  * Birhane et al. (2022) Abeba Birhane, William Isaac, Vinodkumar Prabhakaran, Mark Díaz, Madeleine Clare Elish, Iason Gabriel, and Shakir Mohamed. 2022\.  Power to the People? Opportunities and Challenges for Participatory AI.  _Equity and Access in Algorithms, Mechanisms, and Optimization_ (2022), 1–8.
  * Bongard-Blanchy et al. (2021) Kerstin Bongard-Blanchy, Arianna Rossi, Salvador Rivas, Sophie Doublet, Vincent Koenig, and Gabriele Lenzini. 2021.  ” I am Definitely Manipulated, Even When I am Aware of it. It’s Ridiculous!”-Dark Patterns from the End-User Perspective. In _Designing Interactive Systems Conference 2021_. 763–776.
  * Brooks (2019) Richard RW Brooks. 2019\.  The Economics of Fiduciary Law.  In _The Oxford Handbook of Fiduciary Law_.
  * Burr et al. (2020) Christopher Burr, Mariarosaria Taddeo, and Luciano Floridi. 2020.  The ethics of digital well-being: a thematic review.  _Science and engineering ethics_ (2020), 1–31.
  * Calo (2013) Ryan Calo. 2013\.  Digital market manipulation.  _Geo. Wash. L. Rev._ 82 (2013), 995.
  * Carroll et al. (2017) Christopher Carroll, Jiri Slacalek, Kiichi Tokuoka, and Matthew N White. 2017.  The distribution of wealth and the marginal propensity to consume.  _Quantitative Economics_ 8, 3 (2017), 977–1020.
  * Chabris et al. (2008) Christopher F Chabris, David Laibson, Carrie L Morris, Jonathon P Schuldt, and Dmitry Taubinsky. 2008.  Individual laboratory-measured discount rates predict field behavior.  _Journal of risk and uncertainty_ 37, 2 (2008), 237–269.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017\.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ 30 (2017).
  * Cohen (2016) Julie E Cohen. 2016\.  The regulatory state in the information age.  _Theoretical Inquiries in Law_ 17, 2 (2016), 369–414.
  * Committee et al. (2020) IEEE Standards Committee et al. 2020\.  _IEEE Recommended Practice for Assessing the Impact of Autonomous and Intelligent Systems on Human Well-Being: IEEE Standard 7010-2020_.  IEEE.
  * Cusumano et al. (2019) Michael A Cusumano, Annabelle Gawer, and David B Yoffie. 2019\.  _The business of platforms: Strategy in the age of digital competition, innovation, and power_.  Harper Business New York.
  * Datta et al. (2011) Anupam Datta, Jeremiah Blocki, Nicolas Christin, Henry DeYoung, Deepak Garg, Limin Jia, Dilsun Kaynar, and Arunesh Sinha. 2011\.  Understanding and protecting privacy: Formal semantics and principled audit mechanisms. In _International Conference on Information Systems Security_. Springer, 1–27.
  * Dewey (2011) Daniel Dewey. 2011\.  Learning what to value. In _International Conference on Artificial General Intelligence_. Springer, 309–314.
  * Dey et al. (2001) Anind K Dey, Gregory D Abowd, and Daniel Salber. 2001\.  A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications.  _Human–Computer Interaction_ 16, 2-4 (2001), 97–166.
  * Diener et al. (2009) Ed Diener, Ed Sandvik, and William Pavot. 2009.  Happiness is the frequency, not the intensity, of positive versus negative affect.  In _Assessing well-being_. Springer, 213–231.
  * Dourish (2004) Paul Dourish. 2004\.  What we talk about when we talk about context.  _Personal and ubiquitous computing_ 8, 1 (2004), 19–30.
  * Easterbrook and Fischel (1993) Frank H Easterbrook and Daniel R Fischel. 1993.  Contract and fiduciary duty.  _The Journal of law and economics_ 36, 1, Part 2 (1993), 425–446.
  * Eckersley (2018) Peter Eckersley. 2018\.  Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function).  _arXiv preprint arXiv:1901.00064_ (2018).
  * Egelman and Peer (2015) Serge Egelman and Eyal Peer. 2015.  The myth of the average user: Improving privacy and security systems through individualization. In _Proceedings of the 2015 New Security Paradigms Workshop_. 16–28.
  * Europe (2020) IAB Europe. 2020\.  IAB Europe Transparency & Consent Framework Policies.  https://iabeurope.eu/wp-content/uploads/2020/11/TCFv2-0Policyversion2020-11-18-3.2a.docx-1.pdf
  * Everitt et al. (2021a) Tom Everitt, Ryan Carey, Eric Langlois, Pedro A Ortega, and Shane Legg. 2021a.  Agent incentives: A causal perspective. In _Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence,(AAAI-21). Virtual. Forthcoming_.
  * Everitt et al. (2021b) Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. 2021b.  Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective.  _Synthese_ 198, 27 (2021), 6435–6467.
  * Fickinger et al. (2020) Arnaud Fickinger, Simon Zhuang, Dylan Hadfield-Menell, and Stuart Russell. 2020.  Multi-principal assistance games.  _arXiv preprint arXiv:2007.09540_ (2020).
  * Frankel (2011) Tamar Frankel. 2011\.  Fiduciary law in the twenty-first century.  _BUL Rev._ 91 (2011), 1289.
  * Friedman et al. (2000) Batya Friedman, Edward Felten, and Lynette I Millett. 2000\.  Informed consent online: A conceptual model and design principles.  _University of Washington Computer Science & Engineering Technical Report 00–12–2_ 8 (2000).
  * Fu et al. (2019) Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. 2019.  From language to goals: Inverse reinforcement learning for vision-based instruction following. In _Proceedings of the 2019 International Conference on Learning Representations (ICLR)_.
  * Gebru et al. (2021) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. 2021\.  Datasheets for datasets.  _Commun. ACM_ 64, 12 (2021), 86–92.
  * Gelter and Helleringer (2019) Martin Gelter and Geneviève Helleringer. 2019.  Fiduciary Principles in European Civil Systems.  In _The Oxford Handbook of Fiduciary Law_.
  * Gibbard (1973) Allan Gibbard. 1973\.  Manipulation of voting schemes: a general result.  _Econometrica: journal of the Econometric Society_ (1973), 587–601.
  * Gilbert et al. (2022) Thomas Krendl Gilbert, Sarah Dean, Tom Zick, and Nathan Lambert. 2022.  Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems.  _arXiv preprint arXiv:2202.05716_ (2022).
  * Gleave et al. (2020) Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. 2020.  Quantifying differences in reward functions.  _arXiv preprint arXiv:2006.13900_ (2020).
  * Gold (2019) Andrew Gold. 2019\.  The Fiduciary Duty of Loyalty.  In _The Oxford Handbook of Fiduciary Law_.
  * Graham et al. (2018) Carol Graham, Kate Laffan, and Sergio Pinto. 2018\.  Well-being in metrics and policy.  _Science_ 362, 6412 (2018), 287–288.
  * Green and Myerson (2004) Leonard Green and Joel Myerson. 2004.  A discounting framework for choice with delayed and probabilistic rewards.  _Psychological bulletin_ 130, 5 (2004), 769\.
  * Griffith et al. (2013) Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. 2013\.  Policy shaping: Integrating human feedback with reinforcement learning.  _Advances in neural information processing systems_ 26 (2013).
  * Grimmelmann (2019) James Grimmelmann. 2019\.  When All You Have Is A Fiduciary.  https://lpeproject.org/blog/when-all-you-have-is-a-fiduciary/
  * Gunantara (2018) Nyoman Gunantara. 2018\.  A review of multi-objective optimization: Methods and its applications.  _Cogent Engineering_ 5, 1 (2018), 1502242.
  * Hadfield-Menell and Hadfield (2019) Dylan Hadfield-Menell and Gillian K Hadfield. 2019.  Incomplete contracting and AI alignment. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. 417–422.
  * Hadfield-Menell et al. (2017) Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. 2017\.  Inverse reward design.  _Advances in neural information processing systems_ 30 (2017).
  * Hadfield-Menell et al. (2016) Dylan Hadfield-Menell, Stuart J Russell, Pieter Abbeel, and Anca Dragan. 2016.  Cooperative inverse reinforcement learning.  _Advances in neural information processing systems_ 29 (2016), 3909–3917.
  * Hartzog and Richards (2022) Woodrow Hartzog and Neil M. Richards. 2022.  Legislating Data Loyalty.  _Notre Dame Law Review Reflection_ 97 (2022), 365.
  * Hils et al. (2020) Maximilian Hils, Daniel W Woods, and Rainer Böhme. 2020\.  Measuring the emergence of consent management on the web. In _Proceedings of the ACM Internet Measurement Conference_. 317–332.
  * Ibarz et al. (2018) Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. 2018\.  Reward learning from human preferences and demonstrations in atari.  _Advances in neural information processing systems_ 31 (2018).
  * Jensen and Meckling (1976) Michael Jensen and William Meckling. 1976.  Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure.  _J. Fin. Econ._ 3 (1976), 305–360.
  * Jeon et al. (2020) Hong Jun Jeon, Smitha Milli, and Anca Dragan. 2020\.  Reward-rational (implicit) choice: A unifying formalism for reward learning.  _Advances in Neural Information Processing Systems_ 33 (2020), 4415–4426.
  * Kahneman and Krueger (2006) Daniel Kahneman and Alan B Krueger. 2006.  Developments in the measurement of subjective well-being.  _Journal of Economic perspectives_ 20, 1 (2006), 3–24.
  * Katell et al. (2020) Michael Katell, Meg Young, Dharma Dailey, Bernease Herman, Vivian Guetler, Aaron Tam, Corinne Bintz, Daniella Raz, and PM Krafft. 2020.  Toward situated interventions for algorithmic equity: lessons from the field. In _Proceedings of the 2020 conference on fairness, accountability, and transparency_. 45–55.
  * Kelly (2019) Daniel B. Kelly. 2019\.  Fiduciary Principles in Fact-Based Fiduciary Relationships.  In _The Oxford Handbook of Fiduciary Law_. Oxford Handbooks.
  * Khan and Pozen (2019) Lina M Khan and David E Pozen. 2019.  A skeptical view of information fiduciaries.  _Harv. L. Rev._ 133 (2019), 497.
  * Kirschner et al. (2020) Johannes Kirschner, Ilija Bogunovic, Stefanie Jegelka, and Andreas Krause. 2020.  Distributionally robust Bayesian optimization. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 2174–2184.
  * Knowles et al. (2023) Bran Knowles, Jasmine Fledderjohan, John T. Richards, and Kush R. Varshney. 2023.  Trustworthy AI and the Logics of Intersectional Resistance.
  * Knox and Stone (2010) W Bradley Knox and Peter Stone. 2010.  Combining manual feedback with subsequent MDP reward signals for reinforcement learning. In _Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1_. Citeseer, 5–12.
  * Krakovna et al. (2018) Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. 2018.  Penalizing side effects using stepwise relative reachability.  _arXiv preprint arXiv:1806.01186_ (2018).
  * Kross et al. (2013) Ethan Kross, Philippe Verduyn, Emre Demiralp, Jiyoung Park, David Seungjae Lee, Natalie Lin, Holly Shablack, John Jonides, and Oscar Ybarra. 2013.  Facebook use predicts declines in subjective well-being in young adults.  _PloS one_ 8, 8 (2013), e69841.
  * Krueger et al. (2020) David Krueger, Tegan Maharaj, and Jan Leike. 2020\.  Hidden incentives for auto-induced distributional shift.  _arXiv preprint arXiv:2009.09153_ (2020).
  * Krusell and Smith (1998) Per Krusell and Anthony A Smith, Jr. 1998.  Income and wealth heterogeneity in the macroeconomy.  _Journal of political Economy_ 106, 5 (1998), 867–896.
  * Kuem et al. (2020) Jungwon Kuem, Soumya Ray, Pei-Fang Hsu, and Lara Khansa. 2020\.  Smartphone addiction and conflict: An incentive-sensitisation perspective of addiction for Information Systems.  _European Journal of Information Systems_ (2020), 1–22.
  * Laibson (1997) David Laibson. 1997\.  Golden eggs and hyperbolic discounting.  _The Quarterly Journal of Economics_ 112, 2 (1997), 443–478.
  * Leike et al. (2018) Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018\.  Scalable agent alignment via reward modeling: a research direction.  _arXiv preprint arXiv:1811.07871_ (2018).
  * Leike et al. (2017) Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. 2017\.  AI safety gridworlds.  _arXiv preprint arXiv:1711.09883_ (2017).
  * Linton et al. (2016) Myles-Jay Linton, Paul Dieppe, and Antonieta Medina-Lara. 2016\.  Review of 99 self-report measures for assessing well-being in adults: exploring dimensions of well-being and developments over time.  _BMJ open_ 6, 7 (2016), e010641.
  * Mandhani (2021) Apoorva Mandhani. 2021\.  Non-personal data, social media - what new ’data protection bill’ could look like.  https://theprint.in/theprint-essential/non-personal-data-social-media-what-new-data-protection-bill-could-look-like/776389/
  * Markowitz (1968) Harry M Markowitz. 1968\.  _Portfolio selection_.  Yale university press.
  * Marler and Arora (2004) R Timothy Marler and Jasbir S Arora. 2004.  Survey of multi-objective optimization methods for engineering.  _Structural and multidisciplinary optimization_ 26, 6 (2004), 369–395.
  * Martin Jr et al. (2020) Donald Martin Jr, Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S Isaac. 2020\.  Participatory problem formulation for fairer machine learning through community based system dynamics.  _arXiv preprint arXiv:2005.07572_ (2020).
  * Marwick et al. (2018) Alice E Marwick et al. 2018\.  Privacy at the margins— understanding privacy at the margins—introduction.  _International Journal of Communication_ 12 (2018), 9.
  * McDonald and Cranor (2008) Aleecia M McDonald and Lorrie Faith Cranor. 2008.  The cost of reading privacy policies.  _Isjlp_ 4 (2008), 543.
  * Miller (2019) Paul B. Miller. 2019\.  The Identification of Fiduciary Relationships.  In _The Oxford Handbook of Fiduciary Law_. Oxford Handbooks.
  * Mitchell et al. (2019) Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019.  Model cards for model reporting. In _Proceedings of the conference on fairness, accountability, and transparency_. 220–229.
  * Mittelstadt (2017) Brent Mittelstadt. 2017\.  From individual to group privacy in big data analytics.  _Philosophy & Technology_ 30, 4 (2017), 475–494.
  * Möller (2012) Niklas Möller. 2012\.  The concepts of risk and safety.  _Handbook of risk theory: epistemology, decision theory, ethics, and social implications of risk_ 1 (2012), 55–85.
  * Ng et al. (2000) Andrew Y Ng, Stuart J Russell, et al. 2000\.  Algorithms for inverse reinforcement learning.. In _Icml_ , Vol. 1. 2\.
  * Nir (2020) Shachar Nir. 2020\.  One duty to all: the fiduciary duty of impartiality and stockholders’ conflict of interest.  _Hastings Bus. LJ_ 16 (2020), 1.
  * Nissenbaum (2020) Helen Nissenbaum. 2020\.  _Privacy in context_.  Stanford University Press.
  * Noothigattu et al. (2019) Ritesh Noothigattu, Djallel Bouneffouf, Nicholas Mattei, Rachita Chandra, Piyush Madan, Kush R Varshney, Murray Campbell, Moninder Singh, and Francesca Rossi. 2019\.  Teaching AI agents ethical values using reinforcement learning and policy orchestration.  _IBM Journal of Research and Development_ 63, 4/5 (2019), 2–1.
  * Noothigattu et al. (2018) Ritesh Noothigattu, Snehalkumar Gaikwad, Edmond Awad, Sohan Dsouza, Iyad Rahwan, Pradeep Ravikumar, and Ariel Procaccia. 2018.  A voting-based system for ethical decision making. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Nouwens et al. (2020) Midas Nouwens, Ilaria Liccardi, Michael Veale, David Karger, and Lalana Kagal. 2020.  Dark patterns after the GDPR: Scraping consent pop-ups and demonstrating their influence. In _Proceedings of the 2020 CHI conference on human factors in computing systems_. 1–13.
  * Obar and Oeldorf-Hirsch (2020) Jonathan A Obar and Anne Oeldorf-Hirsch. 2020.  The biggest lie on the internet: Ignoring the privacy policies and terms of service policies of social networking services.  _Information, Communication & Society_ 23, 1 (2020), 128–147.
  * O’Donoghue and Rabin (2001) Ted O’Donoghue and Matthew Rabin. 2001.  Choice and procrastination.  _The Quarterly Journal of Economics_ 116, 1 (2001), 121–160.
  * Osyczka (1984) Andrzej Osyczka. 1984\.  _Multicriterion optimisation in engineering_.  Halsted Press.
  * Panger (2018) Galen Panger. 2018\.  People tend to wind down, not up, when they browse social media.  _Proceedings of the ACM on Human-Computer Interaction_ 2, CSCW (2018), 1–29.
  * Parfit (1984) Derek Parfit. 1984\.  _Reasons and persons_.  OUP Oxford.
  * Peysakhovich (2019) Alexander Peysakhovich. 2019\.  Reinforcement learning and inverse reinforcement learning with system 1 and system 2. In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_. 409–415.
  * Polites et al. (2018) Greta L Polites, Christina Serrano, Jason Bennett Thatcher, and Kevin Matthews. 2018.  Understanding social networking site (SNS) identity from a dual systems perspective: an investigation of the dark side of SNS use.  _European Journal of Information Systems_ 27, 5 (2018), 600–621.
  * Prasad (2018) Mahendra Prasad. 2018\.  Social choice and the value alignment problem.  _Artificial intelligence safety and security_ (2018), 291–314.
  * Puaschunder (2021) Julia M Puaschunder. 2021\.  Data Fiduciary in Order to Alleviate Principal–Agent Problems in the Artificial Big Data Age.  In _Information for Efficient Decision Making: Big Data, Blockchain and Relevance_. World Scientific, 41–90.
  * Richards and Hartzog (2020) Neil M Richards and Woodrow Hartzog. 2020.  A Duty of Loyalty for Privacy Law.  _Available at SSRN_ (2020).
  * Romanosky et al. (2006) Sasha Romanosky, Alessandro Acquisti, Jason Hong, Lorrie Faith Cranor, and Batya Friedman. 2006.  Privacy patterns for online interactions. In _Proceedings of the 2006 conference on Pattern languages of programs_. 1–9.
  * Rothkopf and Dimitrakakis (2011) Constantin A Rothkopf and Christos Dimitrakakis. 2011.  Preference elicitation and inverse reinforcement learning. In _Joint European conference on machine learning and knowledge discovery in databases_. Springer, 34–48.
  * Russell (2019) Stuart Russell. 2019\.  _Human compatible: Artificial intelligence and the problem of control_.  Penguin.
  * Santos et al. (2021) Cristiana Santos, Midas Nouwens, Michael Toth, Nataliia Bielova, and Vincent Roca. 2021.  Consent Management Platforms under the GDPR: processors and/or controllers?. In _Annual Privacy Forum_. Springer, 47–69.
  * Schwarcz (2009) Steven L Schwarcz. 2009\.  Fiduciaries with Conflicting Obligations.  _Minn. L. Rev._ 94 (2009), 1867.
  * Schwartz and Peifer (2017) Paul M Schwartz and Karl-Nikolaus Peifer. 2017.  Transatlantic data privacy law.  _Geo. LJ_ 106 (2017), 115.
  * Selbst et al. (2019) Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019\.  Fairness and abstraction in sociotechnical systems. In _Proceedings of the conference on fairness, accountability, and transparency_. 59–68.
  * Shah et al. (2019) Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, and Anca Dragan. 2019\.  Preferences implicit in the state of the world.  _arXiv preprint arXiv:1902.04198_ (2019).
  * Sitkoff ([n. d.) ]sitkoffother Robert H Sitkoff. [n. d.].  Other Fiduciary Duties.  In _The Oxford Handbook of Fiduciary Law_.
  * Sitkoff (2011) Robert H Sitkoff. 2011\.  The economic structure of fiduciary law.  _BUL Rev._ 91 (2011), 1039.
  * Sloane et al. (2022) Mona Sloane, Emanuel Moss, Olaitan Awomolo, and Laura Forlano. 2022.  Participation is not a design fix for machine learning.  In _Equity and Access in Algorithms, Mechanisms, and Optimization_. 1–6.
  * Soares and Fallenstein (2014) Nate Soares and Benja Fallenstein. 2014.  Aligning superintelligence with human interests: A technical research agenda.  _Machine Intelligence Research Institute (MIRI) technical report_ 8 (2014).
  * Solove (2021) Daniel J Solove. 2021\.  The myth of the privacy paradox.  _Geo. Wash. L. Rev._ 89 (2021), 1.
  * Squire et al. (2015) Shawn Squire, Stefanie Tellex, Dilip Arumugam, and Lei Yang. 2015\.  Grounding English commands to reward functions. In _Robotics: Science and Systems_.
  * Stadler (1988) Wolfram Stadler. 1988\.  Fundamentals of multicriteria optimization.  In _Multicriteria Optimization in Engineering and in the Sciences_. Springer, 1–25.
  * Streinz (2021) Thomas Streinz. 2021\.  The Evolution of European Data Law.  _The Evolution of EU Law (OUP, 3rd edn 2021)_ (2021), 902–936.
  * Susser et al. (2019) Daniel Susser, Beate Roessler, and Helen Nissenbaum. 2019\.  Technology, autonomy, and manipulation.  _Internet Policy Review_ 8, 2 (2019).
  * Taylor et al. (2016) Linnet Taylor, Luciano Floridi, and Bart Van der Sloot. 2016\.  _Group privacy: New challenges of data technologies_. Vol. 126.  Springer.
  * Thomaz and Breazeal (2008) Andrea L Thomaz and Cynthia Breazeal. 2008.  Teachable robots: Understanding human teaching behavior to build more effective robot learners.  _Artificial Intelligence_ 172, 6-7 (2008), 716–737.
  * Tromholt (2016) Morten Tromholt. 2016\.  The Facebook experiment: Quitting Facebook leads to higher levels of well-being.  _Cyberpsychology, behavior, and social networking_ 19, 11 (2016), 661–666.
  * Tseng et al. (2022) Emily Tseng, Mehrnaz Sabet, Rosanna Bellini, Harkiran Kaur Sodhi, Thomas Ristenpart, and Nicola Dell. 2022.  Care infrastructures for digital security in intimate partner violence. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_. 1–20.
  * Tuch (2021) Andrew Tuch. Forthcoming 2021\.  A General Defense of Information Fiduciaries.  _Wash.U. L. Review_ 98 (Forthcoming 2021).
  * Tversky and Kahneman (1985) Amos Tversky and Daniel Kahneman. 1985.  The framing of decisions and the psychology of choice.  In _Behavioral decision making_. Springer, 25–41.
  * Tversky and Kahneman (1989) Amos Tversky and Daniel Kahneman. 1989.  Rational choice and the framing of decisions.  In _Multiple criteria decision making and risk analysis using microcomputers_. Springer, 81–126.
  * Ulrich (1987) Werner Ulrich. 1987\.  Critical heuristics of social systems design.  _European Journal of Operational Research_ 31, 3 (1987), 276–283.
  * Urban and Hoofnagle (2014) Jennifer M Urban and Chris Jay Hoofnagle. 2014.  The privacy pragmatic as privacy vulnerable. In _Symposium on Usable Privacy and Security (SOUPS 2014) Workshop on Privacy Personas and Segmentation (PPS)_.
  * Varshney (2016) Kush R Varshney. 2016\.  Engineering safety in machine learning. In _2016 Information Theory and Applications Workshop (ITA)_. IEEE, 1–5.
  * Varshney (2019) Kush R Varshney. 2019\.  Trustworthy machine learning and artificial intelligence.  _XRDS: Crossroads, The ACM Magazine for Students_ 25, 3 (2019), 26–29.
  * Varshney and Alemzadeh (2017) Kush R Varshney and Homa Alemzadeh. 2017.  On the safety of machine learning: Cyber-physical systems, decision sciences, and data products.  _Big data_ 5, 3 (2017), 246–255.
  * Viljoen (2021) Salome Viljoen. 2021\.  A relational theory of data governance.  _Yale LJ_ 131 (2021), 573.
  * Viljoen et al. (2021) Salomé Viljoen, Jake Goldenfein, and Lee McGuigan. 2021\.  Design choices: Mechanism design and platform capitalism.  _Big Data & Society_ 8, 2 (2021), 20539517211034312\.
  * Waldman (2019) Ari Ezra Waldman. 2019\.  Privacy Law’s False Promise.  _Wash. UL Rev._ 97 (2019), 773.
  * Waldman (2021) Ari Ezra Waldman. 2021\.  _Industry unbound: The inside story of privacy, data, and corporate Power_.  Cambridge University Press.
  * Walzer (2008) Michael Walzer. 2008\.  _Spheres of justice: A defense of pluralism and equality_.  Basic books.
  * Wing (2021) Jeannette M Wing. 2021\.  Trustworthy AI.  _Commun. ACM_ 64, 10 (2021), 64–71.
  * Xu et al. (2019) Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. 2019.  Learning a prior over intent via meta-inverse reinforcement learning. In _International Conference on Machine Learning_. PMLR, 6952–6962.
  * Yuan et al. (2019) Yinlong Yuan, Zhu Liang Yu, Zhenghui Gu, Xiaoyan Deng, and Yuanqing Li. 2019.  A novel multi-step reinforcement learning method for solving reward hacking.  _Applied Intelligence_ 49, 8 (2019), 2874–2888.
  * Yudkowsky (2016) Eliezer Yudkowsky. 2016\.  The AI alignment problem: why it is hard, and where to start.  _Symbolic Systems Distinguished Speaker_ (2016).
  * Zawojska and Czajkowski (2017) Ewa Zawojska and Mikołaj Czajkowski. 2017.  Re-examining empirical evidence on stated preferences: importance of incentive compatibility.  _Journal of Environmental Economics and Policy_ 6, 4 (2017), 374–403.
  * Zhuang and Hadfield-Menell (2020) Simon Zhuang and Dylan Hadfield-Menell. 2020.  Consequences of misaligned AI.  _Advances in Neural Information Processing Systems_ 33 (2020), 15763–15773.
  * Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. 2008\.  Maximum entropy inverse reinforcement learning.. In _Aaai_ , Vol. 8. Chicago, IL, USA, 1433–1438.