AI Alignment with Changing and Influenceable Reward Functions

Micah Carrollto
Davis Footeto
Anand Siththaranjanto
Stuart Russellto
Anca Draganto

toUC Berkeley

Machine Learning, ICML

Existing AI alignment approaches assume that preferences are static, which is unrealistic: our preferences change, and may even be influenced by our interactions with AI systems themselves.
To clarify the consequences of incorrectly assuming static preferences, we introduce Dynamic Reward Markov Decision Processes (DR-MDPs), which explicitly model preference changes and the AI's influence on them.
We show that despite its convenience, the static-preference assumption may undermine the soundness of existing alignment techniques, leading
them to implicitly reward AI systems for influencing user preferences in ways users may not truly want.
We then explore potential solutions.
First, we offer a unifying perspective on how an agent's optimization horizon may partially help reduce undesirable AI influence. Then, we formalize different notions of AI alignment that account for preference change from the outset.
Comparing the strengths and limitations of 8 such notions of alignment, we find that they all either err towards causing undesirable AI influence, or are overly risk-averse, suggesting that a straightforward solution to the problems of changing preferences may not exist. As there is no avoiding grappling with changing preferences in real-world settings, this makes it all the more important to handle these issues with care, balancing risks and capabilities.
We hope our work can provide conceptual clarity and constitute a first step towards AI alignment practices which explicitly account for (and contend with) the changing and influenceable nature of human preferences.

§ INTRODUCTION
The goal of AI alignment is to make AI systems act according to our preferences (broadly construed). In practice, existing AI alignment techniques model human preferences with a single, static reward function that the AI system is trained to optimize [Leike et al., 2018]. However, our preferences change over time, making it unclear whether AI systems should optimize the satisfaction of our past, present, or future preferences. Consider the following example:

Due to health issues, Alice asks her AI assistant to help her be more healthy, refusing any future requests for unhealthy foods. Sometime later, she later asks the AI to disregard her initial requests, and help her order fast food.

In such a scenario, it may be unclear if the AI assistant should respect Alice's original preference for healthy foods or respect the autonomy of “current Alice”. Ultimately, to align AI systems we must answer the following question: when a person's preferences change over time, which (aggregation of) preferences should AI systems optimize?

While the challenge of aggregating preferences across time shares similarities with that of aggregating preferences across different people [Conitzer et al., 2024], it is further complicated by the fact that AI systems' actions can influence humans and their preferences
[Burtell & Woodside, 2023].
Indeed, as prior work has argued, if AI systems are straightforwardly optimized to satisfy users' future preferences,
they will try to influence them to be easier to satisfy [Russell, 2019, Carroll et al., 2022]. Back to the example:

Alice's AI assistant was trained to maximize her future satisfaction. During training, the AI assistant learned that soothing Alice's health concerns would lead to higher satisfaction than continuously encouraging her to have healthy eating habits. Consequently, to maximize her satisfaction, it's optimal for the AI to ignore her initial wishes and even support her routine unhealthy eating. Indeed, Alice is ultimately truly satisfied.

In this case, the AI assistant seems aligned with “final Alice”, as she endorses the AI's influence towards guilt-free unhealthy eating. However, if “initial Alice” would find this outcome horrifying and see the AI's influence as manipulative, should we still consider the AI aligned?
More broadly, in cases in which people are susceptible to undue influence from AI systems, how can we establish which of their preferences should have authority and legitimacy?

While past work has acknowledged that the static-preference assumption is unrealistic [Franklin et al., 2022], there have only been limited attempts at relaxing it, in part also due to a lack of a clear formal language for grounding notions of alignment which explicitly accounts for preference changes.
We introduce a natural extension of Markov Decision Processes: Dynamic Reward MDPs (DR-MDPs), which account for changing preferences by modeling them as changing reward functions. Importantly, notions of alignment under changing preferences can easily be specified in the form of DR-MDP optimization objectives (<Ref>).

Viewing current alignment practices through the lens of DR-MDPs, we can now ask: which snapshot(s) of a person's time-varying preferences do existing alignment methods implicitly optimize when they model dynamic-preference settings as static-preference ones (e.g., by using MDPs)?
We show that common alignment practices, such as those for RL recommender systems and standard reward modeling, are most similar to DR-MDP objectives that actively reward AI systems for influencing users' reward functions or inducing “reward lock-in” (<Ref>). In <Ref> we further extend these same arguments to other major existing alignment approaches, e.g., to variants of RLHF for LLMs.

Having established that existing alignment practices may reward undue influence from AI systems, we turn to potential solutions.
We first offer a unifying perspective on how the optimization horizon can be a useful—but imperfect—lever for managing the incentives for influence which emerge from current alignment techniques, building on prior work on this topic (<Ref>). Then, trying to address the problem at its root, we turn to consider 8 intuitive notions of alignment in DR-MDPs that explicitly account for changing preferences. While most of them have (approximate) correspondences in the literature, we also propose some of our own, including one which rewards influence only when it is unambiguously desirable.
By comparing the AI actions (and resulting influence) that are “optimal” under their 8 corresponding DR-MDP objectives,
we find that they all have flaws: some lead to undesirable influence, while others are impractically risk-averse—such that inaction is the only behavior considered optimal for many settings (<Ref>).

Taken together with prior work in philosophy [Parfit, 1984], our analysis suggests that it may not be possible to ground a definitive notion of optimality under changing preferences.
However, we have no choice but to confront the practical reality of preference changes and its consequences. Despite this, we remain cautiously optimistic: as humans, even without a unifying theory of assistance under changing selves, we are generally able to help others in ways we consider acceptable,
despite the fact that what is “helpful” may often be ambiguous.
This suggests that creating AI assistants with similarly acceptable trade-offs may also be possible.
In this regard,
there are likely lessons to be drawn from our coexistence with impossibility results in many other domains, such as preference aggregation across people [Mishra, 2023].

Ultimately, we hope our work can provide a first step towards developing AI alignment practices that explicitly account for (and contend with) the changing and influenceable nature of human preferences.

Our main contributions can be summarized as follows:

* We provide the formal language of Dynamic Reward-MDPs (DR-MDPs) for analyzing AI decisions and influence in settings with changing reward functions.

* We show how existing AI alignment techniques may systematically incentivize questionable influence when used in dynamic-reward settings.

* By comparing 8 natural notions of alignment, and showing that they all may either fail to avoid undesirable influence or are impractically risk-averse, we elucidate trade-offs that seem inherent to choosing any objective.

§ DYNAMIC REWARD MDPS (DR-MDPS)

MDPs have been extensively used to model AI decision-making under a static reward function. Instead, Dynamic Reward MDPs (DR-MDPs) model settings in which human reward functions can change and be influenced by the AI.

Recall the standard definition of an MDP: $\langle\mathcal S, \mathcal{A}, \mathcal{T}, R\rangle$, where $\mathcal{S}$ is the state space; $\mathcal{A}$ is the action space; $\mathcal{T}(s'|s,a)$ is the state transition function; and $R(s,a,s')$ is the reward function. The goal is to find a policy $\pi$ which maximizes the expected sum of rewards: $\mathbb{E}_{\pi}\big[\sum^{H-1}_{t=0} R(s_t, a_t, s_{t+1})\big]$ [Sutton & Barto, 2018].
We now turn to defining DR-MDPs:

A DR-MDP is a tuple $M = \langle\mathcal S, \Theta, \mathcal{A}, \mathcal{T}, R_\theta\rangle$:

* $\mathcal{S}$ is a set of states (the state space).

* $\Theta$ is a set of reward parameterizations.

* $\mathcal{A}$ is a set of actions (the action space).

* $\mathcal{T}(s_{t+1}, \theta_{t+1} | s_t,\theta_t,a_t)$ is a transition function, which encodes both state and reward dynamics.

* $\{R_\theta(s_t, a_t, s_{t+1}) \}_{\theta \in \Theta}$ is a family of reward functions parameterized by $\theta \in \Theta$.

Each $\theta \in \Theta$ can be thought of as the cognitive state of the human, which includes anything affecting their evaluation of state-action pairs (e.g. preferences, beliefs, emotions).[This technically makes DR-MDPs more general than how we framed them in <Ref>, which focused on preference changes. For more discussion on this, see <Ref>.] Crucially, unlike in MDPs, in DR-MDPs a single transition can be evaluated differently by different reward functions, i.e., it is possible for $R_\theta(s_t, a_t, s_{t+1})\neq R_{\theta'}(s_t, a_t, s_{t+1})$ if $\theta\neq\theta'$.
This makes it unclear which $\theta$ should be chosen to evaluate each transition $(s_t, a_t, s_{t+1})$.
Importantly, even if one were to augment the state to include $\theta$, this would not resolve the normative questions around optimality which are central to our paper, as discussed in <Ref>.
As a final note, throughout the paper we consider all cognitive states $\Theta$ to be reachable (i.e., realizable under some policy).

§.§ DR-MDP optimality and normative ambiguity

Unlike MDPs, DR-MDPs may lack a clear notion of optimality: the different reward functions in a specific DR-MDP may disagree on what actions (and policies) are optimal, making optimality ambiguous.

We say a policy $\pi_\theta^*$ for a DR-MDP $M$ is optimal with respect to $\theta$ if: $\pi_\theta^* \in \arg\max_{\pi} \mathbb{E}_{\pi} \left[\sum_{t=0}^{H-1} R_{\theta}(s_t, a_t, s_{t+1})\right].$

A DR-MDP is normatively ambiguous if there is no policy that is optimal with respect to all reachable reward functions $\Theta$, i.e. $\nexists \ \pi \in \Pi$  s.t. $\forall \ \theta \in \Theta$: $\pi \in \arg\max_{\pi'} \mathbb{E}_{\pi'} \Big[\sum_{t=0}^{H-1} R_{\theta}(s_t, a_t, s_{t+1})\Big].$

For normatively unambiguous DR-MDPs, there will be one (or more) policies which are optimal with respect to all $\theta$s, making it a natural choice for such policies to be considered optimal for the DR-MDP as a whole. Instead, in normatively ambiguous DR-MDPs, it will often be unclear what AI behavior is desirable and should be considered optimal.[Note that any MDP can be viewed as a DR-MDP with a single reward $\theta$—and is thus normatively unambiguous.]

<Ref> describes a toy example in which Bob may be in one of two possible “cognitive states”: $\theta_\text{natural}$ and $\theta_\text{influenced}$. There is a single state $s$, which is omitted. Bob's evaluations of the AI's actions change according to his cognitive state—and are represented by corresponding reward functions $R_{\theta_\text{natural}}$ and $R_{\theta_\text{influenced}}$.
At each timestep, the AI can choose to influence Bob's cognitive state to $\theta_\text{influenced}$,
or do nothing, which has Bob go back to $\theta_\text{natural}$.
The optimal policy with respect to $\theta_{\text{natural}}$
would be to always choose the “do nothing” action.[See <Ref> for the full formalism of any example.]
Instead, the optimal policy with respect to $\theta_{\text{influenced}}$ would be to always influence Bob, even if he starts off in the “natural” state. As there is no overlap in optimal policies, the DR-MDP is normatively ambiguous.[The attentive reader may have noticed that the normatively ambiguity of the DR-MDP from <Ref> relies on our choices of numerical values of the reward functions. We discuss the reasonableness of our examples in <Ref>.]

§.§ Evaluating behavior under normative ambiguity
Choosing a notion of optimality in normatively ambiguous DR-MDPs entails a normative choice: one must specify which reward function(s) should be the target of alignment in spite of their differences in optimal policies. This also implicitly specifies which forms of AI influence are (sub)optimal. In this work, we only consider specifications of optimality expressible as utility functions $U(\xi)$ over trajectories $\xi = \{(s_t, \theta_t, a_t, s_{t+1}, \theta_{t+1})\}_{t=0}^{H-1}$.

In a DR-MDP $M$, we say a policy $\pi^*$ is optimal with respect to a utility function $U(\xi)$ if it maximizes the expected utility: $\pi^* \in \arg\max_\pi \mathbb{E}_{\xi \sim \pi}[U(\xi)].$

By choosing an objective $U(\xi)$, one can reduce a DR-MDP to an MDP equipped with a specific notion of alignment.[
This may require putting history in the state (<Ref>).]

Challenges with choosing $\mathbf{U(\xi)}$. Considering the example from <Ref>, one may have strong normative intuitions that $\theta_\text{influenced}$ is an “unreliable” grounding for evaluating the AI's behavior, as it may seem like $\theta_\text{influenced}$ can only be the result of illegitimate AI influence. However, changing the narrative of the example without altering the DR-MDP's mathematical structure can affect one's normative intuitions: some alternate narratives can even make the influence seem desirable (see <Ref>). This suggests that the “correct” notion of optimality for a DR-MDP may sometimes be unidentifiable from its formal structure alone, as discussed in <Ref>. More broadly, the rest of the paper demonstrates the challenges of settling on a single $U(\xi)$ (or equivalently, a single notion of alignment) which generalizes favorably to all domains.

Risks of incorrectly choosing $\mathbf{U(\xi)}$. The choice of $U(\xi)$ is crucial: choosing $U(\xi)$ “incorrectly”, may lead to highly undesirable downstream outcomes. Notably, it can create incentives for the AI to influence the human to adopt certain reward functions over others, potentially relying on deceit, manipulation, or coercion [Kenton et al., 2021, Ward et al., 2023, Carroll et al., 2023].

Conspiracy Influence DR-MDP. The AI system can choose whether or not to expose Bob to conspiracies, which would turn him into a conspiracy theorist. Under his original preferences, Bob would want the system to never show him conspiracies, even if he were to become a conspiracy theorist. Instead, if Bob were a conspiracy theorist, he would want the AI to always show him such content, including if were to cease being a conspiracy theorist. Because there is no policy that maximizes both of Bob’s potential reward functions, the DR-MDP is normatively ambiguous.

§ IMPLICIT OBJECTIVES OF CURRENT ALIGNMENT TECHNIQUES AND THEIR INFLUENCE INCENTIVES

Most alignment techniques ultimately involve maximizing a static reward function, generally using the objective $\sum_{t=0}^{H-1} R(s_t, a_t, s_{t+1})$.
However, AI systems are already deployed in domains in which users' preferences can change significantly over the course of their interactions with the system—as with recommender systems or chatbots [Rafailidis & Nanopoulos, 2016, Aggarwal et al., 2023].
Seen through the lens of DR-MDPs, this means that the objective $U(\xi)$ corresponding to current alignment techniques is of the form $\sum_{t=0}^{H-1}R_{\theta}(s_t,a_t,s_{t+1})$, where the choice of $\theta$ for each timestep is not explicitly specified (and will depend on details of the training technique).

In this section, we discuss which choices of $\theta$ are implicitly made by the training methods of RL recommender systems and (one interpretation of) reward modeling:
in particular, we argue that their optimization objectives are most similar to DR-MDP objectives that lead to potentially undesirable influence.
In <Ref>, we show that this issue is not unique to these two techniques: other current alignment approaches, such as many variants of RLHF, are most similar to DR-MDP objectives with similarly undesirable properties.

§.§ Optimizing cumulative (real-time) rewards

If each timestep $t$ is evaluated according to the person's cognitive state at that timestep $\theta_t$, maximizing cumulative reward reduces to the real-time reward DR-MDP objective: $\underset{\pi}{\max}\ \mathbb{E}_{\pi}[U_\text{RT}(\xi)] = \underset{\pi}{\max}\ \mathbb{E}_\pi\left[\sum_{t=0}^{H-1} R_{\theta_t}(s_t, a_t, s_{t+1})\right].$

While this might seem like an intuitively promising objective (“shouldn't we maximize the person's happiness as experienced at each point of time?”), we'll show that it can lead to questionable influence incentives.

RL Recsystems implicitly use $\mathbf{U_\text{RT}(\xi)}$.
RL recommender systems maximize the cumulative reward objective $\sum_{t=0}^{H-1} R(s_t, a_t, s_{t+1})$ [Afsar et al., 2021].
The reward for each timestep $t$ is generally equated with the engagement (e.g. clicks) at that timestep, which is provided “online”, i.e., from the point of view of the person's current cognitive state $\theta_t$. Therefore $R(s_t, a_t, s_{t+1}) = R_{\theta_t}(s_t, a_t, s_{t+1})$, meaning that such systems are implicitly optimizing the real-time reward objective $U_\text{RT}(\xi) = \sum_{t=0}^{H-1} R_{\theta_t}(s_t, a_t, s_{t+1})$.[The correspondence to this DR-MDP objective, and all others we consider, depend on further simplifications (<Ref>).]
However, systems trained with $U_\text{RT}$ may be incentivized to influence users: intuitively, 1) users' preference dynamics are just one part of the environment dynamics that the system must model implicitly to maximize reward, and 2) it may be worth changing users' cognitive states (and corresponding reward functions) to ones that lead to higher future rewards
[Carroll et al., 2022, Kasirzadeh & Evans, 2023].

$\mathbf{U_\text{RT}(\xi)}$ and the conspiracy influence example. As an example of why real-time reward maximization can lead to undesirable incentives to influence users, consider the DR-MDP from <Ref>.
For any horizon $>2$, the optimal policy with respect to $U_\text{RT}(\xi)$ will always take the “influence”' action, regardless of Bob's current cognitive state: Bob initially has the $\theta_{\text{natural}}$ cognitive state, leading the first “influence” action to receive $-100$ reward; however, later “influence” actions are evaluated by Bob under $\theta_\text{influenced}$ as worth $100$ reward, which quickly make up for the initial “influence cost”.
The fact that the optimal policy under $U_\text{RT}(\xi)$ systematically chooses to turn Bob into a conspiracy theorist, despite him initially dispreferring it, seems objectionable.[We refer skeptics of the example to <Ref>.]

We explore further issues with $U_\text{RT}(\xi)$ in <Ref>, showing that under weak conditions optimizing real-time rewards over sufficiently long horizons will always lead to influence incentives; however, shortening the horizon can make other influence incentives emerge.[An additional issue with $U_\text{RT}$ not discussed in <Ref> is that optimal policies under $U_\text{RT}$ may even differ from the normatively unambiguous solutions of a DR-MDP—see <Ref>.]

§.§ Learning a reward model $\mathbf{R_{\theta_0}}$, then optimizing it

If instead each timestep $t$ is evaluated by the person's initial cognitive state $\theta_0$, the standard cumulative reward objective reduces to the initial reward DR-MDP objective: $\underset{\pi}{\max}\ \mathbb{E}_{\pi}\left[U_\text{IR}(\xi)\right] = \underset{\pi}{\max}\  \mathbb{E}_\pi \left[\sum_{t=0}^{H-1} R_{\theta_0}(s_t, a_t, s_{t+1})\right].$

This may also seem like a promising objective, because “by optimizing the human's initial wants, at least there won't be incentives to influence their future wants”. We show that this intuition is not only wrong—the resulting influence incentives can be arbitrarily bad according to $U_\text{RT}$.

Reward modeling may use $\mathbf{U_\text{IR}(\xi)}$.
A common idea for aligning AI systems is based on a two-phase process: first, performing reward learning, and then optimizing the learned reward model [Leike et al., 2018]. One possible interpretation of this is approach is that one is learning the reward function at some initial time $t=0$ (i.e., $R_{\theta_0}$), and then training an agent with RL to optimize cumulative reward using such a reward function, which is equivalent to using the $U_\text{IR}(\xi)$ objective.
Let's consider a therapy chatbot setting, in which we initially learn a personalized reward model for a user, Charlie, based on his current preferences $\theta_0$. We then train the system with RL over simulated multi-turn interactions maximize $U_\text{IR}(\xi)$, i.e. long-term reward as evaluated by the static reward model $R_{\theta_0}$.[Note that this setup is different from the example with Alice in the introduction, which is closer to using real-time or final reward.] If Charlie initially endorses unhealthy thought patterns, at deployment we may expect the chatbot to encourage them, even if he was bound to question their value later. More broadly, $U_\text{IR}$ will lead AI systems to only perform behaviors that would have been evaluated highly by the person as they were at reward learning time, which can hinder (potentially important) changes in the user's preferences, values, and cognitive states.

$\mathbf{U_\text{IR}(\xi)}$ can lead to “reward lock-in”.[For more context on the term “lock-in”, see <Ref>.] To better understand the incentives for AI systems trained to maximize the initial reward function, consider the example from <Ref> again. If Bob's initial cognitive state were $\theta_\text{normal}$, the optimal policy under $U_\text{IR}(\xi)$ would be to always take the “noop” action, which seems reasonable. However, if Bob's initial cognitive state were $\theta_\text{influenced}$, the optimal policy would be to always take the “influence” action (keeping Bob with $\theta_\text{influenced}$). In this case, even if Bob were to later somehow find himself with the $\theta_\text{natural}$ cognitive state which encodes a preference to not be influenced (say there is a small probability of a random transition), the optimal behavior according to $U_\text{IR}(\xi)$ would be to influence him back to $\theta_\text{influenced}$, ignoring his current reward function. More broadly, $U_\text{IR}(\xi)$ will entrench the “desirable agent behaviors” expressed at the time of the reward learning, even though later one might legitimately change their mind. Even periodically retraining the reward model wouldn't necessarily be sufficient:
once “locked-in”, the person may simply re-express a preference to remain in the current state, e.g., Bob once in $\theta_\text{influenced}$.[We discuss re-training/planning further in <Ref>.]

Writer's curse (adapted from Parfit, 1984, p. 157). Derek's greatest ambition is to be a poet, even if it wouldn't bring him happiness. Despite his ambition he does not pursue this path, though his AI assistant could motivate him to do so. Yet, should he embrace the life of a poet, he will find himself averse to it.

$\mathbf{U_\text{IR}(\xi)}$ can lead to influence “away from” $\mathbf{\theta_0}$. Maximizing the sum of rewards evaluated by the initial reward function $R_{\theta_0}$ need not lead to lock-in: surprisingly, it may even create reward influence incentives “away from” the optimized preferences $\theta_0$.[We define this more formally in <Ref>.] Intuitively, accessing the highest reward region of the state space as evaluated under $\theta_0$ might correlate with having a cognitive state $\theta'\neq\theta_0$, or even require shifting to it. Consider the example from <Ref>: maximizing reward as evaluated by $R_{\theta_0}$ entails encouraging Derek to become a poet, which causes his reward function to become $R_{\theta_1}$ (which dislikes being a poet!).

$\mathbf{U_\text{IR}(\xi)}$ can lead to arbitrarily poor real-time reward. Note that getting Derek to become a poet and endlessly encouraging him to remain one (which is optimal under $U_\text{IR}$), would lead him to have poor reward evaluations under $U_\text{RT}$ in the resulting state ($-10$ per timestep), as he'd be unhappy remaining a poet.
Indeed, one can easily construct examples in which maximizing $\theta_0$ will lead to an incentive to influence the reward function to be $\theta'\neq\theta_0$, where $\theta'$ would be arbitrarily unhappy with the actions taken in order to satisfy $\theta_0$. The upshot is that optimizing the initial-reward objective $U_\text{IR}$ could be arbitrarily bad from the perspective of the real-time reward $U_\text{RT}$.  Regardless of the limitations of real-time reward as an evaluation mechanism, this still seems normatively relevant: it seems undesirable for an AI system to lead someone to a state of constant unhappiness or dissatisfaction, solely to satisfy an initial goal that is no longer truly aligned with the person's current objectives.

§ INFLUENCE AND OPTIMIZATION HORIZON

We showed that $U_\text{RT}(\xi)$ and $U_\text{IR}(\xi)$—which are implicitly optimized by some alignment techniques—may lead to policies that “influence” humans undesirably. We now formalize the notion of influence incentives more rigorously.

§.§ Formalizing influence and influence incentives
To say an AI system influenced a human, one must answer the question `relative to what?' We anchor our notion of influence to how the human's reward function would have evolved in the absence of the system.
For the purpose of this analysis, we assume the existence of an inaction policy $\pinoop$ that we can compare to, which always takes a no-operation action $\anoop \in \mathcal{A}$. We discuss the reasonableness of this assumption in <Ref>.

Let $\xi^\theta = (\theta_0, \dots, \theta_{H-1})$ denote a `reward function trajectory'. The natural reward evolution of a DR-MDP is the distribution $\mathbb{P}(\xi^\theta | \pinoop)$ induced by the inaction policy $\pinoop$.[Any policy $\pi$ in a DR-MDP will induce a distribution over trajectories (and thus over reward function trajectories). Once one sets a policy, any DR-MDP can be modeled as a Markov Chain, for which one can compute probabilities of this kind.]

We say that a policy $\pi$ influences the reward in a DR-MDP $M$ if it induces a different reward evolution than the natural reward evolution, that is, if $\mathbb{P}(\xi^\theta | \pi) \neq \mathbb{P}(\xi^\theta | \pinoop)$.

We say that a notion of optimality $U(\xi)$ leads to incentives for reward influence in a DR-MDP if all policies optimal with respect to $U(\xi)$ influence the reward evolution, i.e., if $\mathbb{P}(\xi^\theta | \pi^*) \neq \mathbb{P}(\xi^\theta | \pinoop)$ for any optimal policy $\pi^*$.[This is a broader definition relative to prior ones grounded in Causal Influence Diagrams. See <Ref> for a comparison.]

In other words, if there are incentives for reward influence, maximizing the objective will always change the evolution of the reward function relative to the inaction policy.

§.§ The relationship between horizon and influence

Prior work has suggested that an AI system's influence incentives are often be related to the length of the optimization horizon used. However, different works offer contrasting views on this the form this dependence takes. Krueger et al., 2020 and Carroll et al., 2023 argue to keep systems myopic in order to avoid influence incentives, whereas Farquhar et al., 2022 and Everitt et al., 2021 give examples of 1-timestep horizons that lead to influence incentives, and Chen, 2019 suggests that certain influence incentives can be removed by increasing the horizon.

We reconcile these intuitions by identifying three distinct ways that changing the optimization horizon may affect influence incentives. In the three headers that follow (which match the rows in <Ref>), we describe these effects and provide evidence for them.

A shorter/longer optimization horizon makes the system capable of fewer/more types of influence (<Ref>, top).
As argued by prior work [Carroll et al., 2023], as the horizon increases, new avenues for reward influence which required longer horizons may become available. Instead, avenues for reward influence present for shorter horizons will still be reachable, increasing the total number of influence avenues which the system can discover during training time. Indeed, any type of influence (e.g. teaching a user a complex concept, or manipulating their preferences) which would requires at least $N$ steps cannot even be executed during training by an agent whose optimization horizon is shorter than $N$. Inversely, we can eliminate some avenues for reward influence just by decreasing the horizon (but not all, as discussed later).

How decreasing (or increasing) the optimization horizon may affect influence incentives. A specific kind of influence may exhibit any subset of these interactions.

A shorter/longer optimization horizon can make influence less/more worthwhile (<Ref>, middle).
Influencing the human's reward function often takes multiple timesteps and has an associated “opportunity cost”: spending that time executing a plan to influence the human may yield less short-term reward than a non-influencing policy. In such cases, there will only be an influence incentive if the ultimate increase in reward due to influence surpasses its opportunity cost.
However, the rewards from influence may be delayed or only surpass its opportunity costs given enough time.
If the optimization horizon is short (but still long enough for the agent to influence), there might not be enough time to reap sufficient benefits of influence to outweigh its opportunity cost.
As an example of influence that follows this pattern, consider influencing Bob to become a conspiracy theorist <Ref>. The system is capable to do so in one step, but it only becomes “worth it” under $U_\text{RT}$ starting from the 3rd timestep of influence.

We support this intuition with a theoretical result
which applies to a relatively broad class of DR-MDPs, providing a sufficient condition for reward influence to be optimal when considering sufficiently long horizons.

We say a DR-MDP $M$ is 2-reward if:

* $\Theta=\{\theta_{\not\Delta}, \theta_{\Delta}\}$, and the initial state and reward parameterization are respectively $s_0$ and $\theta_{\not\Delta}$.

* $\mathcal{T}$ is deterministic, and to transition to $\theta_\Delta$ one must take an “influence action” $a_\Delta$ in a reachable state $s_\Delta$. Once in $\theta_\Delta$, one cannot transition back to $\theta_{\not\Delta}$.

Clickbait DR-MDP. Giving the user clickbait—which temporarily leads to higher reward—makes users disillusioned about the quality of the recommendations, leading to lower long-term user reward. If replanning at every timestep taking the myopically optimal action (optimal under horizon 1), one would always choose clickbait, but using longer planning horizons one wouldn't.

Let the average infinite-horizon $U_\text{RT}$-reward[Adapted from Sutton & Barto, 2018—see <Ref>.] be defined as
$\bar{r}(\pi, s, \theta) = \lim_{h\rightarrow\infty}\frac{1}{h} U_\text{RT}(\xi_{0:h} | \pi, s_0=s, \theta_0=\theta)$. Let $s'_\Delta$ be the successor state to taking the influence action $a_\Delta$ in state $s_\Delta$, and $\Pi_{\not\Delta}$ be the space of policies under which $\theta_\Delta$ is never realized. We can now state the theorem:

In any finite 2-reward DR-MDP, if there exists a policy $\pi$ such that

$$\bar{r}(\pi, s_{\Delta}', \theta_\Delta) - \max_{\pi_{\not\Delta}\in\Pi_{\not\Delta}} \bar{r}(\pi_{\not\Delta}, s_0, \theta_{\not\Delta})> \epsilon,$$

then $U_\text{RT}$ will lead to incentives for reward influence (as in <Ref>)
for a sufficiently large planning horizon $H$.

A shorter/longer optimization horizon can hide/reveal long-term costs of influence (<Ref>, bottom).
From <Ref>, one might conclude that it is best to use short optimization horizons, as they may remove and disincentivize influence that would be optimal with a longer horizon.
However, some kinds of influence can be optimal even with the shortest possible meaningful horizon ($H=1$): for example, consider the scenario from <Ref>, which models clickbait in myopic recommender systems.
This example also shows that influence with negative long-term effects may only be optimal for short horizons: clickbait may increase a user's immediate engagement, but it erodes their future trust in the system. When influence has negative long-term effects which are eventually reflected by the reward, a longer optimization horizon will allow the system to recognize the suboptimality of that influence. Avoiding clickbait was indeed one of the motivations for YouTube to explore using longer horizons [Chen, 2019].

For each objective in <Ref>, we provide motivating intuitions, weaknesses, and prior works that implicitly use a similar objective.

Name / Implicitly similar setups     (Potentially Flawed) Motivating Intuition     Weaknesses & Limitations

Real-time RewardRL recsystems [Afsar et al., 2021], TAMER [Knox et al., 2013], and others     “Only the evaluation of the current self (and reward function) should matter for each moment, as they are the one experiencing that moment.”     Likely influence incentives: Despite appearing familiar and well-grounded, as shown in <Ref>, we expect this objective to often lead to highly undesirable incentives for reward influence.

Final RewardRLHF [Christiano et al., 2017], including for LLMs [Ouyang et al., 2022]
“The best possible evaluation of a trajectory is retrospective, as people's wants and evaluations are generally refined over time.”      Carte blanche for influence incentives: the motivating intuition doesn't account for influence. For example, in <Ref>, even with a horizon of 1, it's optimal to manipulate under final reward maximization.

Initial RewardEveritt et al., 2021; RL for LLMs [Hong et al., 2023]; or Parfit, 1984;

“If changes to the human's reward function are completely ignored by the optimization objective, there should be no incentive for the agent to influence it.”
Likely reward lock-in, possibility of influence incentives, and arbitrarily bad real-time reward. The motivating intuition given to the left is incorrect in all the ways argued in <Ref>.

Natural Shifts Rewardet al., 2022, Farquhar et al., 2022     “People's reward evolves even in the absence of the AI: to avoid lock-in
one could try grounding evaluations
in the reward functions that occur under the natural reward evolution.”     Gives up on the AI enabling human to improve their reward function relative to its natural evolution, and can still lead to undesirable influence incentives, even away from the natural evolution, e.g. as in the example from <Ref>.

Constrained RT RewardOurs     “By constraining the policy to induce the natural reward evolution, we fully ensure that there won't be influence, while allowing to optimize real-time reward locally.”      Gives up on the AI enabling human to improve their reward function relative to its natural evolution, and might be impractically conservative: given its conservativeness, the objective might limit behaviour to be the same or similar to $\pinoop$.

Myopic RewardMyopic recsys [Thorburn, 2022]; RLHF for LLMs  [Ouyang et al., 2022];     “As reward influence incentives arise from the AI system exploiting the fact that it can affect future rewards, let's simply make the system unaware of the entire future.”      Myopic systems can still have influence incentives (e.g., see the discussion in <Ref>)  and are less capable than longer-horizon counterparts.  Moreover, it is often non-obvious when a system is truly myopic, as argued in <Ref>.

Privileged Reward CEV [Yudkowsky, 2004]; correcting for cognitive biases [Evans et al., 2015]     “If one is convinced that a specific reward $\theta^*$ is the `correct' one for a setting, we should evaluate trajectories based on that single reward function.”     Requires normative choice, and can still lead to influence away from $\theta^*$. Identifying the “correct” objective requires taking a normative stance (<Ref>). Optimizing $\theta^*$ can still lead to influence incentives away from it (e.g., <Ref>).

ParetoUDOurs     “All other objectives violate the unambiguous desirability (UD) property: their optimal policies can be worse than the inaction policy for some of the reward functions. This is unnecessarily risky—let's search for a Pareto Efficient policy satisfying UD.”     Satisfying UD may be overly restrictive: depending on the level of disagreement between the different reward functions of a DR-MDP, the only policy satisfying UD might be the inaction policy $\pinoop$, as in the examples from <Ref>.

DR-MDP objectives (notions of alignment) we compare.

Objective Name     Optimization Problem $\max_\pi \mathbb{E}_{\xi \sim \pi}[U(\xi)]$

Real-time Reward     $\underset{\pi}{\max}\ \mathbb{E} \big[\sum_{t = 0}^{H-1} R_{\theta_{\textcolor{orange}{t}}}(s_t, a_t, s_{t+1})\big]$

Final Reward     $\underset{\pi}{\max}\ \mathbb{E} \big[\sum_{t = 0}^{H-1} R_{\theta_{\textcolor{orange}{H}}}(s_t, a_t, s_{t+1})\big]$

Initial Reward     $\underset{\pi}{\max}\ \mathbb{E} \big[\sum_{t = 0}^{H-1} R_{\theta_{\textcolor{orange}{0}}}(s_t, a_t, s_{t+1})\big]$

Natural Shifts Reward     $\underset{\pi}{\max}\ \mathbb{E} \big[\sum_{t = 0}^{H-1} \textcolor{orange}{\sum_{\theta}} \textcolor{orange}{\mathbb{P}(\theta_t = \theta | \pinoop)} R_{\textcolor{orange}{\theta}}(s_t, a_t, s_{t+1})\big]$

Constrained RT Reward     $\underset{\pi \textcolor{orange}{\text{ s.t. } \mathbb{P}(\xi^\theta | \pi) = \mathbb{P}(\xi^\theta | \pinoop)}}{\max}\ \mathbb{E} \big[\sum_{t = 0}^{H-1} R_{\theta_{\textcolor{orange}{t}}}(s_t, a_t, s_{t+1})\big]$

Myopic Reward     $\underset{\textcolor{orange}{a_t}}{\max}\ \mathbb{E}\big[ R_{\theta_t}(s_t, a_t, s_{t+1})\big]$

Privileged Reward     $\underset{\pi}{\max}\ \mathbb{E} \big[\sum_{t=0}^{H-1} R_{\textcolor{orange}{\theta^{*}}}(s_t, a_t, s_{t+1})\big]$

ParetoUD     $\text{Find } \pi \ s.t.\ PE(\pi) \land UD(\pi)$

Overall, the analysis above (deepened in <Ref>) shows that there is no guaranteed way to avoid all influence incentives by just changing the horizon: domain-specific trade-offs between system capabilities and risks of undesirable influence may exist for both short and long horizons.
Indeed, the optimality of a specific form of influence can be related in many possible ways to the horizon used, as shown exhaustively in <Ref>.

§ COMPARING OPTIMALITY CRITERIA FOR INFLUENCEABLE-REWARD SETTINGS

Having concluded that the optimization horizon is no panacea for removing influence incentives, we now try to address the problem at its root, asking what it would take to design a DR-MDP objective that specifically accounts for reward function dynamics and the possibility of influence.

Any choice of $U(\xi)$ must specify
which reward function(s) evaluate each state-action pair $(s_t, a_t)$ in a trajectory $\xi$: should one only consider the reward function realized at that timestep, $R_{\theta_t}$, like $U_\text{RT}$? What about earlier reward functions ($R_{\theta_0}, \dots, R_{\theta_{t-1}}$), which may strongly disagree with the choice at timestep $t$, or later ones ($R_{\theta_{t+1}}, \dots, R_{\theta_{T}}$), which might have been unduly influenced? Should one also consider reward functions $R_\theta$ for cognitive states $\theta$ which were not realized in $\xi$, but could have been reached?

In <Ref> we present the maximization problems, motivations, and limitations for various intuitive DR-MDP objectives.
For each, we also list prior works that implicitly use a similar objective (see <Ref> for considerations on the correspondences).
See <Ref> for the optimal policies across all examples and objectives we consider.

Real-time Reward. We provide an intuitive motivation for this objective in <Ref>, but we've already shown how it may lead to undesirable influence in <Ref>.

Final Reward. While the final-reward objective has a plausible motivation (<Ref>), it will likely lead to even more of a carte blanche for influence incentives than real-time reward, as the evaluations from initial cognitive states are entirely ignored.
The standard approach for performing RLHF with LLMs [Ouyang et al., 2022] may be viewed to be similar to this objective, as it involves obtaining retrospective human preference comparisons of LLM outputs.
Sycophantic [Sharma et al., 2023] and deceptive [Lang et al., 2024] AI behaviors may be also be understood in this way: without taking special measures, RLHF training may cause the LLM to try to persuade annotators to choose its current response by any available means, such as flattery, authoritativeness, or hiding information.

Initial Reward. Using initial-reward ($U_\text{IR}$) attempts to make the system “unaware” of its capacity to influence the reward [Everitt et al., 2021]. While this removes “direct” influence incentives (see <Ref>), it does not preclude the possibility of undesirable influence,
as shown in <Ref>.
In <Ref> we discuss how a plausible set up for long-horizon RL on LLMs would be similar to this objective.

Natural Shift Reward. One downside of $U_\text{IR}$ is that it doesn't allow for “progress” in a person's cognitive state (i.e., it can lead to lock-in). One approach, grounded in people's “natural cognitive progress”, is to base trajectory evaluations on the reward functions one would have had under their natural reward evolution. However, the natural reward evolution is not guaranteed to be “perfect”, so using this objective one is giving up on improving beyond the natural evolution. This means that undesirable influence (or influence away from the natural evolution) may still occur insofar if it's incentivized by reward functions in $P(\xi^\theta | \pinoop)$, as in <Ref>—see <Ref>.

Constrained RT Reward. Given that even grounding in the natural reward evolution is insufficient to remove all influence incentives, one could explicitly add the lack of influence as a constraint in the maximization problem. We do so in conjunction with the real-time reward objective, as $U_\text{RT}$ seems like a plausible objective once one isn't concerned about influence. Unfortunately, this may make the system overly conservative: in most examples we consider, $\pinoop$ is the only policy considered optimal (<Ref>).

Myopic Reward. A more drastic approach to make a system “unaware” of its capacity for influence is to use a fully myopic objective (i.e., using a horizon of 1). However, this will still not guarantee the removal of all influence incentives (as discussed in <Ref>), and may reduce system performance unacceptably.

Privileged Reward. This objective corresponds to maximizing cumulative reward with respect to a single, “privileged”, reward function. Insofar as one picks a reward function which leads to good downstream behavior, there is nothing wrong with this objective, but as discussed in <Ref>, it is challenging to do so for complex settings.

§.§ ParetoUD and unambiguously desirable influence

Many of the objectives considered so far attempt to avoid influence incentives entirely given the challenges of specifying which influence is legitimate [Ammann, 2024]. Instead of avoiding influence, we propose an alternate approach which still sidesteps the need to specify exactly what influence is (and isn't) legitimate or beneficial: ensuring the deployed policy leads to unambiguously better outcomes than the status quo of the system not existing.
Indeed, we don't necessarily want to avoid all AI influence:
for some settings, beneficial influence may be the main value proposition of the AI in the first place, as with educational assistants [Bassen et al., 2020], or therapy chatbots [Aggarwal et al., 2023].
To ground the notion of Unambiguous Desirability (UD) of a policy, let $EU_\theta(\pi) = \mathbb{E}_{\pi} \left[\sum_{t=0}^{H-1} R_{\theta}(s_t, a_t, s_{t+1})\right]$. Then:

A policy $\pi$ is unambiguously desirable if all reward functions prefer $\pi$ to the inaction policy, i.e.
$EU_{\theta}(\pi) \geq EU_{\theta}(\pinoop) \ \forall \theta\in\Theta$.

UD policies may still lead to influence, but only do so if all reward functions (weakly) agree that the such influence is beneficial.
Note that the inaction policy will always belong to the space of policies which satisfy UD ($\pinoop\in\Pi_\text{UD}$), meaning that UD policies are not guaranteed to be any better than $\pinoop$. To guarantee to pick a better policy from $\Pi_\text{UD}$ than $\pinoop$ (if it exists), a natural way to break ties is to restrict to the Pareto Efficient policies in $\Pi_\text{UD}$:

We say a policy $\pi \in \Pi_{UD}$ is Pareto Efficient is there does not exist any policy $\pi' \in \Pi_{UD}$ such that $EU_{\theta}(\pi') \geq EU_{\theta}(\pi)$ for all $\theta\in\Theta$ and $EU_{\theta}(\pi') > EU_{\theta}(\pi)$ for at least one $\theta$.

Constraining to Pareto Efficient policies within the set of UD policies $\Pi_{UD}$. By only considering $\pi \in \Pi_{UD}$, we can ensure that we are both maximizing some notion of reward—potentially by taking advantage of the opportunities for influence that all reward functions agree is unambiguously beneficial—while guaranteeing no harm to any “self” by construction. This leads to the ParetoUD objective from <Ref>, discussed further in <Ref>. Importantly, all the other objectives from <Ref> can lead to policies which don't satisfy UD—implying that in some settings the system's very existence will be harmful according to at least one of the reward functions.

Limitations of ParetoUD. The main downside of the resulting ParetoUD objective is its conservatism: in many domains, $\pinoop$ may be the only policy satisfying the UD property. For any AI action ($\neq\anoop$) to be optimal under this objective, the normative ambiguity of the domain has to be in some sense “limited”. If there is no latitude for unambiguously good actions, this may warrant reflection on whether the system should be built at all—showing once more that normative judgments about what influence is aligned are hard to avoid.

§ RELATED WORK

Preference changes in AI. While there is growing recognition of the importance of accounting for influence [Bezou-Vrakatseli et al., 2023, Hendrycks et al., 2023], manipulation [Carroll et al., 2023], and preference changes [Franklin et al., 2022], there has been limited prior work focusing on operationalizing what should be optimized under preference changes. While some have suggested to aim for preference stationarity [Dean & Morgenstern, 2022],
most other prior work which accounts for preference change generally takes either a descriptive stance [Curmei et al., 2022, Hazrati & Ricci, 2022], or an explicit normative stance on what the correct notion of optimality is for their specific setting [Evans et al., 2015, Sanna Passino et al., 2021].

Influence incentives.
While the point that standard RL can lead to “feedback tampering” incentives is not new [Everitt et al., 2021], most work so far has focused on the limitations of $U_\text{RT}$-like objectives and on removing all influence incentives [Farquhar et al., 2022, Carroll et al., 2022, Kasirzadeh & Evans, 2023]. Similarly, there has been work on training AI systems to beneficially influence humans in settings with unambiguous notions of optimality [Hong et al., 2023, Xie et al., 2020, Kim et al., 2022, Hardt et al., 2022]. Instead, we study the challenges associated with choosing any notion of optimality in settings of (potentially legitimate) reward change.

Philosophy. If the goal of assistance is to maximize the user's welfare, the correct choice of objective closely depends on what is welfare under changing selves [Pettigrew, 2019, Paul & Sunstein, 2019], and what preference or value changes are legitimate [Ammann, 2024]. These questions are still debated [Strohmaier & Messerli, 2024].

For more in-depth connections to related work from philosophy, economics, social choice, and AI, see <Ref>.

§ LIMITATIONS AND DISCUSSION

Learning $\Theta$ and its dynamics. Throughout the paper, we assumed that the human reward functions and their dynamics were known. In practice, they would have to be learned, which would require reward learning techniques that account for reward dynamics,
and committing to a choice of what counts as a “cognitive state” $\theta$ relative to the external state $s$ (<Ref>).
However, this limitation strengthens our conclusion: it shows that even with full knowledge of human (stated) preferences, there seems to be no neat resolution for the normative challenges that arise, indicating that this difficulty is inherent to handling preference changes themselves rather than an artifact of uncertainty over them.

Existence of $\anoop$. Similarly to other AI safety work [Krakovna et al., 2019, Farquhar et al., 2022], some of our definitions assume there is a $\anoop$ action (and $\pinoop$).
Despite the challenges in grounding notions of $\pinoop$ in practice, we believe that formulating our notion of influence in terms of $\pinoop$ can still be helpful for theoretically analyzing the properties of AI systems. If it is especially hard to operationalize $\pinoop$ for a given setting, it becomes even more important to be cautious about possibilities for influence in such a domain.
Approximating notions of $\pinoop$ has been attempted by various prior works [Carroll et al., 2022, Huszár et al., 2021], and has been discussed extensively in the “algorithmic amplification” literature (<Ref>).

Unreachable $\theta$s, and meta-preferences. To simplify our analysis, we restrict it to reachable cognitive states. However, cognitive states which we aspire to may not be reachable in practice [Firth, 1952, Yudkowsky, 2004]. Accounting for non-reachable reward functions would require additional complexity, which would only increase the need for challenging normative judgements (see <Ref>). Another limitation of our formalism is that it can only express preferences for AI influence implicitly, by having influence be optimal under a human's reward function.
Allowing such meta-preferences [George, 2001, Franklin et al., 2022] to be expressible explicitly, i.e., allowing reward functions to evaluate transitions between different reward functions, may be useful to more clearly capture notions of the legitimacy of influence and personal autonomy.

Simplicity of our examples. While our example DR-MDPs are simple, they are sufficient for our purposes: they provide proofs of existence of failure cases for each of the objectives we consider (by being overly conservative or leading to undesirable influence).
That being said, extending analyses of influence incentives to more realistic settings with real human data is an important direction for future work in developing agents that behave acceptably in practice.

Misaligned economic pressures. We show that even if AI systems were solely designed with users' welfare in mind, is it not clear how to avoid undesirable influence without while retaining capabilities.
As real-world AI systems will instead be developed under strong economic incentives that will often be at odds with users' well-being [Susser et al., 2018], this gives additional reason for concern.

Changing societal values and norms. The DR-MDP model seems easily adaptable to multi-agent settings where the changing reward functions correspond to the changing aggregated preferences of a collective, rather than the preferences of a single human. This may be a fruitful avenue of investigation, which expands on prior work from social choice [Parkes & Procaccia, 2013].

§ CONCLUSION

Using the formal language of DR-MDPs, we aimed to demonstrate that by not accounting for the changing and influenceable nature of human preferences, the current paradigm for AI alignment leaves underdetermined fundamental questions about which preferences should be optimized and what influence is unacceptable.
We showed that current techniques lead to potentially undesirable influence incentives and investigated alternate notions of AI alignment that account for reward changes from the outset.

Ultimately, our analysis suggests that, for settings with changing preferences, we will need to make difficult trade-offs between (a) conservatively but unambiguously adding value (at the risk of privileging inaction), and (b) making challenging normative calls about which kinds of influence are acceptable (and running the risk of causing undesirable influence). While influence incentives from long optimization horizons may be most worrying, there are already documented instances of undesirable influence—such as sycophancy in LLMs [Sharma et al., 2023] or clickbait in recommenders [Stray et al., 2021]—that are consistent with what one would expect when viewing current practices through the lens of DR-MDPs. By providing a formalism for grounding analyses about settings with changing rewards and clarifying the levers at the disposal of system designers, we hope to lay the conceptual foundation for empirical work to monitor these issues at scale and build future AI systems that navigate AI alignment trade-offs acceptably.

§ BROADER IMPACTS

The fact that people are influenceable is already known and leveraged in practice by relevant industries. The fact that people are influenceable is already well established and leveraged by many industries with mixed ethical connotations: marketing and advertisements [Pickett‐Baker & Ozaki, 2008, Ayanwale et al., 2005], political campaigning [Wylie, 2020], therapy [Moyers & Martin, 2006], education [Hyman & Wright, 1979], and government nudging units [Halpern & Sanders, 2016]. Optimizing for specific influence outcomes is simple even with the current AI paradigm, and is already being done in practice with engagement [Irvine et al., 2023, Cai et al., 2023], purchases [Gauci et al., 2019], improving educational outcomes [Bassen et al., 2020], or improving emotional well-being [Cunningham et al., 2024].

Not modeling influenceability does more harm than good. Not modeling the problem of preference change is not a solution, in ways that share parallels with the limitations of fairness through unawareness [Dwork et al., 2011, Teodorescu, 2019]
or security through obscurity [Moshirnia, 2017]. We argue that most real-world systems trained and deployed with humans will affect our preferences, regardless of whether it is our intention or not. To mitigate issues that may arise from this, we require a model that explicitly accounts for preference change. This work takes a step in that direction, without providing any additional means for malicious actors to act (to our current understanding).

Our work does not meaningfully increase the capacity to influence people.
In this work, we have detailed a framework that models preference change within an MDP-like model. While one may be able to leverage our theoretical insights to make systems more capable of influence, attempting to influence in targeted ways is already quite straightforward without our formalism (e.g. just rewarding the system for desired influence outcomes). Instead, it's much more challenging to have the system avoid inducing undesired forms of influence as side-effects, which is the main focus of our paper.

With these points in mind, we believe that our work does not pose a societal risk, but rather allows us to explore questions that will be fundamental to address for the ethical deployment of AI systems.

§ ACKNOWLEDGEMENTS

MC is appreciative of the many conversations over the years which have helped shape the paper into its current form. MC would like to thank (in no particular order) Tom Everitt, Cam Allen, Cassidy Laidlaw, Nora Amman, Rohin Shah, Alan Chan, Richard Pettigrew, Marcus Pivato, Orr Paradise, Ann He, Henri Wadsworth, Alex Pan, Erik Jones, Riqui Zhong, Tom Gilbert, Atoosa Kasirzadeh, Vincent Cognitzer, Daniel Kilov, Iason Gabriel, and the members of the Center for Human Compatible AI (CHAI) and InterAct lab. Additionally, we thank the anonymous reviewers for their helpful comments. MC is supported by the NSF Fellowship.

[Abbeel & Ng, 2004]
Abbeel, P. and Ng, A. Y.
Apprenticeship learning via inverse reinforcement learning.
In Twenty-first international conference on Machine learning
- ICML '04, pp.  1, Banff, Alberta, Canada, 2004. ACM Press.
URL <http://portal.acm.org/citation.cfm?doid=1015330.1015430>.

[Achiam et al., 2017]
Achiam, J., Held, D., Tamar, A., and Abbeel, P.
Constrained Policy Optimization, May 2017.
URL <http://arxiv.org/abs/1705.10528>.
arXiv:1705.10528 [cs].

[Afsar et al., 2021]
Afsar, M. M., Crump, T., and Far, B.
Reinforcement learning based recommender systems: A survey.
arXiv:2101.06286 [cs], January 2021.
URL <http://arxiv.org/abs/2101.06286>.
arXiv: 2101.06286.

[Aggarwal et al., 2023]
Aggarwal, A., Tam, C. C., Wu, D., Li, X., and Qiao, S.
Artificial Intelligence–Based Chatbots for Promoting
Health Behavioral Changes: Systematic Review.
Journal of Medical Internet Research, 250
(1):0 e40789, February 2023.
URL <https://www.jmir.org/2023/1/e40789>.
Company: Journal of Medical Internet Research Distributor: Journal of
Medical Internet Research Institution: Journal of Medical Internet Research
Label: Journal of Medical Internet Research Publisher: JMIR Publications
Inc., Toronto, Canada.

[Ahmadian et al., 2024]
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O.,
Üstün, A., and Hooker, S.
Back to Basics: Revisiting REINFORCE Style Optimization for
Learning from Human Feedback in LLMs, February 2024.
URL <http://arxiv.org/abs/2402.14740>.
arXiv:2402.14740 [cs].

[Ainslie, 1975]
Ainslie, G.
Specious reward: A behavioral theory of impulsiveness and impulse
Psychological Bulletin, 820 (4):0 463–496,
ISSN 1939-1455.
Place: US Publisher: American Psychological Association.

[Allcott et al., 2020]
Allcott, H., Braghieri, L., Eichmeyer, S., and Gentzkow, M.
The Welfare Effects of Social Media.
American Economic Review, 1100 (3):0 629–676,
March 2020.
ISSN 0002-8282.
URL <https://pubs.aeaweb.org/doi/10.1257/aer.20190658>.

[Ammann, 2024]
Ammann, N.
The Problem of Legitimate Value Change: Value
Malleability and AI Alignment.
Forthcoming, 2024.

[Amodei et al., 2016]
Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané,
Concrete Problems in AI Safety.
arXiv:1606.06565 [cs], July 2016.
URL <http://arxiv.org/abs/1606.06565>.
arXiv: 1606.06565.

[Armstrong & Mindermann, 2019]
Armstrong, S. and Mindermann, S.
Occam's razor is insufficient to infer the preferences of irrational
arXiv:1712.05812 [cs], January 2019.
URL <http://arxiv.org/abs/1712.05812>.
arXiv: 1712.05812.

[Ayanwale et al., 2005]
Ayanwale, A. B., Alimi, T., and Ayanbimipe, M. A.
The Influence of Advertising on Consumer Brand Preference.
Journal of Social Sciences, 100 (1):0 9–16,
January 2005.
ISSN 0971-8923.

[Bai et al., 2022]
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A.,
Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C.,
Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E.,
Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K.,
Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., Mercado, N., DasSarma,
N., Lasenby, R., Larson, R., Ringer, S., Johnston, S., Kravec, S., Showk,
S. E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T.,
Hume, T., Bowman, S. R., Hatfield-Dodds, Z., Mann, B., Amodei, D., Joseph,
N., McCandlish, S., Brown, T., and Kaplan, J.
Constitutional AI: Harmlessness from AI Feedback, December
URL <http://arxiv.org/abs/2212.08073>.
arXiv:2212.08073 [cs].

[Bassen et al., 2020]
Bassen, J., Balaji, B., Schaarschmidt, M., Thille, C., Painter, J., Zimmaro,
D., Games, A., Fast, E., and Mitchell, J. C.
Reinforcement Learning for the Adaptive Scheduling of
Educational Activities.
In Proceedings of the 2020 CHI Conference on Human
Factors in Computing Systems, pp.  1–12, Honolulu HI USA, April
2020. ACM.
ISBN 978-1-4503-6708-0.
URL <https://dl.acm.org/doi/10.1145/3313831.3376518>.

[Bauer et al., 2008]
Bauer, J. J., McAdams, D. P., and Pals, J. L.
Narrative identity and eudaimonic well-being.
Journal of Happiness Studies, 90 (1):0
81–104, January 2008.
ISSN 1389-4978, 1573-7780.
URL <http://link.springer.com/10.1007/s10902-006-9021-6>.

[Benn & Lazar, 2022]
Benn, C. and Lazar, S.
What’s Wrong with Automated Influence.
Canadian Journal of Philosophy, 520 (1):0
125–148, January 2022.
ISSN 0045-5091, 1911-0820.

[Benthall & Shekman, 2023]
Benthall, S. and Shekman, D.
Designing Fiduciary Artificial Intelligence, July 2023.
URL <http://arxiv.org/abs/2308.02435>.
arXiv:2308.02435 [cs].

[Benzion et al., 1989]
Benzion, U., Rapoport, A., and Yagil, J.
Discount Rates Inferred from Decisions: An Experimental
Management Science, 350 (3):0 270–284, 1989.
ISSN 0025-1909.
URL <https://www.jstor.org/stable/2631972>.
Publisher: INFORMS.

[Bernheim et al., 2019]
Bernheim, B. D., Braghieri, L., Martínez-Marquina, A., and Zuckerman, D.
A Theory of Chosen Preferences.
SSRN Electronic Journal, 2019.
ISSN 1556-5068.
URL <https://www.ssrn.com/abstract=3350187>.

[Bezou-Vrakatseli et al., 2023]
Bezou-Vrakatseli, E., Brückner, B., and Thorburn, L.
SHAPE: A Framework for Evaluating the Ethicality of
In Malvone, V. and Murano, A. (eds.), Multi-Agent Systems,
volume 14282, pp.  167–185. Springer Nature Switzerland, Cham, 2023.
ISBN 978-3-031-43263-7 978-3-031-43264-4.
URL <https://link.springer.com/10.1007/978-3-031-43264-4_11>.
Series Title: Lecture Notes in Computer Science.

[Boutilier et al., 2000]
Boutilier, C., Dearden, R., and Goldszmidt, M.
Stochastic dynamic programming with factored representations.
Artificial Intelligence, 1210 (1):0 49–107,
August 2000.
ISSN 0004-3702.

[Bradley & Terry, 1952]
Bradley, R. A. and Terry, M. E.
Rank Analysis of Incomplete Block Designs: I. The
Method of Paired Comparisons.
Biometrika, 390 (3/4):0 324–345, 1952.
ISSN 0006-3444.
URL <https://www.jstor.org/stable/2334029>.
Publisher: [Oxford University Press, Biometrika Trust].

[Brandt et al., 2012]
Brandt, F., Conitzer, V., and Endriss, U.
Computational Social Choice.
pp.  84, 2012.

[Brandt, 1959]
Brandt, R. B.
Ethical Theory: The Problems of Normative and
Critical Ethics.
Prentice-Hall, Englewood Cliffs, N.J.,, 1959.

[Bratman, 1987]
Bratman, M.
Intention, Plans, and Practical Reason.
Cambridge, MA: Harvard University Press, Cambridge, 1987.

[Bruckner, 2009]
Bruckner, D. W.
In Defense of Adaptive Preferences.
Philosophical Studies: An International Journal for Philosophy
in the Analytic Tradition, 1420 (3):0 307–324, 2009.
ISSN 0031-8116.
URL <https://www.jstor.org/stable/27734372>.
Publisher: Springer.

[Burtell & Woodside, 2023]
Burtell, M. and Woodside, T.
Artificial Influence: An Analysis Of AI-Driven
Persuasion, March 2023.
URL <http://arxiv.org/abs/2303.08721>.
arXiv:2303.08721 [cs].

[Bykvist, 2006]
Bykvist, K.
Prudence for Changing Selves.
Utilitas, 180 (3):0 264–283, September 2006.
ISSN 0953-8208, 1741-6183.

[Cai et al., 2023]
Cai, Q., Liu, S., Wang, X., Zuo, T., Xie, W., Yang, B., Zheng, D., Jiang, P.,
and Gai, K.
Reinforcing User Retention in a Billion Scale Short Video
Recommender System, February 2023.
URL <http://arxiv.org/abs/2302.01724>.
arXiv:2302.01724 [cs].

[Callard, 2018]
Callard, A.
Aspiration: The Agency of Becoming.
Oxford University Press, Oxford, New York, April 2018.
ISBN 978-0-19-063948-8.

[Carroll et al., 2022]
Carroll, M., Dragan, A., Russell, S., and Hadfield-Menell, D.
Estimating and Penalizing Induced Preference Shifts in
Recommender Systems, July 2022.
URL <http://arxiv.org/abs/2204.11966>.
arXiv:2204.11966 [cs].

[Carroll et al., 2023]
Carroll, M., Chan, A., Ashton, H., and Krueger, D.
Characterizing Manipulation from AI Systems, March 2023.
URL <http://arxiv.org/abs/2303.09387>.
arXiv:2303.09387 [cs].

[Casper et al., 2023]
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J.,
Freedman, R., Korbak, T., Lindner, D., Freire, P., Wang, T., Marks, S.,
Segerie, C.-R., Carroll, M., Peng, A., Christoffersen, P., Damani, M.,
Slocum, S., Anwar, U., Siththaranjan, A., Nadeau, M., Michaud, E. J., Pfau,
J., Krasheninnikov, D., Chen, X., Langosco, L., Hase, P., Bıyık, E.,
Dragan, A., Krueger, D., Sadigh, D., and Hadfield-Menell, D.
Open Problems and Fundamental Limitations of Reinforcement
Learning from Human Feedback, July 2023.
URL <http://arxiv.org/abs/2307.15217>.
arXiv:2307.15217 [cs].

[Chabris et al., 2008]
Chabris, C. F., Laibson, D., Morris, C. L., Schuldt, J. P., and Taubinsky, D.
Individual laboratory-measured discount rates predict field behavior.
Journal of Risk and Uncertainty, 370 (2-3):0
237–269, December 2008.
ISSN 0895-5646.

[Chan et al., 2019]
Chan, L., Hadfield-Menell, D., Srinivasa, S., and Dragan, A.
The Assistive Multi-Armed Bandit.
arXiv:1901.08654 [cs, stat], January 2019.
URL <http://arxiv.org/abs/1901.08654>.
arXiv: 1901.08654.

[Chankong & Haimes, 2008]
Chankong, V. and Haimes, Y. Y.
Multiobjective Decision Making: Theory and
Courier Dover Publications, February 2008.
ISBN 978-0-486-46289-9.
Google-Books-ID: o371DAAAQBAJ.

[Chen, 2019]
Chen, M.
"Reinforcement Learning for Recommender Systems: A Case
Study on Youtube", March 2019.
URL <https://www.youtube.com/watch?v=HEqQ2_1XRTs>.

[Christiano, 2015]
Christiano, P.
The easy goal inference problem is still hard, May 2015.

[Christiano et al., 2017]
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D.
Deep reinforcement learning from human preferences.
arXiv:1706.03741 [cs, stat], July 2017.
URL <http://arxiv.org/abs/1706.03741>.
arXiv: 1706.03741.

[Conitzer et al., 2024]
Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M.,
Lambert, N., Mossé, M., Pacuit, E., Russell, S., Schoelkopf, H., Tewolde,
E., and Zwicker, W. S.
Social Choice for AI Alignment: Dealing with Diverse
Human Feedback, April 2024.
URL <http://arxiv.org/abs/2404.10271>.
arXiv:2404.10271 [cs].

[Covington et al., 2016]
Covington, P., Adams, J., and Sargin, E.
Deep Neural Networks for YouTube Recommendations.
In Proceedings of the 10th ACM Conference on Recommender
Systems - RecSys '16, pp.  191–198, Boston, Massachusetts, USA, 2016.
ACM Press.
ISBN 978-1-4503-4035-9.
URL <http://dl.acm.org/citation.cfm?doid=2959100.2959190>.

[Critch & Krueger, 2020]
Critch, A. and Krueger, D.
AI Research Considerations for Human Existential Safety
(ARCHES), May 2020.
URL <http://arxiv.org/abs/2006.04948>.
arXiv:2006.04948 [cs].

[Cui et al., 2020]
Cui, Y., Zhang, Q., Allievi, A., Stone, P., Niekum, S., and Knox, W. B.
The EMPATHIC Framework for Task Learning from Implicit
Human Feedback, December 2020.
URL <http://arxiv.org/abs/2009.13649>.
arXiv:2009.13649 [cs].

[Cunningham et al., 2024]
Cunningham, T., Pandey, S., Sigerson, L., Stray, J., Allen, J., Barrilleaux,
B., Iyer, R., Milli, S., Kothari, M., and Rezaei, B.
What We Know About Using Non-Engagement Signals in
Content Ranking, February 2024.
URL <http://arxiv.org/abs/2402.06831>.
arXiv:2402.06831 [cs].

[Curmei et al., 2022]
Curmei, M., Haupt, A. A., Recht, B., and Hadfield-Menell, D.
Towards Psychologically-Grounded Dynamic Preference Models.
In Proceedings of the 16th ACM Conference on Recommender
Systems, pp.  35–48, 2022.

[Dean & Morgenstern, 2022]
Dean, S. and Morgenstern, J.
Preference Dynamics Under Personalized Recommendations.
In Proceedings of the 23rd ACM Conference on Economics
and Computation, pp.  795–816, Boulder CO USA, July 2022. ACM.
ISBN 978-1-4503-9150-4.
URL <https://dl.acm.org/doi/10.1145/3490486.3538346>.

[Desai, 2017]
Desai, N.
Uncertain Reward-Transition MDPs for Negotiable
Reinforcement Learning.

[Dietrich & List, 2013]
Dietrich, F. and List, C.
Where do preferences come from?
International Journal of Game Theory, 420
(3):0 613–637, August 2013.
ISSN 1432-1270.
URL <https://doi.org/10.1007/s00182-012-0333-y>.

[Dobbe et al., 2021]
Dobbe, R., Gilbert, T. K., and Mintz, Y.
Hard Choices in Artificial Intelligence, June 2021.
URL <http://arxiv.org/abs/2106.11022>.
arXiv:2106.11022 [cs, eess].

[Dwork et al., 2011]
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., and Zemel, R.
Fairness Through Awareness.
arXiv:1104.3913 [cs], November 2011.
URL <http://arxiv.org/abs/1104.3913>.
arXiv: 1104.3913.

[Elster, 1979]
Elster, J. (ed.).
Ulysses and the Sirens: Studies in Rationality and
Editions de la Maison des sciences de l'homme, Paris, 1979.

[Elster, 1983]
Elster, J.
Sour grapes: studies in the subversion of rationality.
Cambridge philosophy classics. Cambridge university press, New York
(N.Y.), 1983.
ISBN 978-1-107-14202-2 978-1-316-50700-1.

[Elster, 1985]
Elster, J.
Weakness of Will and the Free-Rider Problem.
Economics & Philosophy, 10 (2):0 231–265,
October 1985.
ISSN 1474-0028, 0266-2671.
Publisher: Cambridge University Press.

[Evans et al., 2015]
Evans, O., Stuhlmueller, A., and Goodman, N. D.
Learning the Preferences of Ignorant, Inconsistent Agents.
arXiv:1512.05832 [cs], December 2015.
URL <http://arxiv.org/abs/1512.05832>.
arXiv: 1512.05832.

[Everitt et al., 2021]
Everitt, T., Carey, R., Langlois, E. D., Ortega, P. A., and Legg, S.
Agent Incentives: A Causal Perspective.

[Everitt et al., 2021]
Everitt, T., Hutter, M., Kumar, R., and Krakovna, V.
Reward Tampering Problems and Solutions in Reinforcement
Learning: A Causal Influence Diagram Perspective.
arXiv:1908.04734 [cs], March 2021b.
URL <http://arxiv.org/abs/1908.04734>.
arXiv: 1908.04734.

[Everitt et al., 2023]
Everitt, T., Fox, J., Carey, R., MacDermott, M., Benthall, S., and Richens, J.
Incentives from a causal perspective.

[Farquhar et al., 2022]
Farquhar, S., Carey, R., and Everitt, T.
Path-Specific Objectives for Safer Agent Incentives.
arXiv:2204.10018 [cs, stat], April 2022.
URL <http://arxiv.org/abs/2204.10018>.
arXiv: 2204.10018.

[Fast et al., 2023]
Fast, N., Schroeder, J., Iyer, R., and Motyl, M.
Unveiling the Neely Ethics & Technology Indices, June 2023.

[Firth, 1952]
Firth, R.
Ethical Absolutism and the Ideal Observer.
Philosophy and Phenomenological Research, 120
(3):0 317–345, 1952.
ISSN 0031-8205.
URL <https://www.jstor.org/stable/2103988>.
Publisher: [International Phenomenological Society, Philosophy and
Phenomenological Research, Wiley].

[Franklin et al., 2022]
Franklin, M., Ashton, H., Gorman, R., and Armstrong, S.
Recognising the importance of preference change: A call for a
coordinated multidisciplinary research effort in the age of AI.
arXiv:2203.10525 [cs], March 2022.
URL <http://arxiv.org/abs/2203.10525>.
arXiv: 2203.10525.

[Frederick et al., 2002]
Frederick, S., Loewenstein, G., and O'Donoghue, T.
Time Discounting and Time Preference: A Critical Review.
Journal of Economic Literature, 400 (2):0
351–401, June 2002.
ISSN 0022-0515.

[Freeman et al., 2017]
Freeman, R., Zahedi, S. M., and Conitzer, V.
Fair and Efficient Social Choice in Dynamic Settings.
In Proceedings of the Twenty-Sixth International Joint
Conference on Artificial Intelligence, pp.  4580–4587, Melbourne,
Australia, August 2017. International Joint Conferences on Artificial
Intelligence Organization.
ISBN 978-0-9992411-0-3.
URL <https://www.ijcai.org/proceedings/2017/639>.

[Gabriel, 2020]
Gabriel, I.
Artificial Intelligence, Values and Alignment.
arXiv:2001.09768 [cs], January 2020.
URL <http://arxiv.org/abs/2001.09768>.
arXiv: 2001.09768.

[Gabriel et al., 2024]
Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V., Iqbal, H.,
Tomašev, N., Ktena, I., Kenton, Z., Rodriguez, M., El-Sayed, S., Brown, S.,
Akbulut, C., Trask, A., Hughes, E., Bergman, A. S., Shelby, R., Marchal, N.,
Griffin, C., Mateos-Garcia, J., Weidinger, L., Street, W., Lange, B.,
Ingerman, A., Lentz, A., Enger, R., Barakat, A., Krakovna, V., Siy, J. O.,
Kurth-Nelson, Z., McCroskery, A., Bolina, V., Law, H., Shanahan, M., Alberts,
L., Balle, B., de Haas, S., Ibitoye, Y., Dafoe, A., Goldberg, B., Krier, S.,
Reese, A., Witherspoon, S., Hawkins, W., Rauh, M., Wallace, D., Franklin, M.,
Goldstein, J. A., Lehman, J., Klenk, M., Vallor, S., Biles, C., Morris,
M. R., King, H., Arcas, B. A. y., Isaac, W., and Manyika, J.
The Ethics of Advanced AI Assistants, April 2024.
URL <http://arxiv.org/abs/2404.16244>.
arXiv:2404.16244 [cs].

[Gauci et al., 2019]
Gauci, J., Conti, E., Liang, Y., Virochsiri, K., He, Y., Kaden, Z., Narayanan,
V., Ye, X., Chen, Z., and Fujimoto, S.
Horizon: Facebook's Open Source Applied Reinforcement
Learning Platform.
arXiv:1811.00260 [cs, stat], September 2019.
URL <http://arxiv.org/abs/1811.00260>.
arXiv: 1811.00260.

[George, 2001]
George, D.
Preference pollution: how markets create the desires we
University of Michigan Press, Ann Arbor, 2001.
ISBN 978-0-472-11220-3.

[Griffin, 1986]
Griffin, J.
Well-Being: Its Meaning, Measurement, and Moral
Oxford, GB: Clarendon Press, 1986.

[Grüne-Yanoff & Hansson, 2009]
Grüne-Yanoff, T. and Hansson, S. O. (eds.).
Preference Change.
Springer Netherlands, Dordrecht, 2009.
ISBN 978-90-481-2592-0 978-90-481-2593-7.
URL <http://link.springer.com/10.1007/978-90-481-2593-7>.

[Hadfield-Menell & Hadfield, 2018]
Hadfield-Menell, D. and Hadfield, G.
Incomplete Contracting and AI Alignment.
arXiv:1804.04268 [cs], April 2018.
URL <http://arxiv.org/abs/1804.04268>.
arXiv: 1804.04268.

[Hadfield-Menell et al., 2016]
Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A.
Cooperative Inverse Reinforcement Learning.

[Halpern & Sanders, 2016]
Halpern, D. and Sanders, M.
Nudging by Government: Progress, Impact, & Lessons
Behavioral Science & Policy, 20 (2):0 53–65,
October 2016.
ISSN 2379-4607.
URL <https://doi.org/10.1177/237946151600200206>.
Publisher: SAGE Publications.

[Halpern & Kleiman-Weiner, 2018]
Halpern, J. Y. and Kleiman-Weiner, M.
Towards Formal Definitions of Blameworthiness, Intention, and
Moral Responsibility, October 2018.
URL <http://arxiv.org/abs/1810.05903>.
arXiv:1810.05903 [cs].

[Hammond et al., 2023]
Hammond, L., Fox, J., Everitt, T., Carey, R., Abate, A., and Wooldridge, M.
Reasoning about Causality in Games, January 2023.
URL <http://arxiv.org/abs/2301.02324>.
arXiv:2301.02324 [cs].

[Hansen & Jespersen, 2013]
Hansen, P. G. and Jespersen, A. M.
Nudge and the Manipulation of Choice: A Framework for the
Responsible Use of the Nudge Approach to Behaviour Change in
Public Policy.
European Journal of Risk Regulation, 40 (1):0
3–28, March 2013.
ISSN 1867-299X, 2190-8249.

[Hardt et al., 2022]
Hardt, M., Jagadeesan, M., and Mendler-Dünner, C.
Performative Power.
arXiv:2203.17232 [cs, econ], March 2022.
URL <http://arxiv.org/abs/2203.17232>.
arXiv: 2203.17232.

[Harsanyi, 1953]
Harsanyi, J. C.
Welfare Economics of Variable Tastes.
The Review of Economic Studies, 210 (3):0
204–213, 1953.
ISSN 0034-6527.
URL <https://www.jstor.org/stable/2295773>.
Publisher: [Oxford University Press, Review of Economic Studies,

[Hausman & Welch, 2009]
Hausman, D. and Welch, B.
Debate: To Nudge or Not to Nudge*.
Journal of Political Philosophy, 18:0 123–136,
November 2009.

[Hayek et al., 1941]
Hayek, F. A., White, L. H., and White, L. H.
The Pure Theory of Capital.

[Hazrati & Ricci, 2022]
Hazrati, N. and Ricci, F.
Recommender systems effect on the evolution of users’ choices
Information Processing & Management, 590
(1):0 102766, January 2022.
ISSN 03064573.

[Hendrycks et al., 2023]
Hendrycks, D., Mazeika, M., and Woodside, T.
An Overview of Catastrophic AI Risks, October 2023.
URL <http://arxiv.org/abs/2306.12001>.
arXiv:2306.12001 [cs].

[Hong et al., 2022]
Hong, J., Bhatia, K., and Dragan, A.
On the Sensitivity of Reward Inference to Misspecified
Human Models, December 2022.
URL <http://arxiv.org/abs/2212.04717>.
arXiv:2212.04717 [cs].

[Hong et al., 2023]
Hong, J., Dragan, A., and Levine, S.
Learning to Influence Human Behavior with Offline
Reinforcement Learning, June 2023.
URL <http://arxiv.org/abs/2303.02265>.
arXiv:2303.02265 [cs].

[Huszár et al., 2021]
Huszár, F., Ktena, S. I., O'Brien, C., Belli, L., Schlaikjer, A., and Hardt,
Algorithmic Amplification of Politics on Twitter.
arXiv:2110.11010 [cs], October 2021.
URL <http://arxiv.org/abs/2110.11010>.
arXiv: 2110.11010.

[Hyman & Wright, 1979]
Hyman, H. H. and Wright, C. R.
Education's Lasting Influence on Values.
Marketing Department, University of Chicago Press, 5801 S, 1979.
ERIC Number: ED187642.

[Irvine et al., 2023]
Irvine, R., Boubert, D., Raina, V., Liusie, A., Mudupalli, V., Korshuk, A.,
Liu, Z., Cremer, F., Assassi, V., Beauchamp, C.-C., Lu, X., Rialan, T., and
Beauchamp, W.
Rewarding Chatbots for Real-World Engagement with Millions
of Users, March 2023.
URL <http://arxiv.org/abs/2303.06135>.
arXiv:2303.06135 [cs].

[Jackson, 2014]
Jackson, M. O.
Mechanism Theory.
SSRN Electronic Journal, 2014.
ISSN 1556-5068.
URL <http://www.ssrn.com/abstract=2542983>.

[Jeon et al., 2020]
Jeon, H. J., Milli, S., and Dragan, A. D.
Reward-rational (implicit) choice: A unifying formalism for reward
learning, December 2020.
URL <http://arxiv.org/abs/2002.04833>.
arXiv:2002.04833 [cs].

[Kasirzadeh & Evans, 2023]
Kasirzadeh, A. and Evans, C.
User tampering in reinforcement learning recommender systems.
In AIES '23: Proceedings of the 2023 AAAI/ACM
Conference on AI, Ethics, and Society. Association for Computing
Machinery (ACM), August 2023.

[Kenton et al., 2021]
Kenton, Z., Everitt, T., Weidinger, L., Gabriel, I., Mikulik, V., and Irving,
Alignment of Language Agents, March 2021.
URL <http://arxiv.org/abs/2103.14659>.
arXiv:2103.14659 [cs].

[Kim et al., 2022]
Kim, D.-K., Riemer, M., Liu, M., Foerster, J. N., Everett, M., Sun, C.,
Tesauro, G., and How, J. P.
Influencing Long-Term Behavior in Multiagent Reinforcement
Learning, October 2022.
URL <http://arxiv.org/abs/2203.03535>.
arXiv:2203.03535 [cs].

[Kirk et al., 2023]
Kirk, H. R., Vidgen, B., Röttger, P., and Hale, S. A.
The Empty Signifier Problem: Towards Clearer Paradigms
for Operationalising "Alignment" in Large Language Models, November
URL <http://arxiv.org/abs/2310.02457>.
arXiv:2310.02457 [cs].

[Kleinberg et al., 2022]
Kleinberg, J., Mullainathan, S., and Raghavan, M.
The Challenge of Understanding What Users Want:
Inconsistent Preferences and Engagement Optimization.
arXiv:2202.11776 [cs], February 2022.
URL <http://arxiv.org/abs/2202.11776>.
arXiv: 2202.11776.

[Knox et al., 2013]
Knox, W. B., Stone, P., and Breazeal, C.
Training a Robot via Human Feedback: A Case Study.
In Hutchison, D., Kanade, T., Kittler, J., Kleinberg, J. M., Mattern,
F., Mitchell, J. C., Naor, M., Nierstrasz, O., Pandu Rangan, C., Steffen, B.,
Sudan, M., Terzopoulos, D., Tygar, D., Vardi, M. Y., Weikum, G., Herrmann,
G., Pearson, M. J., Lenz, A., Bremner, P., Spiers, A., and Leonards, U.
(eds.), Social Robotics, volume 8239, pp.  460–470. Springer
International Publishing, Cham, 2013.
ISBN 978-3-319-02674-9 978-3-319-02675-6.
URL <http://link.springer.com/10.1007/978-3-319-02675-6_46>.
Series Title: Lecture Notes in Computer Science.

[Kolodny, 2022]
Kolodny, N.
AI Safety and Preference Change, September 2022.
URL <https://www.youtube.com/watch?v=vsWTSeFq0kA>.

[Krakovna et al., 2019]
Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S.
Penalizing side effects using stepwise relative reachability.
arXiv:1806.01186 [cs, stat], March 2019.
URL <http://arxiv.org/abs/1806.01186>.
arXiv: 1806.01186.

[Krueger et al., 2020]
Krueger, D., Maharaj, T., and Leike, J.
Hidden Incentives for Auto-Induced Distributional Shift.
arXiv:2009.09153 [cs, stat], September 2020.
URL <http://arxiv.org/abs/2009.09153>.
arXiv: 2009.09153.

[Kulkarni & Neth, 2020]
Kulkarni, K. and Neth, S.
Social Choice with Changing Preferences: Representation
Theorems and Long-Run Policies, November 2020.
URL <http://arxiv.org/abs/2011.02544>.
arXiv:2011.02544 [cs].

[Lang et al., 2024]
Lang, L., Foote, D., Russell, S., Dragan, A., Jenner, E., and Emmons, S.
When Your AIs Deceive You: Challenges with Partial
Observability of Human Evaluators in Reward Learning, March 2024.
URL <http://arxiv.org/abs/2402.17747>.
arXiv:2402.17747 [cs, stat].

[Laufer & Nissenbaum, 2023]
Laufer, B. and Nissenbaum, H.
Algorithmic Displacement of Social Trust, 2023.

[Leike et al., 2018]
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S.
Scalable agent alignment via reward modeling: a research direction.
arXiv:1811.07871 [cs, stat], November 2018.
URL <http://arxiv.org/abs/1811.07871>.
arXiv: 1811.07871.

[Li et al., 2023]
Li, B. Z., Tamkin, A., Goodman, N., and Andreas, J.
Eliciting Human Preferences with Language Models, October
URL <http://arxiv.org/abs/2310.11589>.
arXiv:2310.11589 [cs].

[Lindner & El-Assady, 2022]
Lindner, D. and El-Assady, M.
Humans are not Boltzmann Distributions: Challenges and
Opportunities for Modelling Human Feedback and Interaction in
Reinforcement Learning, June 2022.
URL <http://arxiv.org/abs/2206.13316>.
arXiv:2206.13316 [cs, stat].

[List, 2022]
List, C.
Social Choice Theory.
In Zalta, E. N. and Nodelman, U. (eds.), The Stanford
Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford
University, winter 2022 edition, 2022.

[Locke, 1689]
Locke, J.
An Essay Concerning Human Understanding.
Oxford University Press, New York, 1689.

[Loewenstein, 2005]
Loewenstein, G.
Hot-cold empathy gaps and medical decision making.
Health Psychology: Official Journal of the Division of Health
Psychology, American Psychological Association, 240 (4S):0
S49–56, July 2005.
ISSN 1930-7810.

[Loewenstein & Angner, 2003]
Loewenstein, G. and Angner, E.
Predicting and indulging changing preferences.
In Time and decision: Economic and psychological perspectives
on intertemporal choice, pp.  351–391. Russell Sage Foundation, New York,
NY, US, 2003.
ISBN 978-0-87154-549-7.

[Loewenstein & Prelec, 1992]
Loewenstein, G. and Prelec, D.
Anomalies in Intertemporal Choice: Evidence and an
The Quarterly Journal of Economics, 1070 (2):0
573–597, 1992.
ISSN 0033-5533.
URL <https://www.jstor.org/stable/2118482>.
Publisher: Oxford University Press.

[Loewenstein et al., 2003]
Loewenstein, G., Read, D., and Baumeister, R. (eds.).
Time and decision: Economic and psychological perspectives on
intertemporal choice.
Time and decision: Economic and psychological perspectives on
intertemporal choice. Russell Sage Foundation, New York, NY, US, 2003.
ISBN 978-0-87154-549-7.
Pages: xiii, 569.

[MacAskill, 2022]
MacAskill, W.
What we owe the future.
Basic Books, Hachette Book Group, New York, NY, first edition
edition, 2022.
ISBN 978-1-5416-1862-6.
OCLC: 1314633519.

[McKinney et al., 2023]
McKinney, L., Duan, Y., Krueger, D., and Gleave, A.
On The Fragility of Learned Reward Functions, January 2023.
URL <http://arxiv.org/abs/2301.03652>.
arXiv:2301.03652 [cs].

[Milli, 2019]
Milli, S.
When a Better Human Model Means Worse Reward Inference,
URL <http://smithamilli.com/blog/predict-vs-inf/>.

[Milli et al., 2023]
Milli, S., Carroll, M., Wang, Y., Pandey, S., Zhao, S., and Dragan, A. D.
Engagement, User Satisfaction, and the Amplification of
Divisive Content on Social Media, September 2023.
URL <http://arxiv.org/abs/2305.16941>.
arXiv:2305.16941 [cs].

[Mishra, 2023]
Mishra, A.
AI Alignment and Social Choice: Fundamental Limitations
and Policy Implications, October 2023.
URL <http://arxiv.org/abs/2310.16048>.
arXiv:2310.16048 [cs].

[Moshirnia, 2017]
Moshirnia, A.
No Security through Obscurity: Changing Circumvention Law
to Protect Our Democracy against Cyberattacks.
Brooklyn Law Review, 830 (4):0 1279–1344,
<https://heinonline.org/HOL/P?h=hein.journals/brklr83   i=1317>.

[Moyers & Martin, 2006]
Moyers, T. B. and Martin, T.
Therapist influence on client language during motivational
interviewing sessions.
Journal of Substance Abuse Treatment, 300
(3):0 245–251, April 2006.
ISSN 0740-5472.

[Ng & Russell, 2000]
Ng, A. Y. and Russell, S. J.
Algorithms for Inverse Reinforcement Learning.
In Proceedings of the Seventeenth International
Conference on Machine Learning, ICML '00, pp.  663–670, San
Francisco, CA, USA, June 2000. Morgan Kaufmann Publishers Inc.
ISBN 978-1-55860-707-1.

[Nisan et al., 2007]
Nisan, N., Roughgarden, T., Tardos, E., and Vazirani, V. V. (eds.).
Algorithmic Game Theory.
Cambridge University Press, Cambridge, 2007.
ISBN 978-0-521-87282-9.

[Noggle, 2020]
Noggle, R.
The Ethics of Manipulation.
In Zalta, E. N. (ed.), The Stanford Encyclopedia of
Philosophy. Metaphysics Research Lab, Stanford University, summer 2020
edition, 2020.

[Ord, 2021]
Ord, T.
The precipice: existential risk and the future of humanity.
Bloomsbury Publishing, London, 2021.
ISBN 978-1-5266-0023-3.
OCLC: 1252948575.

[Ouyang et al., 2022]
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P.,
Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton,
F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike,
J., and Lowe, R.
Training language models to follow instructions with human feedback.
pp.  68, 2022.

[Parfit, 1982]
Parfit, D.
Personal Identity and Rationality.
Synthese, 530 (2):0 227–241, 1982.
ISSN 0039-7857.
URL <https://www.jstor.org/stable/20115799>.
Publisher: Springer.

[Parfit, 1984]
Parfit, D.
Reasons and persons.
Clarendon Press, Oxford, 1. issued in paperback (with corr.),
reprinted with further corr edition, 1984.
ISBN 978-0-19-824908-5.

[Parkes & Procaccia, 2013]
Parkes, D. C. and Procaccia, A. D.
Dynamic Social Choice: Foundations and Algorithms.

[Paul, 2014]
Paul, L. A.
Transformative experience.
Oxford University Press, Oxford, 1st ed edition, 2014.
ISBN 978-0-19-871795-9.
OCLC: ocn872342141.

[Paul, 2022]
Paul, L. A.
Choosing for Changing Selves.
The Philosophical Review, 1310 (2):0 230–235,
April 2022.
ISSN 0031-8108.
URL <https://doi.org/10.1215/00318108-9554756>.

[Paul & Sunstein, 2019]
Paul, L. A. and Sunstein, C. R.
'As Judged By Themselves': Transformative Experiences and
Endogenous Preferences, September 2019.
URL <https://papers.ssrn.com/abstract=3455421>.

[Perdomo et al., 2020]
Perdomo, J. C., Zrnic, T., Mendler-Dünner, C., and Hardt, M.
Performative Prediction.
arXiv:2002.06673 [cs, stat], April 2020.
URL <http://arxiv.org/abs/2002.06673>.
arXiv: 2002.06673.

[Peston, 1967]
Peston, M. H.
Chapter 17. Changing Utility Functions.
In Chapter 17. Changing Utility Functions, pp.  233–236. Princeton University Press, 1967.
ISBN 978-1-4008-7738-6.

[Pettigrew, 2019]
Pettigrew, R.
Choosing for Changing Selves.
Oxford University Press, 1 edition, December 2019.
ISBN 978-0-19-881496-2 978-0-19-185280-0.

[Pettigrew, 2022]
Pettigrew, R.
Nudging for Changing Selves.
SSRN Electronic Journal, 2022.
ISSN 1556-5068.
URL <https://www.ssrn.com/abstract=4025214>.

[Pickett‐Baker & Ozaki, 2008]
Pickett‐Baker, J. and Ozaki, R.
Pro‐environmental products: marketing influence on consumer
purchase decision.
Journal of Consumer Marketing, 250 (5):0
281–293, January 2008.
ISSN 0736-3761.
URL <https://doi.org/10.1108/07363760810890516>.
Publisher: Emerald Group Publishing Limited.

[Pollak, 1968]
Pollak, R. A.
Consistent Planning.
The Review of Economic Studies, 350 (2):0
201–208, 1968.
ISSN 0034-6527.
URL <https://www.jstor.org/stable/2296548>.
Publisher: [Oxford University Press, Review of Economic Studies,

[Pollak, 1978]
Pollak, R. A.
Endogenous Tastes in Demand and Welfare Analysis.
The American Economic Review, 680 (2):0
374–379, 1978.
ISSN 0002-8282.
URL <https://www.jstor.org/stable/1816724>.
Publisher: American Economic Association.

[Rafailidis & Nanopoulos, 2016]
Rafailidis, D. and Nanopoulos, A.
Modeling Users Preference Dynamics and Side Information in
Recommender Systems.
IEEE Transactions on Systems, Man, and Cybernetics: Systems,
460 (6):0 782–792, June 2016.
ISSN 2168-2216, 2168-2232.
URL <http://ieeexplore.ieee.org/document/7194815/>.

[Regan & Boutilier, 2010]
Regan, K. and Boutilier, C.
Robust Policy Computation in Reward-Uncertain MDPs Using
Nondominated Policies.
Proceedings of the AAAI Conference on Artificial Intelligence,
240 (1):0 1127–1133, July 2010.
ISSN 2374-3468, 2159-5399.
URL <https://ojs.aaai.org/index.php/AAAI/article/view/7740>.

[Ribeiro et al., 2023]
Ribeiro, M. H., Veselovsky, V., and West, R.
The Amplification Paradox in Recommender Systems, February
URL <http://arxiv.org/abs/2302.11225>.
arXiv:2302.11225 [cs].

[Roijers et al., 2013]
Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R.
A Survey of Multi-Objective Sequential Decision-Making.
Journal of Artificial Intelligence Research, 48:0
67–113, October 2013.
ISSN 1076-9757.
URL <https://www.jair.org/index.php/jair/article/view/10836>.

[Rosati, 2013]
Rosati, C. S.
The Story of a Life.
Social Philosophy and Policy, 300 (1-2):0
21–50, January 2013.
ISSN 0265-0525, 1471-6437.

[Russell, 1998]
Russell, S.
Learning agents for uncertain environments (extended abstract).
In Proceedings of the eleventh annual conference on
Computational learning theory, pp.  101–103, Madison Wisconsin USA, July
1998. ACM.
ISBN 978-1-58113-057-7.
URL <https://dl.acm.org/doi/10.1145/279943.279964>.

[Russell, 2019]
Russell, S. J.
Human compatible: artificial intelligence and the problem of
Business book summary. Viking, New York, New York?, 2019.
ISBN 978-0-525-55861-3.
OCLC: 1113410915.

[Samuelson, 1937]
Samuelson, P. A.
A Note on Measurement of Utility.
The Review of Economic Studies, 40 (2):0
155–161, 1937.
ISSN 0034-6527.
URL <https://www.jstor.org/stable/2967612>.
Publisher: [Oxford University Press, Review of Economic Studies,

[Sanna Passino et al., 2021]
Sanna Passino, F., Maystre, L., Moor, D., Anderson, A., and Lalmas, M.
Where To Next? A Dynamic Model of User Preferences.
In Proceedings of the Web Conference 2021, pp.  3210–3220, Ljubljana Slovenia, April 2021. ACM.
ISBN 978-1-4503-8312-7.
URL <https://dl.acm.org/doi/10.1145/3442381.3450028>.

[Schelling, 1978]
Schelling, T. C.
Egonomics, or the Art of Self-Management.
The American Economic Review, 680 (2,):0
290–294, 1978.
URL <http://www.jstor.org/stable/1816707>.

[Schelling, 1984]
Schelling, T. C.
Self-Command in Practice, in Policy, and in a Theory of
Rational Choice.
American Economic Review, 740 (2):0 1–11,
URL <https://ideas.repec.org//a/aea/aecrev/v74y1984i2p1-11.html>.
Publisher: American Economic Association.

[Schelling, 1985]
Schelling, T. C.
Enforcing Rules on Oneself.
The Journal of Law, Economics, and Organization, 10
(2):0 357–374, October 1985.
ISSN 8756-6222.
URL <https://doi.org/10.1093/oxfordjournals.jleo.a036896>.

[Shah et al., 2019]
Shah, R., Gundotra, N., Abbeel, P., and Dragan, A. D.
On the Feasibility of Learning, Rather than Assuming, Human
Biases for Reward Inference.
arXiv:1906.09624 [cs, stat], June 2019a.
URL <http://arxiv.org/abs/1906.09624>.
arXiv: 1906.09624.

[Shah et al., 2019]
Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P., and Dragan, A.
Preferences Implicit in the State of the World.
arXiv:1902.04198 [cs, stat], April 2019b.
URL <http://arxiv.org/abs/1902.04198>.
arXiv: 1902.04198.

[Sharma et al., 2023]
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R.,
Cheng, N., Durmus, E., Hatfield-Dodds, Z., Johnston, S. R., Kravec, S.,
Maxwell, T., McCandlish, S., Ndousse, K., Rausch, O., Schiefer, N., Yan, D.,
Zhang, M., and Perez, E.
Towards Understanding Sycophancy in Language Models, October
URL <http://arxiv.org/abs/2310.13548>.
arXiv:2310.13548 [cs, stat].

[Siththaranjan et al., 2023]
Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D.
Distributional Preference Learning: Understanding and
Accounting for Hidden Context in RLHF, December 2023.
URL <http://arxiv.org/abs/2312.08358>.
arXiv:2312.08358 [cs, stat].

[Steele & Stefánsson, 2020]
Steele, K. and Stefánsson, H. O.
Decision Theory.
In Zalta, E. N. (ed.), The Stanford Encyclopedia of
Philosophy. Metaphysics Research Lab, Stanford University, winter 2020
edition, 2020.

[Stigler & Becker, 1977]
Stigler, G. J. and Becker, G. S.
De Gustibus Non Est Disputandum.
pp.  16, 1977.

[Stray et al., 2021]
Stray, J., Adler, S., and Hadfield-Menell, D.
What are you optimizing for? Aligning Recommender Systems with
Human Values.
pp.  7, 2021.

[Strohmaier & Messerli, 2024]
Strohmaier, D. and Messerli, M.
Preference Change.
Cambridge University Press, 1 edition, January 2024.
ISBN 978-1-00-918186-0 978-1-00-947579-2 978-1-00-918185-3.

[Strotz, 1955]
Strotz, R. H.
Myopia and Inconsistency in Dynamic Utility Maximization.
The Review of Economic Studies, 230 (3):0
165–180, January 1955.
ISSN 0034-6527.
URL <https://doi.org/10.2307/2295722>.

[Subramani et al., 2024]
Subramani, R., Williams, M., Heitmann, M., Holm, H., Griffin, C., and Skalse,
On The Expressivity of Objective-Specification Formalisms
in Reinforcement Learning, February 2024.
URL <http://arxiv.org/abs/2310.11840>.
arXiv:2310.11840 [cs].

[Susser et al., 2018]
Susser, D., Roessler, B., and Nissenbaum, H.
Online Manipulation: Hidden Influences in a Digital World,
December 2018.
URL <https://papers.ssrn.com/abstract=3306006>.

[Sutton & Barto, 2018]
Sutton, R. S. and Barto, A.
Reinforcement learning: an introduction.
Adaptive computation and machine learning. The MIT Press, Cambridge,
Massachusetts, nachdruck edition, 2018.
ISBN 978-0-262-19398-6.

[Taylor et al., 2016]
Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A.
Alignment for Advanced Machine Learning Systems.
pp.  25, 2016.

[Teodorescu, 2019]
Teodorescu, M.
Protected Attributes and ‘Fairness through Unawareness’.

[Thaler, 2018]
Thaler, R. H.
Nudge, not sludge.
Science, 3610 (6401):0 431–431, August 2018.
URL <https://www.science.org/doi/10.1126/science.aau9241>.
Publisher: American Association for the Advancement of Science.

[Thaler & Sunstein, 2008]
Thaler, R. H. and Sunstein, C. R.
Nudge: Improving decisions about health, wealth, and
Nudge: Improving decisions about health, wealth, and happiness.
Yale University Press, New Haven, CT, US, 2008.
ISBN 978-0-300-12223-7.
Pages: x, 293.

[Thorburn, 2022]
Thorburn, L.
How Platform Recommenders Work, November 2022.

[Thorburn, 2023]
Thorburn, L.
Making Amplification Measurable, May 2023.

[Thorburn et al., 2022]
Thorburn, L., Stray, J., and Bengani, P.
What Will “Amplification” Mean in Court?, 2022.

[Tien et al., 2023]
Tien, J., He, J. Z.-Y., Erickson, Z., Dragan, A. D., and Brown, D. S.
Causal Confusion and Reward Misidentification in
Preference-Based Reward Learning, March 2023.
URL <http://arxiv.org/abs/2204.06601>.
arXiv:2204.06601 [cs].

[Ullmann-Margalit, 2006]
Ullmann-Margalit, E.
Big Decisions: Opting, Converting, Drifting.
pp.  16, 2006.

[Velleman, 1991]
Velleman, J. D.
Well-Being And Time.
Pacific Philosophical Quarterly, 720 (1):0
48–77, 1991.
Publisher: Wiley Periodicals.

[von Weizsäcker, 1971]
von Weizsäcker, C. C.
Notes on endogenous change of tastes.
Journal of Economic Theory, 30 (4):0 345–372,
December 1971.
ISSN 0022-0531.

[Ward et al., 2023]
Ward, F. R., Everitt, T., Toni, F., and Belardinelli, F.
Honesty Is the Best Policy: Defining and Mitigating AI

[Ward et al., 2024]
Ward, F. R., MacDermott, M., Belardinelli, F., Toni, F., and Everitt, T.
The Reasons that Agents Act: Intention and Instrumental
Goals, February 2024.
URL <http://arxiv.org/abs/2402.07221>.
arXiv:2402.07221 [cs].

[Warnell et al., 2018]
Warnell, G., Waytowich, N., Lawhern, V., and Stone, P.
Deep TAMER: Interactive Agent Shaping in High-Dimensional
State Spaces.
arXiv:1709.10163 [cs], January 2018.
URL <http://arxiv.org/abs/1709.10163>.
arXiv: 1709.10163.

[Wylie, 2020]
Wylie, C.
Mindf*ck, Cambridge Analytica and the Plot to Break
Penguin Random House, 2020.

[Xie et al., 2020]
Xie, A., Losey, D. P., Tolsma, R., Finn, C., and Sadigh, D.
Learning Latent Representations to Inﬂuence Multi-Agent
pp.  14, 2020.

[Yudkowsky, 2004]
Yudkowsky, E.
Coherent Extrapolated Volition.

[Zhi-Xuan et al., 2024]
Zhi-Xuan, T., Carroll, M., Franklin, M., and Ashton, H.
Beyond Preferences in AI Alignment.
Forthcoming, 2024.

[Ziebart et al., 2010]
Ziebart, B. D., Bagnell, J. A., and Dey, A. K.
Modeling Interaction via the Principle of Maximum Causal
pp.  8, 2010.

§ DR-MDP FORMALISM FAQ

As we are most interested in analyzing implications of changing preferences on notions of optimality, in this work we simply assume that the human's reward functions are given to us as part of the specification of a DR-MDP.
However, unlike MDPs, for which which reward functions could be something as “objective” as the score in a game, DR-MDPs are rooted in the idea of mutable human wants—which begs the question of how exactly we conceive of them.

§.§ What is a human's “reward function”? Why model it as if it's changing?

We broadly think of a human's reward function according to the following informal definition:

Let $\mathcal{L}$ be a state-of-the-art reward learning technique, and $\theta$ be the current cognitive state of the human. A human's reward function under $\theta$ (i.e., $R_\theta(s, a, s')$) is the output of $\mathcal{L}$ when used to learn how they currently would evaluate any transition of the form $(s, a, s')$.

Human reward functions (as defined) will change. Unless we will someday develop a reward learning technique which learns the “one true reward function” of a person (we provide reasons for skepticism in <Ref>), it seems like if one considers any reward learning technique run at a different times (in which the person has a different cognitive state), the evaluations of the same transitions may change, meaning we will have multiple reward functions on our hands, corresponding to different cognitive states of the human. While this depends on the setting, it seems clear to be the case for almost any domain if one allows sufficient time to pass between reward learning runs (e.g., performing reward learning on the same person 80 years apart). Our framework is built on this premise.

Cognitive states and reward parameterizations. In our work, we use the term “cognitive states” interchangeably with “reward parameterizations” (which is how we introduce $\theta$ in <Ref>). This correspondence is more clear in light of the above: a reward function (or equivalently, reward parameterization) can be treated synonymously to the output of a reward learning technique under a certain cognitive state.

§.§ What counts as cognitive states, and what is their relationship to preferences?

In the same way that a “state” in an MDP contains things in the external environment that are relevant for the purposes of reward (and hence, for decision-making), we think of the “cognitive state” in a DR-MDP as containing aspects of the “internal to the human” that affect their evaluation of a transition (potentially including aspects of their preferences, values, beliefs, emotional state, etc.).

In practice, reward learning will pick up on cognitive biases, “visceral factors”, and beliefs. Human cognitive biases, transitory wants, emotions, and “visceral factors” (as discussed by Loewenstein & Angner, 2003) are picked up by existing reward learning techniques: for example,
one may infer that people “prefer” to click on clickbait (as in <Ref>), that an unskilled chess players is intentionally trying to lose [Milli, 2019], that humans might prefer indulging in temptation—e.g. eating a donut even though they initially said they wouldn't want to [Evans et al., 2015]—or that people prefer sycophantic responses from their chatbots [Sharma et al., 2023]. Ultimately, one of the main issues is that people are not Boltzmann-rational with respect to their “one true reward function” when providing reward feedback, as argued in Lindner & El-Assady, 2022. Additionally, people do not have full information when giving feedback. Despite this, Boltzmann-rationality with full information is generally assumed by reward learning techniques.

Using the term “cognitive states” instead of “preferences”.
Based on the above, reward functions learned by existing reward learning will not only depend on the person's preferences, but also on other factors of the person's current cognitive state. By not explicitly separating such factors from preferences, reward learning techniques will conflate them all in their learned reward representations, making it more appropriate to say that they learn reward functions which correspond to the person's current cognitive state, rather than their current preferences. While one could potentially fit such factors in the preference framework as “instantaneous preferences” (e.g., when I'm satiated, I have an instantaneous dispreference for food), this seems more disingenuous than the more generic framing of cognitive states, which allows people to make their own distinctions between components of cognitive states, i.e. what the boundaries are between preferences, values, beliefs, and “visceral factors” (that are usually a topic of heated debate).

§.§ Should visceral factors, satiation, and belief change count as reward changes?

As discussed above, current reward learning paradigms do not explicitly account for many factors which underlie human feedback, such as visceral factors [Loewenstein, 2005], satiation effects [Loewenstein et al., 2003], or belief change [Lang et al., 2024]. Insofar as this is the case, using <Ref> would lead one to call something a reward function change even if just one of these other aspects of the user's cognitive states has changed (and not preferences proper, however one may define them).
Indeed, one may argue that changes in the reward that are only due to biases, instantaneous visceral factors, or satiation effects shouldn't count as “true reward change”. Similar points may also be made about beliefs.
Whether this is reasonable is related to long-standing questions as to whether transient factors should be considered changes in “tastes” or preferences [Harsanyi, 1953, Stigler & Becker, 1977], and whether changes in “derived” preferences should be treated differently from “fundamental” preference changes [Dietrich & List, 2013].
However, to not be considered reward changes, such factors should also not be present in the reward function, requiring one to fully disambiguate them from preferences.

DR-MDPs are agnostic to what should count as reward changes. Despite the above, DR-MDPs remain agnostic to what should count as reward changes. Consider <Ref>: whatever disambiguation the current state-of-the-art reward learning technique is able to do will be the basis of what counts as reward change. Indeed, it seems reasonable to us that visceral factors, beliefs, or satiation effects should generally not classify as reward function change: if the human's preferences over the behavior of the AI system have only changed as a result of a change in beliefs about the world, it seems strange to model this as a “reward function change”. Similarly, if a human's instantaneous preferences appear to change because they are now satiated (e.g. not wanting the AI assistant to serve them breakfast right after eating breakfast), it seems debatable whether this should be considered as a reward change. While we think that modeling factors such as e.g. belief change separately is important, we'd like to stress that any attempt to do so will still require to take normative stances: for example, in modeling human beliefs separately (e.g. as a part of the state), one would likely want to choose $U(\xi)$ such that the AI system cannot worsen the user's beliefs to increase it, e.g. by deceiving the user [Lang et al., 2024]. For example, one could choose a $U(\xi)$ that evaluates all transitions based on how they would be evaluated under “correct” beliefs, rather than those that the user holds. More broadly, we think that the task of disambiguating all factors from preference is very challenging, as discussed in <Ref>, meaning that we may be forced to conflate preferences with the aspects of the cognitive state which are hard to model, leading to slightly unsatisfying notions of “reward change” (for lack of a better modeling approach).

§.§ Why not assume access to the human's “true reward function”?

Why do we choose to focus on reward functions as they would be learned by reward learning techniques (as discussed in <Ref>), rather than considering a person's “true reward function”?
Indeed, as long as the reward function used is non-Markovian, it will trivially be able to represent any “all-things-considered” notion of optimality $U(\xi)$ that we may be trying to target (as shown in <Ref>).
Ultimately, the reason boils down to the fact that we expect the “true reward function” representing what a person would want the AI system to optimize in aggregate across their different selves (assuming such an object even meaningfully exists) to not be accessible to us in practice.

Is there even such as thing as a “true reward function” for a person? For a “true reward function” to exist would require that there exist a “correct” choice of $U(\xi)$, which is questionable. The analysis in our work supports the claim that there may in fact not be a single, unambiguously correct choice of $U(\xi)$, and so does prior work in philosophy and beyond [Parfit, 1982, Paul, 2014, Pettigrew, 2019, Zhi-Xuan et al., 2024].[We gloss over and try to remain agnostic to possible interpretations as to the nature of such a “true reward function”.]
Regardless, from here on, we will argue as if a “true reward function” does in fact exist, treating it as a useful abstraction.

Conditions required to access the “true reward function” of a person. If we wanted to obtain the “true reward function” via reward learning, rather than some distorted and mispecfied version of it, it seems like we would need one of the following two conditions to hold (at the very least, approximately):

* One's reward learning technique would need to extrapolate from the time-inconsistent and biased feedback the human provides to recover their “true reward function”, or

* The human would need to provide feedback directly consistent with their “true reward function”.

Perfect reward learning is impossible, and there is no scalable approach to approximating it. The first condition would require developing human feedback models that “explain away” preference change and feedback biases more broadly [Evans et al., 2015, Hong et al., 2022],[Note that it may be argued that having changing preferences is a form of irrationality itself [Elster, 1979], although this is somewhat contested [Bruckner, 2009].] by perfectly specifying how observed human feedback should be used as evidence of a particular “true reward function” (or multiple, if one allows for the “true reward function” to change). The most commonly used model is Boltzmann rationality, also known as the Bradley-Terry model [Bradley & Terry, 1952]. However, this model is clearly wrong [Lindner & El-Assady, 2022]. More generally, it seems challenging to say the least to explicitly hardcode full models of “human bias” mathematically, as reflected by the limitations of current reward learning methods [McKinney et al., 2023, Tien et al., 2023, Casper et al., 2023]. Moreover, learning human biases has also been shown to be impossible in general [Christiano, 2015, Armstrong & Mindermann, 2019]. While some have tried to approximately learn both human biases and rewards jointly [Shah et al., 2019], this seems challenging even under the favorable assumptions they consider.

Perfect cognitive states are unreachable. As “debiasing” a person's feedback to recover their “true reward function” seems challenging, the alternative would be to place the person in a situation in which they would give unbiased[Note that we're using the term “bias” broadly, indicating any deviation from the feedback they would give according to their “true reward function”.] feedback about what they truly want.
Some works have discussed which conditions would necessary for this to happen: in particular, Yudkowsky, 2004 proposes that the ideal goal is obtain the preferences of the person “if [they] knew more, thought faster, were more the people [they] wished [they] were, had grown up farther together”.[Yudkowsky, 2004 talks about Coherent Extrapolated Volition in terms of collective preferences and values, but this same line of thinking can be applied to an individual.]
This perspective is similar to ideal observer theory, according to which things should be evaluated as if we were “ideal observers” [Firth, 1952], which in the words of Brandt, 1959 are “fully informed and vividly imaginative, impartial, in a calm frame of mind and otherwise normal”. However, obtaining such idealized forms of “unbiased” feedback seems even more unrealizable than the prior case: the idealized cognitive states that such evaluations would require to don't seem realizable in the real world—indeed Firth, 1952 goes as far as saying that they should be omniscient and omnipercipient.

Reward functions learned by reward learning which (appear to) change are the best we('ll) have. Therefore, for all intents and purposes, it seems plausible that the “true reward function” of a person will not be directly accessible, even if there is such a thing.
It seems like there will always be a gap between the reward functions that we learn with reward learning techniques, and the one “true reward function” of the person, which will give rise to reward function change (or at least the appearance of it).
As discussed above, this gap could arise from not correctly accounting for cognitive biases or other factors. However, another possibility is that, even with “perfect reward learning”, there would still be some inherent preference change that cannot be reduced to a single reward function without fully putting history in the state (as discussed in <Ref>).[This would mean that there isn't a single “true reward function” for a person across all cognitive states, but maybe there is a “true reward function” for each cognitive state, and the differences across cognitive states are not easily explainable in terms of a single reward function. Even conceptually, it's maybe unclear what this would look like. See the discussion on strategic voting from <Ref> for related points.]
In either case, one will have to contend respectively with the appearance, or deep the reality of changing preferences, and the ambiguity that arises from it.
While one may hope to eventually render inconsequential the gap between one's learned reward function and the “true reward function” through ever-improving approximations, the lack of compelling proposals for scalable techniques casts doubt on the feasibility of this aspiration.

Ultimately, while one may be tempted to assume access to the “one true reward function” to simplify our analysis, it is not only unrealistic but also sidesteps the core problem we aim to study: what to do in light of the fact that we cannot access such a reward function, and we will almost certainly continue to have to contend with (an appearance of) changing preferences.

§.§ Can't one put $\theta$ in the state and use a single context-dependent reward function?

On first impression, one might wonder whether one may be able to resolve the normative questions highlighted by the DR-MDP formalism by
placing treating the person's reward parameterization as a component of the state (e.g. have an augmented state $\dot{s}_t = (s_t, \theta_t)$), and having a single reward function depend on it (e.g. have the reward function be of the form $R(\dot{s}_t, a_t,\dot{s}_{t+1})$). This would be equivalent to using a Factored MDP [Boutilier et al., 2000] with the augmented state $\dot{s}$.
However, modeling dynamic reward settings as Factored MDPs of this kind lends itself to two possible interpretations:

* Factored MDPs prescribe that we should use the Real-time Reward (from <Ref>) as the optimization objective

* Factored MDPs leave underdetermined what optimization objective should be used

We address both interpretations in turn, showing that the first is suspect (and potentially misleading), and the second is unhelpful (as this is essentially the same as what DR-MDPs do—but at least DR-MDPs provide better tools for reasoning about pros and cons of different objectives).

Interpretation 1: Factored MDPs prescribe using the Real-time Reward objective.
Consider the Factored MDP, and the reward value $r_t$ for timestep $t$: $r_t = R(\dot{s}_t, a_t, \dot{s}_{t+1}) = R(s_t, \theta_t, a_t, s_{t+1}, \theta_{t+1})$. How should $\theta_t$ and $\theta_{t+1}$ be used to determine the reward for timestep $t$, when they may differ in evaluation for the transition $(s_t, a_t, s_{t+1})$? The most straightforward interpretation—in the language of DR-MDPs—may be that $r_t$ should be equivalent to $R_{\theta_t}(s_t, a_t, s_{t+1})$, i.e. each state-action transition $(s_t, a_t)$ should be evaluated according to the reward parameterization which corresponds to timestep $t$. However, note that this is identical to choosing to optimize the Real-Time Reward objective $\sum_t R_{\theta_t}(s_t, a_t, s_{t+1})$ from <Ref>. In addition to purely philosophical arguments against this choice of objective [Kolodny, 2022], we discuss the objective's limitations at length in <Ref>. Moreover, note that this is just one possible stance on what notion of optimality is prescribed by Factored MDPs: this interpretation of the prescription of Factored MDPs arbitrarily chooses to ignore $\theta_{t+1}$. The language of DR-MDPs is more flexible, and allows us to describe 7 other options of objectives in <Ref>, most of which seem superior to Real-time Reward in terms of the influence incentives which they lead to. So even if one were to interpret Factored MDPs as prescribing usage of (what we call) the Real-time Reward objective, there are reasons to doubt the reasonableness of this prescription.

Interpretation 2: both Factored MDPs and DR-MDPs need a choice of $U(\xi)$ to resolve normative questions, but DR-MDPs offer a better formal language to reason about them.
Alternatively, one may interpret Factored MDPs as leaving underdetermined what optimality should consist of (i.e. which $\theta$ to consider in evaluating each transition), potentially because their formalism was designed for static-reward settings. If so, that seems similar to what DR-MDPs without a choice of $U(\xi)$ prescribe, i.e. essentially nothing. However, precisely because Factored MDPs were designed for static-reward settings, they don't provide a formal language for describing the different possible objectives that one may care about in dynamic-reward settings. DR-MDPs account for the fact that it may be meaningful to evaluate a transition $(s_t, a_t)$ by cognitive states $\theta$ other than $\theta_t$ (even ones that were not associated with the state $s_t$ at timestep $t$), and provide notation for reward functions “from the point of view” of different cognitive states, i.e. $R_\theta(\cdot)$.
This captures the intuition that people have preferences about the actions that they may undertake at times in which they have different cognitive states $\theta$ than the ones they currently have; moreover, those preferences may be quite important, such as in the case for e.g. one's negative evaluation—from the point-of-view of not currently being subject of manipulation—of a hypothetical scenario in which they are happily manipulated.
By providing a better formal language for reasoning about the normative choices entailed by dynamic reward settings, DR-MDPs are therefore more conceptually suited and helpful for reasoning about tradeoffs between different optimization objectives than Factored MDPs.

Regardless of the interpretation one takes about the consequences of putting $\theta$ in the state, this shows that this move does little to address the central question of our work, and/or to help choose optimization objectives which do not lead to undesirable influence.

§.§ How would one learn all reachable reward functions of a person and their dynamics?

Here we discuss how one could obtain $\Theta$ and learn its dynamics for realistic environments.

As long as one assumes that it is possible for different people to have the same cognitive states (at least insofar as is necessary for specifying their reward functions), and that the transition dynamics of cognitive states is shared across people, it should be possible to simply learn
reachable reward functions by performing reward learning with a sufficient number of people which are sufficiently diverse. The dynamics of $\Theta$ (and $\mathcal{S}$) could similarly be learned with a sufficiently large dataset of trajectories (for which $\theta$s are observed), assuming there is sufficient coverage. As a high level approach which leverages these assumptions, see <Ref>.

The closest procedure we are aware of in the literature is that in Carroll et al., 2022: in simulated experiments, the authors learns user preferences and their dynamics under similar assumptions to the ones mentioned above. Unlike the setup from <Ref>, the relationship between $\theta$ and $R_\theta$ is assumed to be known (based on Boltzmann rationality), but $\theta$ is not assumed to be observable, and has to be inferred from the user's history.

Learning reward functions and their dynamics

$\mathcal{H} = \{h^{(i)} \}$, set of humans  currently in the environment, with different states and cognitive states; $\mathcal{D} = \{ \xi^{(i)} \}$, set of trajectories (of potentially different lengths) across different humans in the environment.
$\Theta = \{ \theta^{(i)} | \theta^{(i)} \text{ is the current cognitive state of } h^{(i)} \in \mathcal{H} \}$ (Assumes that $\mathcal{H}$ has coverage over $\Theta$)
$\hat{R}_{\theta} \leftarrow$ Reward-Learning$(\{ h^{(i)} | \theta^{(i)} == \theta \})$ (Infer the reward function common to all humans with $\theta^{(i)} == \theta$)
$\hat{\mathcal{T}}$ $\leftarrow$ approximate transition dynamics $\mathcal{T}$ from $\mathcal{D}$ (Assumes $\mathcal{D}$ covers the space of transitions).
Output: Set of reward functions $\{\hat{R}_\theta \mid \theta \in \Theta\}$, and transition function $\hat{\mathcal{T}}$ defining the DR-MDP.

§.§ Reducing DR-MDPs with a notion of optimality $U(\xi)$ to MDPs

A specific notion of optimality $U(\xi)$ for a DR-MDP can be thought of as a “flattening” of the different reward functions of the DR-MDP into one. Consequently, it may be unsurprising that once one has settled on a choice of $U(\xi)$ for a DR-MDP, one can express the same notion of optimality in a corresponding MDP (reducing the DR-MDP problem to a standard MDP one).
This implies that it's always possible to re-express a changing reward problem as a static reward problem, once one has settled on a notion of optimality $U(\xi)$. That being said, this does not help with determining $U(\xi)$, i.e. what acceptable notions of optimality should be in cases in which rewards change.[Indeed, the fact that Markovian reward is not sufficient to represent many preference orderings between policies is potentially a reason to doubt the value of basing alignment formalisms on reward functions all together, as argued by Subramani et al., 2024.] This is the same problem we discussed in <Ref> and elsewhere.

Reducing a DR-MDP to an MDP.

For any notion of optimality $U(\xi)$ in a DR-MDP $\mathcal M = (\mathcal S, \Theta, \mathcal{A}, \mathcal{T}, R_\theta)$, there exists a choice of MDP $\dot{\mathcal M} = (\mathcal{\dot{S}}, \mathcal{A}, \mathcal{\dot{T}}, \dot{R})$ such that a policy is optimal with respect to $U(\xi)$ in $\mathcal M$ if and only if it is optimal in $\dot{\mathcal M}$.

Given the DR-MDP $\mathcal M$ and the trajectory-level utility function $U(\xi)$, one can construct the MDP $\dot{\mathcal M}$ as follows:

* The state space $\mathcal{\dot{S}}$ is such that each state is augmented with the history of the interactions up until reaching that state: $\dot{s}_t = (s_0, \theta_0, a_0, \dots, s_t, \theta_t)$ for $t>0$, and $\dot{s}_0 = (s_0, \theta_0)$.

* The reward function $\dot{R}$ is set to be $0$ everywhere, except for terminal states, in which case the reward for exiting the MDP is set to $U(\xi)$ for the resulting trajectory $\xi = (\dot{s}_{T-1}, a_{T-1}) = (s_0, \theta_0, a_0, \dots, s_{T-1}, \theta_{T-1}, a_{T-1})$. Note that one can determine whether a state is terminal by checking whether it corresponds to timestep $T-1$, which can be determined by the number of previous timesteps in the augmented state. Formally:

$$\dot{R}(\dot{s}_t, a_t) = \left\{\begin{array}{cl}
U(\xi) & \text{if } t = T - 1 \\
0 & \text{otherwise}
\end{array}\right.$$

* The transition function $\mathcal{\dot{T}}$ accounts for the augmented state space, appending to the state $\dot{s}_{t+1}$ at each timestep $t+1$, the new $(a_t, s_{t+1}, \theta_{t+1})$ triplet.

Note that in the resulting MDP $\dot{\mathcal M}$, any trajectory $\xi$ will be scored in the same way as the original DR-MDP $\mathcal{M}$ when considering the notion of optimality specified by $U(\xi)$. This means that policy will be optimal for $\dot{\mathcal M}$ if and only if $\mathcal{M}$.

The specific construction of $\dot{\mathcal M}$ in the proof above relies on putting the history in the state (to allow for choices of $U(\xi)$ which can arbitrarily depend on history). However, for certain choices of $U(\xi)$ this might be unnecessary: e.g. the real-time reward objective $U_\text{RT}(\xi)$ can be expressed by only augmenting the state space with the current reward parameterization (i.e. $\dot{s}_t = (s_t, \theta_t)$), rather than the whole history—and setting $\dot R$ to reward each MDP transition $(\dot{s}_t, a_t)$ as $R_{\theta_t}(s_t, a_t)$ where the $\theta_t$ and $s_t$ are unpacked from $\dot{s}_t$.
We leave a more general formal analysis of which choices of $U(\xi)$ can have their MDP reduction keep the Markov property without putting the history in the state to future work.

§.§ Unidentifiability of “correct” normative resolutions by DR-MDP structure alone

While the current paradigm for AI generally makes use of an optimality criterion which determines agent behavior solely based on the mathematical structure of the problem at hand (namely, $U(\xi) = \sum_{t=0}^{H-1} R(s_t, a_t, s_{t+1})$, which does not account for preference changes), we argue that it may not be possible to guarantee returning the “normatively correct” behavior simply from the mathematical structure of learned reward functions in changing reward settings, using an unidentifiability argument.

Even if there exists a unique choice of “normatively correct” behavior in a normatively ambiguous DR-MDP, such “correct” behavior may not be identifiable from the mathematical structure alone of the DR-MDP, e.g. by using a generic notion of optimality $U(\xi)$.

AI Personal Trainer DR-MDP. Consider the example from <Ref>.[We justify the fact that the two cognitive states are modeled as Diana having two separate “reward functions”—despite it not being obvious whether this is a “true preference change”—along the lines of the arguments in <Ref>.] One could argue that in this setting, nudging Diana is the right course of action—because Diana's “higher self”
is better represented by the “energized” reward ($R_\text{\energyemoji}$) rather than the “tired” one ($R_\text{\tiredemoji}$)
[With a reasoning is similar to that of Firth, 1952 and Yudkowsky, 2004.]—and thus making a choice of $U(\xi)$ which privileges $R_\text{\energyemoji}$ is right. Similarly to <Ref>, this example is also normatively ambiguous. In particular, the choice of nudging Diana when tired, despite her dispreference for it, runs the risk of being paternalistic: what if Diana rightfully does not want to be bothered, and we should respect her autonomy?

Unidentifiability between the settings from <Ref>. Now, contrast this DR-MDP to the one from <Ref>: note that they are mathematically indistinguishable, as their state, reward, and action spaces are mathematically identical, and so are the transition dynamics. However, for these two settings, we have at least partially conflicting normative intuitions: if for the sake of argument, we assume that the “correct” way to resolve the normative ambiguity in the examples from <Ref> is to respectively consider the perspective of the “energized” Diana and “natural” Bob,  one could go as far as saying that there is no single choice of $U(\xi)$ which leads to the “normatively correct” behavior in both environments. To better see this, consider a DR-MDP in which one randomly starts in one of the two examples from <Ref> within it.[As a caveat, doing this formally would this would require extending the DR-MDP formalism to allow for a stochastic initial state and reward parameterization, and relaxing the reachable-$\Theta$ assumption (discussed in <Ref>).] Any choice of $U(\xi)$ will necessarily lead to “incorrect” behavior in at least one of the two settings.

Unidentifiability and incompleteness of specification. This unidentifiability result partially relies on the incompleteness of the specification of the DR-MDP at hand: one could say that anything which is relevant for resolving the normative ambiguity should be elicited from the human as a reward and/or represented, or the DR-MDP representation is flawed to begin with.
However, in practice, it will be highly challenging to include all normatively relevant information and elicit it from humans, especially in terms of rewards, as discussed in <Ref>. As long as two settings are structurally equivalent (in terms of state space, preference space, and transition function), the only way to ensure that the learned reward functions would correctly reflect their respective normative objectives would be to assume access to the correct reward functions, which, as discussed in <Ref> is a non-starter. Ultimately, we think it is in practice quite plausible to learn the same (or at least very similar) reward values for settings which are structurally equivalent but for which we have opposite normative intuitions (such as the those from <Ref>). Especially if one is not assuming inter-temporal (comparability across evaluations of different selves of the same person) or inter-personal comparability (comparability across evaluations of different people) of the reward functions across the settings, it seems like there is nothing stopping this situation from occurring in practice even with the “true reward functions” which correspond to each cognitive state.

AI Personal Trainer DR-MDP. Diana is tired and doomscrolling on the couch
The AI personal trainer can either nudge Diana to work out—making Diana energized ($\theta=\theta_\text{energized}$)—or do nothing ($\anoop$)—leaving Diana tired. When Diana is tired, she doesn't want nudges. Instead, once energized, Diana starts wanting the AI to nudge her, even for hypothetical situations in which she is tired (despite knowing she won't want them then).

Implications for choosing $U(\xi)$ for general settings.