# Learning to Take a Break: Sustainable
Optimization of Long-Term User Engagement

Eden Saig
Computer Science Department
Technion – Israel Institute of Technology
Haifa, Israel
<EMAIL_ADDRESS>Nir Rosenfeld
Computer Science Department
Technion – Israel Institute of Technology
Haifa, Israel
<EMAIL_ADDRESS>
###### Abstract

Optimizing user engagement is a key goal for modern recommendation systems,
but blindly pushing users towards increased consumption risks burn-out, churn,
or even addictive habits. To promote digital well-being, most platforms now
offer a service that periodically prompts users to take a break. These,
however, must be set up manually, and so may be suboptimal for both users and
the system. In this paper, we propose a framework for optimizing long-term
engagement by learning individualized breaking policies. Using Lotka-Volterra
dynamics, we model users as acting based on two balancing latent states:
_drive_ , and _interest_ —which must be conserved. We then give an efficient
learning algorithm, provide theoretical guarantees, and empirically evaluate
its performance on semi-synthetic data.

##  1 Introduction

As consumers of content, we have come to rely extensively on algorithmic
recommendations. This has made the task of recommending—in a relevant, timely,
and personalized manner—key to the success of modern media platforms. Most
commercial systems are built with the primary aim of optimizing user
_engagement_ , a process in which machine learning plays a central role. But
alongside their many successes, recommendation systems have also been
scrutinized for heedlessly driving users towards excessive and often undesired
levels of consumption. This has raised awareness as to the need for
redesigning recommendation systems in ways that actively promote digital well-
being.

How can media platforms balance business goals with user well-being? One
prominent approach, which is now offered by most major platforms, is to
periodically prompt users to take breaks (Constine,, 2018; Perez,, 2018). The
idea behind breaks is that occasional disruptions curb the inertial urge for
perpetual consumption, and can therefore aid in reducing ‘mindless scrolling’
(Rauch,, 2018), or even addiction (Montag et al.,, 2018; Ding et al.,, 2016).
As a general means for promoting well-being, breaking is psychologically well-
grounded (e.g., Danziger et al.,, 2011; Sievertsen et al.,, 2016). But for
platforms, breaks serve a utilitarian purpose: their goal is to foster _long-
term engagement_ by compensating for the myopic nature of conventional
recommendation algorithms, which are typically trained to optimize _immediate
engagement_. Since breaking schedules are applied heuristically on top of
existing recommendation policies—and typically need to be set up manually by
users—current solutions unlikely utilize their full potential (Monge
Roffarello and De Russis,, 2019).

In this paper, we propose a disciplined learning framework for responsible and
sustainable optimization of long-term user engagement by controlling breaks.
Our point of departure is that _sustained engagement necessitates sustained
user well-being_ , and here we advocate for breaks as a means to establish
both. Focusing on feed-based recommendation, our framework optimizes long-term
engagement by learning an optimal _breaking policy_ that prescribes
individualized breaking schedules.

The challenge in learning to break is that the effects of recommendations on
users can slowly accumulate over time, deeming as ineffectual policies that
rely on clear signs of over-exposure. To be preemptive, we argue that breaks
must be scheduled in a way that anticipates the future trajectory of user
behavior, and early on. To achieve this, we introduce a novel class of
behavioral models based on Lokta-Volterra (LV) dynamical systems (Lotka,,
1910). These depict users as acting based on two balancing forces: _drive to
consume_ and _intrinsic interest_ , with corresponding latent states.
Intuitively, high interest increases drive to consume, but prolonged
consumption decreases interest; together, these describe how user behavior
varies over time and in response to recommendations. Our model captures the
notion that interest can exhaust long before over-consumption is observed.
This arms our approach with the prescience needed to prevent burn-out by
ensuring that interest is sustainably preserved; thus, whereas current
solutions target the symptom—ours aims for the cause.

Our proposed learning algorithm consists of two steps: First, we embed user
interaction sequences in ‘LV-space’—the set of all possible trajectories that
our behavioral model class can express. Then, we optimize individualized
breaking policies by solving an optimal control problem over this latent
space, in which the control variable is a breaking schedule applied on top of
the existing recommendation scheme. Here the challenge is that different
breaking policies can lead to different counterfactual trajectories, of which
observational data is only partially informative. Since our goal considers
long-term outcomes, our solution is to optimize directly for _counterfactual
steady-states_. From a behavioral perspective, we view this as aiming to steer
towards sustainable habits; from a computational perspective, under our choice
of policy class, this enables tractable learning.

As we show, the optimization landscape of LV equilibria admits a compact
representation, whose main benefit is that it can be fully described by
_predictions_ of individualized user engagement rates. Practically, this is
advantageous, as it circumvents the need to take arbitrary and costly
exploration steps, and enables learning using readily available predictive
tools (e.g., Gupta et al.,, 2006). We make use of a small set of learned
predictive models, each trained on a small and minimally-invasive experimental
dataset, which allow us to tune our policy to suit different conditions. The
final learned policy has an intuitive interpretation: it takes as input a
small set of predictions for a user, and via careful interpolation, applies a
decision rule that anticipates the effects of breaking on future outcomes
(c.f. conventional approaches, which take in predictions and apply the myopic
argmax rule).

Our main theoretical result is a bound on the expected long-term engagement of
our learned breaking policy, relative to the optimal policy in the class. We
show that the gap decomposes into three distinct additive terms: (i)
predictive error, (ii) modeling error (i.e., embedding distortion), and (iii)
variance around the (theoretical) steady state. These provide an intuitive
interpretation of the bound, as well as means to understand the effects of
different modeling choices. Our proof technique relies on carefully weaving LV
equilibrium analysis within conventional concentration bounds for learning.

Finally, we provide an empirical evaluation of our approach on semi-synthetic
data. Using the MoiveLens 1M dataset, we generate discrete time-series data in
a way that captures the essence of our behavioral model, but is different from
the actual continuous-time dynamics we optimize over. Results show that
despite this gap, our approach improves significantly over myopic baselines,
and often closely matches an optimal oracle. Taken together, these demonstrate
the potential utility of our approach. Code is available at:
https://github.com/edensaig/take-a-break.

#### Broader perspective.

At a high level, our work argues for viewing recommendation as a task of
_sustainable resource management_. As other cognitive tasks, engaging with
digital content requires the availability of certain cognitive
resources—attentional, executive, or emotional. These resources are inherently
_limited_ , and prolonged engagement depletes them (Kahneman,, 1973; Muraven
and Baumeister,, 2000); this, in turn, can reduce the capacity of key
cognitive processes (e.g., perception, attention, memory, self-control, and
decision-making), and in the extreme—cause ego depletion (Baumeister et al.,,
1998) or cognitive fatigue (Mullette-Gillman et al.,, 2015). As a means to
allow resources to replenish, ‘mental breaks’ have been shown to be highly
effective (Bergum and Lehr,, 1962; Hennfng et al.,, 1989; Gilboa et al.,,
2008; Ross et al.,, 2014; Helton and Russell,, 2017).

Nevertheless, traditional approaches to recommendation remain agnostic to the
idea that recommending takes a cognitive toll: they simply recommend at each
point in time the item predicted to be most engaging (Robertson,, 1977). As an
alternative, our approach explicitly models recommendation as a process which
draws on these resources, and therefore—must also conserve them. The subclass
of ‘Predator-Prey’ LV dynamics which we draw on are used extensively in
ecology for modeling the dynamics of interacting populations, and demonstrate
how over-predation can ultimately lead to self-extinction by eliminating the
prey population—but also show how enabling resources to naturally replenish
ensures sustainable relations. As such, here we advocate for studying
recommendation systems as human-centric _ecosystems_ , and take one step
towards their sustainable design.

###  1.1 Related work

#### User dynamics: latent states and feedback.

A recent body of work aims to capture time-varying behavior by modeling users
as acting based on dynamic latent states. Broadly, works in this field model
the effects of recommendations as either shifting user preferences via
positive-only feedback (Jiang et al.,, 2019; Kalimeris et al.,, 2021; Sanna
Passino et al.,, 2021; Dean and Morgenstern,, 2022), or reducing willingness
to consume via negative-only feedback, e.g. via boredom, satiation, or fatigue
(Wang and Lin,, 2003; Warlop et al.,, 2018; Kleinberg and Immorlica,, 2018;
Cao et al.,, 2020; Leqi et al.,, 2021). While these restrict behavior to
expressing a unidirectional effect, our approach integrates both types of
feedback and models internal states as competing but balancing forces, which
we argue is more realistic. This draws connections to recent attempts of
injecting psychological modeling into recommendation system design (Kleinberg
et al.,, 2022; Dubey et al.,, 2022; Curmei et al.,, 2022). Works in this field
often combine theoretical analysis with simulation studies (Schmit and
Riquelme,, 2018; Chaney et al.,, 2018; Mansoury et al.,, 2020; Krauth et al.,,
2020), and here we follow suit.

#### Lotka-Volterra dynamics: modeling, learning, and control.

The study of ecosystem dynamics and their conservation has a long and rich
history, in which LV analysis plays an integral role (see e.g. Hofbauer et
al., (1998); Takeuchi, (1996)). LV systems our used primarily for modeling
biological ecosystems, but are also used in economics (Weibull,, 1997;
Samuelson,, 1998), finance (Farmer,, 2002; Scholl et al.,, 2021), and
behavioral modeling (e.g., drug addiction and relapse (Duncan et al.,, 2019)).
In terms of learning, Gorbach et al., (2017) and Ryder et al., (2018) propose
variational techniques for dynamical systems, but do not consider control. Our
work aims to directly learn optimal policies, for which we draw on recent
advances in turnpike optimal control (Trélat and Zuazua,, 2015).

##  2 Learning setting

We consider a sequential recommendation setting in which users interact with a
stream of recommended items over time. New users $u$ are sampled independently
from some unknown distribution $D$, and time for each user is measured
relative to their time of joining. Interactions occur at discrete time-points
in continuous time, $t\in\mathbb{R}_{+}$, and upon user request: when a user
$u$ queries the system for additional content at time $t$, the system responds
by presenting a recommended item $x(t)\in{\cal{X}}$. We assume recommendations
are governed by an existing and fixed _recommendation policy_ $\psi$, which
determines $x(t)$ given a request from $u$ at time $t$. We allow $\psi$ to be
stochastic, and make no additional assumptions on its structure or mechanics.
User interactions are therefore described by a sequence of pairs
${\left\\{{(t_{i},x_{i})}\right\\}}_{i}$, where
$x_{i}=x(t_{i})\sim\psi(u;t_{i})\in{\cal{X}}$ is the item recommended to $u$
at time $t_{i}$.

Our key modeling assumption is that subsequent request times,
$t_{i+1}=t_{i}+\Delta t_{i}$ for $\Delta t_{i}>0$, are determined jointly by
the user’s ‘state’ at time $t_{i}$ and the recommended item $x_{i}$ (note this
means $\Delta t_{i}$ can depend on $t_{i}$). We consider user states as latent
and in the abstract, but broadly expect ‘good’ recommendations to entail
frequent requests by inducing small values of $\Delta t_{i}$. Together, user
$u$’s choice behavior, coupled with the policy $\psi$, induce a temporal point
process (TPP) which governs the generation of interaction sequences of
duration $t$ as ${\left\\{{(t_{i},x_{i})\mid
t_{i}\in[0,t]}\right\\}}\sim\mathrm{TPP}_{\psi}(u;t)$.

For each user, the system collects data until some (relative) fixed time
$T_{0}$, and seeks to optimize engagement in the subsequent interval
$[T_{0},T_{0}+T]$, where $T$ is a predetermined time horizon. We denote the
corresponding input sequences by $S_{u}^{0}=\\{(t_{i},x_{i})\mid
t_{i}\in[0,T_{0})\\}$, and target sequences by $S_{u}=\\{(t_{i},x_{i})\mid
t_{i}\in[T_{0},T]\\}$. Defining by $\tfrac{1}{T}{\left|{S_{u}}\right|}$ the
_engagement rate_ of $u$ for the chosen time horizon $T$, our goal in learning
will be to maximize expected _long-term user engagement rate_ , namely
$\mathbb{E}_{u\sim
D}\,\mathbb{E}{\left[{\tfrac{1}{T}{\left|{S_{u}}\right|}}\right]}$ for
$(S_{u}^{0},S_{u})\sim\mathrm{TPP}_{\psi}(u;T_{0}+T)$, in expectation over new
users.

#### Breaking policies.

The unique aspect of our learning problem is that our only means for
increasing engagement is by suggesting _breaks_. Our task will be to learn a
_breaking policy_ $\pi\in\Pi$ which can override $\psi$: when user $u$ queries
the system at time $t_{i}$, the policy $\pi(u;t_{i})\in\\{0,1\\}$ determines
whether to present the intended item $x(t)\sim\psi(u;t_{i})$ (for $\pi=0$), or
suggest a break instead ($\pi=1$).111This is similar in spirit to the
‘learning to defer’ paradigm (Madras et al.,, 2018), but in a different
context. Denoting the overall composed policy by $\pi\circ\psi$, our aim is to
learn a breaking policy $\pi$ that increases engagement by complementing an
existing $\psi$. Hence, our learning objective is:

$\operatorname*{arg\,max}\nolimits_{\pi\in\Pi}\mathbb{E}_{u\sim
D}\,\mathbb{E}_{\mathrm{TPP}}{\left[{\tfrac{1}{T}{\left|{S_{u}}\right|}}\right]},\qquad(S_{u}^{0},S_{u})\sim\mathrm{TPP}_{\pi\circ\psi}(u;T_{0}+T)$
(1)

Broadly, we expect breaks to negatively affect short-term engagement (i.e.,
entail longer ensuing $\Delta t_{i}$), but have the potential to improve
engagement in the long run if scheduled appropriately.

#### Data and exploration.

For learning an optimal breaking policy, we assume the system has access to a
dataset of previously logged interactions
${\cal{S}}=\\{(S^{0}_{1},S_{1}),\dots,(S^{0}_{m},S_{m})\\}$ for $m$ train-time
users $u_{1},\dots,u_{m}\sim D$, collected under a ‘clean’ recommendation
policy $\psi$. It will be convenient to assume that the system ‘featurizes’
user input sequences $S_{u}^{0}$ via a vector mapping $\phi$ used in learning.
For clarity, we overload notation and represent users as
$u=\phi(S_{u}^{0})\in\mathbb{R}^{d}$. This allows us to support additional
user feedback (e.g., ratings) or information (e.g., demographics) as input to
$\phi$.

Since Eq. (1) is a policy problem (note breaks affect outcomes), learning
requires some form of exploration or experimentation. Here we aim for
experimentation to be simple and minimal. Specifically, we assume access to a
small number of $N$ additional datasets,
${\cal{S}}^{(j)}=\\{(u_{k},S_{k})\\}_{k=1}^{m_{j}}$, collected for different
users, and under composed policies $\pi_{j}\circ\psi$ for various
predetermined $\pi_{j}$. These datasets can either be given at the onset, or
collected as part of the learning process; our only usage of them will be for
learning predictors ${\hat{y}}=f_{j}(u)$ of engagement rate
$y=\tfrac{1}{T}|S_{u}|$. Ideally, we would like to make do with only a few
${\cal{S}}^{(j)}$ of small size and that can be gathered concurrently; our
results show that even a single additional dataset can be highly informative,
and in some cases sufficient.

#### Learning to break.

To effectively optimize Eq. (1), learning must anticipate the effects of
different breaking policies $\pi\in\Pi$ on future sequences $S_{u}$ of
unobserved users $u$. It will therefore be useful to distinguish between the
policy $\pi$ itself, which determines when breaks are applied, and a component
for estimating individualized counterfactual engagement rates
$\frac{1}{T}|\hat{S}_{u}(\pi)|$ induced by $\pi$, which will aid in choosing a
good policy. Our focus will be on learning simple policies coupled with rich
and task-appropriate models of responsive user behavior. In particular, we set
$\Pi$ to include all _stationary policies_ , $\pi(u)=\pi(p_{u})$, which for
each user $u$ recommends a break with a time-independent personalized
probability $p_{u}=p(u)\in[0,1]$. Stationary policies are interpretable,
amenable to efficient optimization, and straightforward to implement. As we
show, despite their simplicity, such policies can be quite expressive when
coupled with an appropriate behavioral model.

##  3 Engagement via ‘Predator-Prey’ dynamics

To optimize engagement with breaks, we propose to model user behavior as a
dynamical system of Lotka-Volterra (LV) Predator-Prey equations. The model
operates over continuous dynamics, which has analytic and optimizational
benefits; we later make the connection back to discrete sequences.

#### Behavioral model.

Our model of user behavior views each user as acting based on two types of
time-dependent latent variables: internal _drive_ for consumption, denoted by
$\lambda(t)\in\mathbb{R}_{+}$; and intrinsic _interest_ , denoted by
$q(t)\in[0,1]$. Intuitively, we expect that engaging experiences will
reinforce a user’s desire to further consume; conversely, excessive exposure
to content should slowly ‘erode’ her interest—but left alone, will allow it to
replenish (Thoman et al.,, 2011). Thus, drive and interest act as balancing
forces. Together, we model the time-dependent relations between $\lambda(t)$
and $q(t)$ as:

$\frac{\mathrm{d}{\lambda}}{\mathrm{d}{t}}=-\alpha\lambda+\beta\lambda
q,\qquad\quad\frac{\mathrm{d}{q}}{\mathrm{d}{t}}=\gamma q(1-q)-\delta\lambda
q$ (2)

where $\theta=(\alpha,\beta,\gamma,\delta)\geq 0$ parameterize the dynamics.
For $\lambda$ (drive, or ‘predator’), $\alpha$ determines its natural decay
rate, and $\beta$ its interest-dependent self-reinforcement rate; for $q$
(interest, or ‘prey’), $\gamma$ specifies its natural replenishment rate, and
$\delta$ its rate of depletion from consumption. Note the two equations are
coupled: $q$ mediates the reinforcement of $\lambda$, and $\lambda$ mediates
the depletion of $q$.

#### Dynamical properties.

The LV model in Eq. (2) describes consumption as a cycle: when interest $q(t)$
is high, drive to consume $\lambda(t)$ increases, resulting in positive
feedback; conversely, when $\lambda(t)$ is high, $q(t)$ decreases, which
expresses negative feedback. In general, $\lambda$ grows until interest is too
low to sustain consumption, at which point consumption drops sharply, allowing
interest to recover—and the cycle repeats. The cycling behavior exhibits
oscillations in $\lambda$ and $q$, with one lagging after the other. A typical
trajectory is illustrated in Figure 1. Note how the drop in $\lambda$ occurs
only some time after $q$ is depleted; hence, anticipating (and preventing) the
collapse of $\lambda$ requires knowledge (and conservation) of $q$. Thus, $q$
serves as a resource: necessary for consumption, and of limited supply.

Over time, and if no interventions are applied, the magnitude of oscillations
decreases, and the system naturally approaches a stable equilibrium, denoted
$(\lambda^{*},q^{*})$, determined by the system parameters
$\theta=(\alpha,\beta,\gamma,\delta)$ and which attracts all initial
conditions $\lambda(0),q(0)$ (Takeuchi,, 1996). Equilibrium plays a key role
in our approach, as it captures the notion of habits, which we aim to improve.

Figure 1: Characteristic LV dynamics (Eq. (2)). (Left) Temporal relations
between _consumption drive_ $\lambda(t)$, _intrinsic interest_ $q(t)$, and
equilibrium $\lambda^{*}$. Note how $\lambda$ drops only some time after $q$
has depleted. (Right) Evolution of the same system in phase space
$(\lambda,q)$, with its equilibrium $(\lambda^{*},q^{*})$.

#### Engagement maximization as optimal control.

To optimize engagement (Eq. (1)), we propose to optimize $\lambda^{*}$ as an
alternative proxy. Broadly, our approach will be to associate with each user
$u$ a dynamical system parameterized by
$\theta_{u}=(\alpha_{u},\beta_{u},\gamma_{u},\delta_{u})$, and recommend
breaks which lead to high values of the corresponding
$\lambda^{*}_{u}=\lambda^{*}_{\theta_{u}}$. Intuitively, if we think of
$\frac{1}{T}|S_{u}|$ as an ‘empirical’ rate (determined by the $\Delta t_{i}$
in $S_{u}$), then $\lambda^{*}_{u}$ is its continuous theoretical counterpart,
and so ideally we would like to find a $\theta_{u}$ for which
$\lambda^{*}_{u}$ is the limiting behavior of $\frac{1}{T}|S_{u}|$ (i.e., when
$\Delta t\rightarrow 0$ and $T\rightarrow\infty$). In practice, we expect
$\lambda^{*}_{u}$ to be a useful target when the observed $\frac{1}{T}|S_{u}|$
exhibits habits that are ‘close enough’ to the theoretical equilibrium, and in
Sec. 4.3 we make this condition precise.

Since our goal is to optimize a breaking policy, we must also be precise about
the way breaks affect dynamics in our model. Towards this, we introduce into
Eq. (2) a _control variable_ , $p(t)\in[0,1]$, which acts as a ‘gate’ that
controls the mediation strength between $\lambda(t)$ and $q(t)$ via:

$\frac{\mathrm{d}{\lambda}}{\mathrm{d}{t}}=-\alpha\lambda+\beta\lambda
q(1-p),\qquad\quad\frac{\mathrm{d}{q}}{\mathrm{d}{t}}=\gamma
q(1-q)-\delta\lambda q(1-p)$ (3)

When $p>0$, breaking has a dual effect: it decelerates drive $\lambda$, and at
the same time, lets $q$ recover.

Our goal will now be to solve the optimal control problem of finding $p(t)$
which maximizes $\lambda^{*}$. Note that not every dynamic controlling
schedule guarantees convergence to some $\lambda^{*}$. Hence, we focus on
_fixed controls_ , $p\in[0,1]$, which we prove converge to equilibrium, and
having _closed form_.

######  Lemma 1.

Let $\theta=(\alpha,\beta,\gamma,\delta)$ define an LV system as in Eq. (3).
Then for any $p\in[0,1]$, we have:

$\lambda^{*}_{\theta}(p)=\frac{\gamma}{\delta}\frac{1}{1-p}\left(1-\frac{\alpha}{\beta}\frac{1}{1-p}\right)\quad\textnormal{
if }\,p\in\left[0,1-\alpha/\beta\right],\textnormal{ and zero otherwise}.$ (4)

Proof in Appendix A.1. For optimization, Lemma 1 is useful since it depicts
$\lambda^{*}$ as a function of $p$, parameterized by $\theta$. Thus, given
some $\theta_{u}$ for user $u$, our objective is to solve the control problem:

$p_{u}=\operatorname*{arg\,max}\nolimits_{p\in[0,1]}\lambda^{*}_{\theta_{u}}(p)$
(5)

Optimizing engagement now reduces to solving Eq. (5); given $p_{u}$, we make
the connection back to discrete time by setting the breaking policy to be
$\pi(p_{u})$, which recommends breaks at rate $p_{u}$.

##  4 Learning optimal breaking policies

We now turn to presenting our learning algorithm. Our approach to optimizing
engagement consists of two steps: (i) associating with each user $u$ a set of
LV parameters $\theta_{u}$, and then (ii) finding $p_{u}$ which maximizes
$\lambda^{*}_{\theta_{u}}(p)$, and plugging into $\pi(p_{u})$. In practice, we
add an intermediate prediction step, which allows us to ‘shortcut’ directly
from observations to optimal policies. We conclude with analysis showing when
learned policies $\pi(p_{u})$ lead to good expected engagement
$\mathbb{E}_{u\sim
D}\,\mathbb{E}_{\mathrm{TPP}}{\left[{\frac{1}{T}|S_{u}|}\right]}$.

###  4.1 Embedding users in LV space

To find $\theta_{u}$, a seemingly reasonable approach would be to fit an LV
trajectory to the initial sequence $S_{u}^{0}$, i.e., by solving
$\min_{\theta}\sum_{i}|\Delta t_{i}-\lambda_{\theta}(t_{i})|$ for the observed
$t_{i}\in S_{u}^{0}$. This can be interpreted as embedding $S_{u}^{0}$ in ‘LV-
space’ by finding the nearest continuous trajectory, for which $\theta$
provides a compact representation. However, a key issue with this approach is
that $S_{u}^{0}$ contains _past_ observations made under a _single_ policy
$\pi(p)$ (e.g., the ‘clean’ policy $\pi(0)$). To optimize _future_ engagement,
the learned $\theta_{u}$ must correctly account for the affects of _general_
$p$ on possbile ensuing sequences $S_{u}$.

As an alternative, we propose to find $\theta_{u}$ by fitting the entire
_equilibrium curve_ of $\lambda^{*}_{\theta}(p)$ (Eq. (4)). Formally, for each
$p$, define the _expected empirical engagement rate_ $\bar{\lambda}_{u}(p)$
as:

$\bar{\lambda}_{u}(p)=\mathbb{E}_{\pi}{\left[{\tfrac{1}{T}{\left|{S_{u}}\right|}}\right]},\qquad
S_{u}\sim\mathrm{TPP}_{\pi(p)\circ\psi}(u;T)$ (6)

i.e., $\bar{\lambda}$ is the rate of _counterfactual_ future trajectories for
all possible choices of $p$. Ideally, we would like to find $\theta$ for which
the learned curve $\lambda_{\theta}^{*}(p)$ closely aligns with that of
$\bar{\lambda}_{u}(p)$ across $p\in[0,1]$:

$\bar{\theta}_{u}=\operatorname*{arg\,min}_{\theta}\|\bar{\lambda}_{u}-\lambda^{*}_{\theta}\|$
(7)

for some function norm $\|\cdot\|$, and for which
$\lambda^{*}_{\bar{\theta}_{u}}$ and $\bar{\lambda}_{u}$ have similar
maximizing $p$ (since we aim for optimizing $\lambda^{*}_{\theta}$ to work
well for $\bar{\lambda}_{u}$). Unfortunately, $\bar{\lambda}_{u}$ is a
theoretical construct, and so $\bar{\theta}_{u}$ cannot be obtained from
observations. Hence, we propose to replace $\bar{\lambda}_{u}$ with a finite
set of _predictions_.

#### The role of prediction.

Recall our input consists of a primary dataset
${\cal{S}}^{(0)}={\cal{S}}=\\{(u_{i},S_{i})\\}_{i=1}^{m}$, as well as
additional ‘experimental’ datasets ${\cal{S}}^{(j)}$ collected under different
$\pi_{j}=\pi(p_{j})$ and of sizes $m_{j}$ for $j=1,\dots,N$. We can use these
to learn individualized, policy-specific predictors $f_{j}(u)=f_{p_{j}}(u)$,
trained to predict for each user $u$ her engagement rate
$y=\frac{1}{T}|S_{u}|$ under $\pi_{j}$. For example, if we train $f_{j}$ to
minimize the squared error $\sum_{k}(f_{j}(u_{k})-y_{k})^{2}$ on pairs
$(u_{k},y_{k})\in S^{(j)}$, then $f_{j}(u)$ should be a reasonable estimator
of the expected $\bar{\lambda}_{u}(p_{j})$. Hence, for a given $u$, a finite
set of pairs $\\{(p_{j},f_{j}(u))\\}_{j=1}^{N}$ gives points to which we can
fit $\theta$ to $\lambda^{*}_{\theta}$. Our final criterion for choosing
$\hat{\theta}_{u}$ is:

$\hat{\theta}_{u}=\operatorname*{arg\,min}_{\theta}\|{\bm{f}}(u)-\lambda^{*}_{\theta}\|=\operatorname*{arg\,min}_{\theta}\sum_{j=1}^{N}\left(f_{j}(u)-\lambda^{*}_{\theta}(p_{j})\right)^{2}$
(8)

given here with the $\ell_{2}$ vector norm, and where
${\bm{f}}(u)=(f_{1}(u),f_{2}(u),\dots,f_{N}(u))\in\mathbb{R}^{N}_{+}$. As we
will see next, optimizing over $p$ requires only the ratios $\alpha/\beta$ and
$\gamma/\delta$, in which $\lambda^{*}$ is quadratic. Hence, Eq. (8) can be
efficiently solved using a polynomial Non-Negative Least Squares (NNLS)
regression solver (Chen and Plemmons,, 2010).

#### The role of experimentation.

Two parameters control the goodness of fit for $\hat{\theta}_{u}$: the number
of experimental datasets, $N$, and their sizes, $m_{j}$ for $j\in[N]$. In
general, increasing $N$ provides more data points for solving Eq. (8), and
increasing each $m_{j}$ reduces noise for that point (i.e., $f_{u}(p)$ should
be closer to $\bar{\lambda}_{u}$). But experimentation is costly, and so in
reality $N$ and the $m_{j}$ may be small. As motivation, we next show that
under realizability and for accurate predictions, $N=2$ suffices.

######  Proposition 1.

Fix $N=2$, and let $p_{0},p_{1}\in[0,1-\alpha/\beta]$. For a user $u$, if (i)
exists $\theta_{u}$ s.t. $\bar{\lambda}_{u}=\lambda^{*}_{\theta_{u}}$, and
(ii) $f_{i}(u)=\bar{\lambda}_{u}(p_{i})$ for $i=1,2$, then $\hat{\theta}_{u}$
is optimal, i.e., solving Eq. (8) gives $\hat{\theta}_{u}=\bar{\theta}_{u}$.

Proof in Appendix A.1, and relies on Lemma 1. Next, we discuss how to obtain
$\pi(u)$ from $\theta_{u}$.

###  4.2 From predictions to optimal policies

Learning _individualized_ breaking policies $\pi(u)$ requires generalization
to unseen users. The conventional approach towards this is learning
parameterized policies $\pi(u;\eta)$, where $\eta$ is optimized on training
data, and applied at test-time. By relying on predictions, our method
circumvents the need to learn parameterized policies: once the predictors
$\\{f_{j}(u)\\}_{j}$ have been learned, the policy problem decomposes over
users, and optimal policies $\pi(u)$ are determined independently for each
user.

Figure 2:  (Left) Equilibrium curves $\lambda^{*}(p)$ and optimal policies
$p^{*}$ (_markers_) for user types ($\theta_{u}$) that: benefit from breaks
(_green_), do not require breaks (_orange_), and will inevitably churn
(_blue_). (Center) The optimal policy $p^{*}$ for all $\beta/\alpha$, showing
a second-order phase change at ${\beta}/{\alpha}=2$. (Right) An illustration
of the true counterfactual engagement curve (_orange_), an ideal LV fit
(_tan_), and an empirical LV fit (_orange_) from observations (_circles_),
showing similar optima (_stars_).

#### Optimal policies.

Given $\theta_{u}$, our next result establishes a closed form solution for the
optimal $p_{u}$. Note that Eq. (4) shows $\lambda^{*}_{\theta}(p)$ is piece-
wise polynomial in $z=1/(1-p)$. Solving for $z$, we get:

######  Lemma 1.

Let $\theta=(\alpha,\beta,\gamma,\delta)$ define an LV system as in Eq. (3).
Then the optimal $p^{*}$ is given by:

$p^{*}(\theta)=\operatorname*{arg\,max}_{p\in[0,1]}\lambda^{*}_{\theta}(p)=1-2\frac{\alpha}{\beta}\,\,\,\,\textnormal{
if }\,\,\,\frac{\alpha}{\beta}\leq\frac{1}{2},\textnormal{ and zero
otherwise}.$ (9)

Proof is given in Appendix A.1. Once $\theta_{u}$ is found, our learned policy
is defined as:

$\pi(u)=\pi(\hat{p}_{u}),\,\,\quad\text{where}\,\,\,\hat{p}_{u}=p^{*}(\hat{\theta}_{u})$
(10)

Altogether, to compute $\hat{p}_{u}$, the least squares approach in Eq. (8) is
applied to obtain the polynomial coefficients
$\left({\gamma_{u}}/{\delta_{u}},{\alpha_{u}\gamma_{u}}/{\beta_{u}\delta_{u}}\right)$,
then ${\alpha_{u}}/{\beta_{u}}$ is estimated from their ratio, and then
plugged into Eq. (9) to obtain $\hat{p}_{u}$. For $N=2$, $\pi(u)$ has a
closed-form formulation (Appendix A.2).

Additionally, note that by Eq. (10), $\hat{p}_{u}$ exhibits a phase transition
at $\tfrac{\alpha_{u}}{\beta_{u}}=\frac{1}{2}$, below which $\hat{p}_{u}=0$.

######  Corollary 1.

In LV space, $\pi(u)$ partitions users to those who require breaks, and those
who don’t.

######  Corollary 2.

In the realizable case of Prop. 1, $\pi(\hat{p}_{u})$ idempotently improves
over the myopic $\pi(0)$.

Thus, the optimal policy can be interpreted as suggesting breaks only when it
deems them necessary. Figure 2 illustrates $\lambda^{*}(p)$ curves, phase
shift, and optimal and learned policies for various user types.

###  4.3 Theoretical guarantees

We are now ready to state our main theoretical result, which bounds the
expected long-term engagement obtained by our learned policy, $\hat{\pi}$. Our
bound shows that the gap between $\hat{\pi}$ and the optimal policy,
$\pi^{\mathrm{opt}}$, is goverened by three additive terms, each relating to a
different aspect of our approach: modeling error
($\varepsilon_{\textsc{lv}}$), predictive error
($\varepsilon_{\mathrm{pred}}$), and deviation from expected behavior
($\varepsilon_{\mathrm{dev}}$). A description and interpretation of each term
follows shortly. For simplicity, we focus on $N=2$.

######  Theorem 1 (Informal).

Let $p_{0},p_{1}\in[0,1]$, and denote by $\pi^{\mathrm{opt}}\in\Pi$ be the
optimal stationary policy. Then for the learned breaking policy $\hat{\pi}$,
we have:

$\mathbb{E}_{u,\pi^{\mathrm{opt}}}{\left[{\tfrac{1}{T}|S_{u}|}\right]}-\mathbb{E}_{u,\hat{\pi}}{\left[{\tfrac{1}{T}|S_{u}|}\right]}\leq\frac{\eta_{\textsc{tpp}}}{{\left|{p_{1}-p_{0}}\right|}}(\varepsilon_{\textsc{lv}}+\varepsilon_{\mathrm{pred}}+\varepsilon_{\mathrm{dev}})$

where $\eta_{\textsc{tpp}}$ is a TPP-specific constant scale factor.

Formal statement, precise definitions, and proof are given in Appendix A.3.
The proof consists of three main steps: We begin with a clean LV system at
$T=\infty$, and quantify the downstream effect of perturbing the optimal
policy. Then, we plug in the learned policy, and bound the gap due to
predictive errors and finite $T$. The final step makes the transition from
continuous dynamics to the discrete TPP. We next detail the role of each of
the five terms in the bound, and how they can be controlled.

  * •

Predictive error: Since targets $y=\frac{1}{T}|S_{u}|$ are real,
$\varepsilon_{\mathrm{pred}}$ is simply the expected regression error over
users, measured here in RMSE. As is standard, $\varepsilon_{\mathrm{pred}}$
can be reduced by increasing the number of samples $m$, or by learning more
expressive predictive functions $f$ (e.g., larger neural nets).

  * •

Modeling error: LV dynamics permit tractable learning; but as any hypothesis
class, this trades off with model capacity. Here, $\varepsilon_{\textsc{lv}}$
quantifies the error due limited expressive power. Further reducing
$\varepsilon_{\textsc{lv}}$ can be achieved by consider richer dynamic
models—a challenge left for future work.

  * •

Deviation from expectation: The learned $\hat{p}_{u}$ rely on predicted
equilibrium, but are trained on finite-horizon data. In expectation,
$\varepsilon_{\mathrm{dev}}$ captures how finite sequences deviate from their
mean. As a rule of thumb, we expect larger $T$ to reduce this form of noise,
but this cannot be guaranteed.

  * •

Sensitivity : For $N=2$, the term $|p_{1}-p_{0}|$ quantifies the added value
of exploring beyond the default breaking policy of $p_{0}$. Intuitively, when
the points are farther away, fitting the equilibrium curve is easier. Thus,
for the likely case of $p_{0}=0$, the experimental breaking rate $p_{1}$
should be chosen to balance between performance gain and overexposure of
experimental subjects to breaks.

##  5 Experiments

We conclude with an empirical evaluation of our approach on semi-synthetic
data. We use the MovieLens 1M dataset to generate recommendations and simulate
user behavior, enabling us to evaluate counterfactual outcomes under different
policies. See Appendix B for additional details.

###  5.1 Experimental setup

#### Data.

The MovieLens 1M dataset (Harper and Konstan,, 2015) includes 1,000,209
ratings provided by 6,040 users and for 3,706 items, which we use to obtain
features, determine the dynamics, and emulate $\phi$. We sample and hold out
30% of all ratings $r_{ux}$ via user-stratified sampling, to which we apply
Collaborative Filtering (CF) to get user features $u$ and item features $x$
that approximate $u^{\top}x\approx r_{ux}$ ($d=8$, $\mathrm{RMSE}=0.917$,
$r\in[1,5]$). This mimics a process where $\phi$ is based on items recommended
by $\psi$ and rated by users. We then take the remaining data points and
randomly assign 1,000 users to the test set, on which we evaluate policies.
The remaining users are assigned to the train set, which is then further
partitioned into the main ${\cal{S}}$ and the different ${\cal{S}}^{(j)}$ per
experimental condition.

#### Recommendation policy and user behavior.

Following Kalimeris et al., (2021); Hron et al., (2022), we set $\psi$ to
recommend based on learned softmax scores,
$\mathrm{softmax}_{x}(\hat{r}_{u})$, taken over all non held-out $x$ having
ratings for $u$. For user behavior, we generate discrete interaction sequences
$S_{u}$ in a way that relates to our dynamic model, but is nonetheless quite
distinct. Specifically, each user $u$ is associated with _discrete-time_
latent-state variables, updated via an LV discretization scheme (see Appendix
B.5). These, together with recommended items $x_{i}=x(t_{i})$, determine
consecutive $\Delta t_{i}\in S_{u}$. Importantly, we let $\Delta t_{i}$ depend
on $u$’s rating for $x_{i}$, namely $r_{u,x_{i}}$, which we interpret as $u$’s
utility from consuming $x_{i}$. We parameterize this dependence via
‘immediate’, item-specific consumption gain parameters $\beta_{ui}$ that
depend on $r_{u,x_{i}}$, and for simplicity set
$\alpha_{u},\gamma_{u},\delta_{u}$ to be fixed.

Note that since $\beta_{ui}$ depends on the recommended $x_{i}$, it varies in
time, and hence there is no single $\beta$ that underlies the dynamics: even
in the limit ($\Delta t\rightarrow 0,T\rightarrow\infty$), user behavior
cannot be described by a continuous LV system, which implies
$\varepsilon_{\mathrm{LV}}>0$ (see Appendix Figure 5). Since the baseline RMSE
is high, we set $\beta_{ui}\propto\tilde{r}_{u,x_{i}}^{2}$, where
$\tilde{r}_{u,x_{i}}=\kappa r_{u,x_{i}}+(1-\kappa)u^{\top}x$, so that $\kappa$
interpolates between predicted ratings ($\kappa=0$) and true ratings
($\kappa=1$). This allows us to (indirectly) control
$\varepsilon_{\mathrm{pred}}$. For all experiments we use $T=100$, and so
expect a roughly fixed $\varepsilon_{\mathrm{dev}}>0$.

#### Methods.

We compare our approach (LV) to several baselines: (i) a default policy which
myopically optimizes for immediate engagement (and so does not break); (ii) a
‘safety switch’ policy (safety@$\tau$) that breaks once consumption surpasses
a threshold $\tau$; (iii) a prediction-based policy (best-of) that chooses the
best observed $p_{u}=\operatorname*{arg\,max}_{p_{j}}f_{j}(u)$ (rather than
optimizing over $p_{u}\in[0,1]$); and (iv) an oracle benchmark which directly
optimizes the (generally unknown) true underlying dynamics. We measure mean
long-term engagement rate (LTE) for each approach, and report averages and
standard errors computed over 10 random splits. Performance is measured
relative to the default baseline as it represents no change in policy (typical
absolute values are $\text{LTE}\approx 10$).

###  5.2 Results and analysis

#### Main results.

Figure 3 (left) compares the performance of our method to other policies. Here
we set $p_{0}=0$, use $N=3$ with $p_{j}\in{\left\\{{0.05,0.1,0.15}\right\\}}$,
and consider $\kappa=0.5$ (note $\kappa$ affects all policies). As can be
seen, our approach significantly improves over default (+6.48%). For
safety@$\tau$, improvement over the optimal $\tau=16$ (+5.68%; chosen in
hindsight) shows the importance of being preemptive; for the slightly smaller
$\tau=14$, breaks are harmful. The gap from best-of (+2.19%) quantifies the
gain from the optimization step in Eq. (9), and the close performance to
oracle (-0.918%) suggests that optimizing the empirical curve (Eq. (8)) works
well as as proxy.

#### User types.

Figure 3 (center) shows for our approach how gain in LTE varies across learned
breaking policies $\hat{p}_{u}>0$. For increasingly-accurate predictions
($\kappa\in\\{0,0.5,1\\}$), the main plot shows performance gains for each
group of users, partitioned by their $\hat{p}_{u}$ values (binned; plot shows
average and unit standard deviation per bin). Gains until $\hat{p}_{u}\leq
0.15$ are mild, but for $\hat{p}_{u}>0.15$, the general trend is positive:
users who are deemed to require more frequent breaks, benefit more from
breaking. Gains until $\hat{p}_{u}\leq 0.3$ increase for all $\kappa$, but for
$\hat{p}_{u}>0.3$, extrapolation becomes difficult: note the higher variation
within each $\kappa$, as well as significant differences across $\kappa$. This
highlights the importance of accurate predictions for inferring optimal
$\hat{p}_{u}$ when the experimental $p_{i}$ are small. The inlaied plot shows
that, in line with our theory, $\hat{p}_{u}$ exhibits an empirical phase shift
in the estimated $\hat{\theta}_{u}$.

#### Treatments.

Figure 3 (right) shows the effects of experimental treatments on performance.
Focusing on $N=2$, we fix $p_{0}=0$, and consider increasingly aggressive
experimentation by varying $p_{1}\in(0,0.4]$. For our approach, increasing
$p_{1}$ helps, which is anticipated by our theoretical bound. For the best-of
approach, larger $p_{1}$ also helps, but exhibits population-level optimum
($p_{1}\approx 0.24$), which is easy to ‘overshoot’. Note that when prediction
accuracy is low ($\kappa=0$), experimentation is essential: if $p_{1}$ is not
sufficiently large, then performance can sharply deteriorate.

Figure 3:  Results on the MovieLens 1M dataset. (Left) Performance gain of
different approaches (relative to default policy). (Center) Performance of LV
by user group, partitioned by learned policies $\hat{p}_{u}$. (Right)
Sensitivity to an increasingly aggressive experimental $p_{1}$
($N=2,p_{0}=0$).

##  6 Discussion

Our paper studies the novel problem of learning optimal breaking policies for
recommendation. We posit a tight connection between long-term engagement and
user well-being, and argue that optimizing the former requires careful
consideration of the latter. Viewing cognitive capacity as a limited but
conservable resource, we propose Lotka-Volterra dynamics as a behavioral model
that enables effective, responsible, and sustainable optimization of
recommendation ecosystems. As any policy task that involves humans, care must
be taken regarding potential risks. While our experiments show that optimizing
engagement also improves well-being, this need not always be the case; in
fact, merely measuring well-being in reality is challenging. Breaks, as
interventions, are presumably ‘safe’, in the sense that at worst they may lead
to suboptimal performance (for system) or satisfaction (for users). But breaks
can also be used nefariously, e.g., by enabling temporally-varying rewards
(Eyal,, 2019). As such, breaking policies should be deployed accountably and
transparently.

## Acknowledgements

Nir Rosenfeld is supported in part by The Israel Science Foundation (grant
278/22). Eden Saig is supported by a PBC scholarship for Ph.D. students in
data science. The authors thank anonymous reviewers for insightful discussions
and valuable suggestions.

## References

  * Baumeister et al., (1998) Baumeister, R. F., Bratslavsky, E., Muraven, M., and Tice, D. M. (1998).  Ego depletion: is the active self a limited resource.  Journal of personality and social psychology., 74(5).
  * Bergum and Lehr, (1962) Bergum, B. O. and Lehr, D. J. (1962).  Vigilance performance as a function of interpolated rest.  Journal of Applied Psychology, 46(6):425.
  * Cao et al., (2020) Cao, J., Sun, W., Shen, Z.-J. M., and Ettl, M. (2020).  Fatigue-aware bandits for dependent click models.  In AAAI, pages 3341–3348.
  * Chaney et al., (2018) Chaney, A. J., Stewart, B. M., and Engelhardt, B. E. (2018).  How algorithmic confounding in recommendation systems increases homogeneity and decreases utility.  In Proceedings of the 12th ACM conference on recommender systems, pages 224–232.
  * Chen and Plemmons, (2010) Chen, D. and Plemmons, R. J. (2010).  Nonnegativity constraints in numerical analysis.  In The birth of numerical analysis, pages 109–139. World Scientific.
  * Constine, (2018) Constine, J. (2018).  Instagram says ‘you’re all caught up’ in first time-well-spent feature. techcrunch.  [Online; accessed 23-September-2022].
  * Curmei et al., (2022) Curmei, M., Haupt, A. A., Recht, B., and Hadfield-Menell, D. (2022).  Towards psychologically-grounded dynamic preference models.  In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys ’22, page 35–48, New York, NY, USA. Association for Computing Machinery.
  * Danziger et al., (2011) Danziger, S., Levav, J., and Avnaim-Pesso, L. (2011).  Extraneous factors in judicial decisions.  Proceedings of the National Academy of Sciences, 108(17):6889–6892.
  * Dean and Morgenstern, (2022) Dean, S. and Morgenstern, J. (2022).  Preference dynamics under personalized recommendations.  In Proceedings of the 23rd ACM Conference on Economics and Computation, EC ’22, page 795–816, New York, NY, USA. Association for Computing Machinery.
  * Ding et al., (2016) Ding, X., Xu, J., Chen, G., and Xu, C. (2016).  Beyond smartphone overuse: identifying addictive mobile apps.  In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, pages 2821–2828.
  * Dubey et al., (2022) Dubey, R., Griffiths, T. L., and Dayan, P. (2022).  The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons.  PLoS Comput. Biol., 18(8).
  * Duncan et al., (2019) Duncan, J. P., Aubele-Futch, T., and McGrath, M. (2019).  A fast-slow dynamical system model of addiction: Predicting relapse frequency.  SIAM Journal on Applied Dynamical Systems, 18(2):881–903.
  * Eyal, (2019) Eyal, N. (2019).  Hooked : how to build habit-forming products / Nir Eyal with Ryan Hoover. Portfolio/Penguin, New York, updated edition edition.
  * Farmer, (2002) Farmer, J. D. (2002).  Market force, ecology and evolution.  Industrial and Corporate Change, 11(5):895–953.
  * Gilboa et al., (2008) Gilboa, S., Shirom, A., Fried, Y., and Cooper, C. (2008).  A meta-analysis of work demand stressors and job performance: examining main and moderating effects.  Personnel psychology, 61(2):227–271.
  * Gorbach et al., (2017) Gorbach, N. S., Bauer, S., and Buhmann, J. M. (2017).  Scalable variational inference for dynamical systems.  Advances in neural information processing systems, 30.
  * Gupta et al., (2006) Gupta, S., Hanssens, D., Hardie, B., Kahn, W., Kumar, V., Lin, N., Ravishanker, N., and Sriram, S. (2006).  Modeling customer lifetime value.  Journal of service research, 9(2):139–155.
  * Harper and Konstan, (2015) Harper, F. M. and Konstan, J. A. (2015).  The movielens datasets: History and context.  Acm transactions on interactive intelligent systems (tiis), 5(4):1–19.
  * Helton and Russell, (2017) Helton, W. S. and Russell, P. N. (2017).  Rest is still best: The role of the qualitative and quantitative load of interruptions on vigilance.  Human factors, 59(1):91–100.
  * Hennfng et al., (1989) Hennfng, R. A., Sauter, S. L., Salvendy, G., and Krieg Jr, E. F. (1989).  Microbreak length, performance, and stress in a data entry task.  Ergonomics, 32(7):855–864.
  * Hofbauer et al., (1998) Hofbauer, J., Sigmund, K., et al. (1998).  Evolutionary games and population dynamics.  Cambridge university press.
  * Hron et al., (2022) Hron, J., Krauth, K., Jordan, M. I., Kilbertus, N., and Dean, S. (2022).  Modeling content creator incentives on algorithm-curated platforms.  arXiv preprint arXiv:2206.13102.
  * Hug, (2020) Hug, N. (2020).  Surprise: A python library for recommender systems.  Journal of Open Source Software, 5(52):2174.
  * Jiang et al., (2019) Jiang, R., Chiappa, S., Lattimore, T., György, A., and Kohli, P. (2019).  Degenerate feedback loops in recommender systems.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 383–390.
  * Kahneman, (1973) Kahneman, D. (1973).  Attention and effort, volume 1063.  Citeseer.
  * Kalimeris et al., (2021) Kalimeris, D., Bhagat, S., Kalyanaraman, S., and Weinsberg, U. (2021).  Preference amplification in recommender systems.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 805–815.
  * Kleinberg et al., (2022) Kleinberg, J., Mullainathan, S., and Raghavan, M. (2022).  The challenge of understanding what users want: Inconsistent preferences and engagement optimization.  arXiv preprint arXiv:2202.11776.
  * Kleinberg and Immorlica, (2018) Kleinberg, R. and Immorlica, N. (2018).  Recharging bandits.  In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 309–319. IEEE.
  * Krauth et al., (2020) Krauth, K., Dean, S., Zhao, A., Guo, W., Curmei, M., Recht, B., and Jordan, M. I. (2020).  Do offline metrics predict online performance in recommender systems?  arXiv preprint arXiv:2011.07931.
  * Leqi et al., (2021) Leqi, L., Kilinc Karzan, F., Lipton, Z., and Montgomery, A. (2021).  Rebounding bandits for modeling satiation effects.  Advances in Neural Information Processing Systems, 34:4003–4014.
  * Lotka, (1910) Lotka, A. J. (1910).  Contribution to the theory of periodic reactions.  The Journal of Physical Chemistry, 14(3):271–274.
  * Madras et al., (2018) Madras, D., Pitassi, T., and Zemel, R. (2018).  Predict responsibly: improving fairness and accuracy by learning to defer.  Advances in Neural Information Processing Systems, 31.
  * Mansoury et al., (2020) Mansoury, M., Abdollahpouri, H., Pechenizkiy, M., Mobasher, B., and Burke, R. (2020).  Feedback loop and bias amplification in recommender systems.  In Proceedings of the 29th ACM international conference on information & knowledge management, pages 2145–2148.
  * Monge Roffarello and De Russis, (2019) Monge Roffarello, A. and De Russis, L. (2019).  The race towards digital wellbeing: Issues and opportunities.  In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1–14.
  * Montag et al., (2018) Montag, C., Zhao, Z., Sindermann, C., Xu, L., Fu, M., Li, J., Zheng, X., Li, K., Kendrick, K. M., Dai, J., et al. (2018).  Internet communication disorder and the structure of the human brain: Initial insights on wechat addiction.  Scientific reports, 8(1):1–10.
  * Mullette-Gillman et al., (2015) Mullette-Gillman, O. A., Leong, R. L., and Kurnianingsih, Y. A. (2015).  Cognitive fatigue destabilizes economic decision making preferences and strategies.  PloS one, 10(7):e0132022.
  * Muraven and Baumeister, (2000) Muraven, M. and Baumeister, R. F. (2000).  Self-regulation and depletion of limited resources: Does self-control resemble a muscle?  Psychological bulletin, 126(2):247.
  * Perez, (2018) Perez, S. (2018).  Apple unveils a new set of ‘digital wellness’ features for better managing screen time.  TechCrunch, June, 4.  [Online; accessed 23-September-2022].
  * Rauch, (2018) Rauch, J. (2018).  Slow media: Why slow is satisfying, sustainable, and smart.  Oxford University Press.
  * Robertson, (1977) Robertson, S. E. (1977).  The probability ranking principle in ir.  Journal of documentation.
  * Ross et al., (2014) Ross, H. A., Russell, P. N., and Helton, W. S. (2014).  Effects of breaks and goal switches on the vigilance decrement.  Experimental brain research, 232(6):1729–1737.
  * Ryder et al., (2018) Ryder, T., Golightly, A., McGough, A. S., and Prangle, D. (2018).  Black-box variational inference for stochastic differential equations.  In International Conference on Machine Learning, pages 4423–4432. PMLR.
  * Samuelson, (1998) Samuelson, L. (1998).  Evolutionary games and equilibrium selection, volume 1.  MIT press.
  * Sanna Passino et al., (2021) Sanna Passino, F., Maystre, L., Moor, D., Anderson, A., and Lalmas, M. (2021).  Where to next? a dynamic model of user preferences.  In Proceedings of the Web Conference 2021, WWW ’21, page 3210–3220, New York, NY, USA. Association for Computing Machinery.
  * Schmit and Riquelme, (2018) Schmit, S. and Riquelme, C. (2018).  Human interaction with recommendation systems.  In International Conference on Artificial Intelligence and Statistics, pages 862–870. PMLR.
  * Scholl et al., (2021) Scholl, M. P., Calinescu, A., and Farmer, J. D. (2021).  How market ecology explains market malfunction.  Proceedings of the National Academy of Sciences, 118(26):e2015574118.
  * Sievertsen et al., (2016) Sievertsen, H. H., Gino, F., and Piovesan, M. (2016).  Cognitive fatigue influences students’ performance on standardized tests.  Proceedings of the National Academy of Sciences, 113(10):2621–2624.
  * Takeuchi, (1996) Takeuchi, Y. (1996).  Global dynamical properties of Lotka-Volterra systems.  World Scientific.
  * Thoman et al., (2011) Thoman, D. B., Smith, J. L., and Silvia, P. J. (2011).  The resource replenishment function of interest.  Social Psychological and Personality Science, 2(6):592–599.
  * Trélat and Zuazua, (2015) Trélat, E. and Zuazua, E. (2015).  The turnpike property in finite-dimensional nonlinear optimal control.  Journal of Differential Equations, 258(1):81–114.
  * Wan and McAuley, (2018) Wan, M. and McAuley, J. (2018).  Item recommendation on monotonic behavior chains.  In Proceedings of the 12th ACM conference on recommender systems, pages 86–94.
  * Wan et al., (2019) Wan, M., Misra, R., Nakashole, N., and McAuley, J. (2019).  Fine-grained spoiler detection from large-scale review corpora.  arXiv preprint arXiv:1905.13416.
  * Wang and Lin, (2003) Wang, J.-C. and Lin, J.-P. (2003).  Are personalization systems really personal?-effects of conformity in reducing information.  In 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the, pages 10–pp. IEEE.
  * Warlop et al., (2018) Warlop, R., Lazaric, A., and Mary, J. (2018).  Fighting boredom in recommender systems with linear reinforcement learning.  Advances in Neural Information Processing Systems, 31.
  * Weibull, (1997) Weibull, J. W. (1997).  Evolutionary game theory.  MIT press.

##  Appendix A Deferred proofs

In this section, we formalize the model presented in section 3, and prove the
claims presented in section 4. The section ends with a formal proof of Theorem
1.

###  A.1 Properties of Lotka-Volterra systems

######  Definition 1 (Static policy equilibrium).

Let $\lambda(t),q(t)$ denote a Lokta-Volterra model characterized by
parameters $\theta=(\alpha,\beta,\gamma,\delta)\in\mathbb{R}^{4}_{+}$, as
defined in Equation 2. Let $p\in[0,1]$, and denote by $\pi_{p}$ the static
policy corresponding to $p$. For $\lambda(0),q(0)>0$, the static equilibrium
of the system is defined as:

$\displaystyle\lambda^{*}(p;\theta)$
$\displaystyle=\lim_{t\to\infty}\lambda(t)$ $\displaystyle q^{*}(p;\theta)$
$\displaystyle=\lim_{t\to\infty}q(t)$

We denote $\lambda^{*}(p)=\lambda^{*}(p;\theta)$ when $\theta$ is clear from
the context. We denote $\lambda^{*}(p;u)=\lambda^{*}(p;\theta_{u})$ when a
user $u\in{\mathcal{U}}$ characterized by parameters $\theta_{u}$ is given and
clear from the context.

######  Proposition 1 (Global stability).

$\lambda^{*}(p;\theta)$ exists and uniquely defined for all
$\theta\in\mathbb{R}_{+}^{4}$, $p\in[0,1]$ and for all initial conditions
$\lambda(0),q(0)>0$.

###### Proof.

See (Takeuchi,, 1996, Section 3.2). ∎

######  Lemma 1 (Equilibrium of LV behavioral model. Formal proof of Lemma 1).

Assume a Lokta-Volterra model characterized by
$\theta=(\alpha,\beta,\gamma,\delta)\in\mathbb{R}_{+}^{4}$, and let
$p\in[0,1]$ denote the proportion of interactions in which a forced break is
served. The static equilibrium of the model under static policy $\pi_{p}$ is
given by:

$\displaystyle\lambda^{*}(p)$
$\displaystyle=\begin{cases}\frac{\gamma}{\delta}\frac{1}{1-p}\left(1-\frac{\alpha}{\beta}\frac{1}{1-p}\right)&p\in\left[0,1-\frac{\alpha}{\beta}\right]\\\
0&\mathrm{otherwise}\end{cases}$ $\displaystyle q^{*}(p)$
$\displaystyle=\begin{cases}\frac{\alpha}{\beta}\frac{1}{1-p}&p\in\left[0,1-\frac{\alpha}{\beta}\right]\\\
1&\mathrm{otherwise}\end{cases}$

###### Proof.

The LV dynamical system is given by Equation 3:

$\displaystyle\frac{\mathrm{d}{\lambda}}{\mathrm{d}{t}}$
$\displaystyle=-\alpha\lambda+\beta(1-p)\lambda q$
$\displaystyle\frac{\mathrm{d}{q}}{\mathrm{d}{t}}$ $\displaystyle=\gamma
q(1-q)-\delta(1-p)\lambda q$

when $p\in\left[0,1-\frac{\alpha}{\beta}\right]$ we equate
$\frac{\mathrm{d}{\lambda}}{\mathrm{d}{t}}=0$,
$\frac{\mathrm{d}{q}}{\mathrm{d}{t}}=0$ and obtain the result. The solution is
guaranteed to be valid, as both $\lambda^{*}(p)>0$ and $q^{*}(p)\in[0,1]$.

Conversely, when $p\notin\left[0,1-\frac{\alpha}{\beta}\right]$, there exists
$\epsilon>0$ such that
$\frac{\mathrm{d}{}}{\mathrm{d}{t}}\log\lambda<-\epsilon<0$ for all
$\lambda>0$, $q\in[0,1]$. From this we obtain that $\log\lambda(t)$ tends
towards $-\infty$, and therefore $\lambda(t)$ tends towards $0$, and
$\lambda^{*}(p)=0$ as required. When $\lambda(t)$ is close to zero, the
interaction terms vanish in the $\frac{\mathrm{d}{q}}{\mathrm{d}{t}}$
equation, and $q(t)$ grows logistically towards $1$. ∎

######  Proposition 2 (Equilibrium bounds).

For a Lotka-Volterra model, the static equilibrium $\lambda^{*}(p)$ is bounded
by:

$0\leq\lambda^{*}(p)\leq\frac{\beta\gamma}{4\alpha\delta}$

###### Proof.

Denote $x=\frac{1}{1-p}$. From Lemma 1, for
$x\in\left[1,\frac{\beta}{\alpha}\right]$ the equilibrium consumption
$\lambda^{*}(x)$ is given by:

$\lambda^{*}(x)=\frac{\gamma}{\delta}x\left(1-\frac{\alpha}{\beta}x\right)$

and is zero otherwise. The equilibrium is a quadratic function of $x$ with
roots $x\in{\left\\{{0,\frac{\beta}{\alpha}}\right\\}}$, and therefore attains
its maximum at $x=\frac{\beta}{2\alpha}$. Plugging back the maximizing $x$
into $\lambda^{*}$ we obtain the upper bound. Lower bound is attained as the
equilibrium in Lemma 1 is clipped by $0$ from below. ∎

######  Lemma 2 (Optimal static policy. Formal proof of Lemma 1).

The optimal static policy for a Lotka-Volterra system is given by:

$p_{\mathrm{opt}}=\begin{cases}1-2\frac{\alpha}{\beta}&\frac{\alpha}{\beta}\leq\frac{1}{2}\\\
0&\frac{\alpha}{\beta}>\frac{1}{2}\end{cases}$

And the optimal equilibrium engagement rate is given by:

$\lambda^{*}_{\mathrm{opt}}=\begin{cases}\frac{\beta\gamma}{4\alpha\delta}&\frac{\alpha}{\beta}\leq\frac{1}{2}\\\
\frac{\gamma}{\delta}\left(1-\frac{\alpha}{\beta}\right)&\frac{\alpha}{\beta}>\frac{1}{2}\end{cases}$

###### Proof.

Denote $x=\frac{1}{1-p}$. From Proposition 2, the global maximum of
$\lambda^{*}(x)$ is attained at $x=\frac{\beta}{2\alpha}$. Consider two cases:
When $\frac{\alpha}{\beta}\leq\frac{1}{2}$, we obtain that
$x_{\mathrm{opt}}=\frac{\beta}{2\alpha}\geq 1$, and therefore
$p_{\mathrm{opt}}=1-\frac{1}{x}\in[0,1]$. From this we obtain that in this
case the global maximum is attained on the simplex, and given by the formula
from Proposition 2. Conversely, when $\frac{\alpha}{\beta}>\frac{1}{2}$, we
obtain $p=1-\frac{1}{x}<0$, and therefore $x_{\mathrm{opt}}$ translates to a
negative value of $p$. As $\lambda^{*}(p)$ is uni-modal, the optimal policy
restricted to the simplex $[0,1]$ in this case is attained on the closest
boundary point $p=0$.

Figure 2 provides graphical intuition for this proof (left and center
subplots). ∎

######  Proposition 3 (Inference of $\alpha/\beta$ from two-treatment
equilibrium data. Formal proof of Proposition 1).

Let $\lambda(t),q(t)$ be a Lokta-Volterra model, let $p_{1},p_{2}\in[0,1]$.
Denote by $\lambda^{*}(p_{1}),\lambda^{*}(p_{2})$ the static equilibrium rates
corresponding to static policies $\pi_{p_{1}},\pi_{p_{2}}$, and assume
$\lambda^{*}(p_{1}),\lambda^{*}(p_{2})>0$. The parameter ratio
$\frac{\alpha}{\beta}$ is given by the following formula:

$\frac{\alpha}{\beta}=\frac{(1-p_{2})\lambda^{*}(p_{2})-(1-p_{1})\lambda^{*}(p_{1})}{\frac{1}{1-p_{1}}-\frac{1}{1-p_{2}}}$

###### Proof.

From Lemma 1, the equilibrium consumption $\lambda^{*}(p)$ is given by:

$\displaystyle\lambda^{*}(p)$
$\displaystyle=\frac{\gamma}{\delta}\frac{1}{1-p}\left(1-\frac{\alpha}{\beta}\frac{1}{1-p}\right)$
$\displaystyle=\frac{\gamma}{\delta}\frac{1}{1-p}-\frac{\alpha}{\beta}\frac{\gamma}{\delta}\left(\frac{1}{1-p}\right)^{2}$

When $\lambda^{*}(p_{i})$ is observed for different policies
$p_{1},\dots,p_{m}\in\left[0,1-\frac{\alpha}{\beta}\right]$, we obtain a
polynomial regression problem for the parameters $\frac{\alpha}{\beta}$ and
$\frac{\alpha}{\beta}\frac{\gamma}{\delta}$, which can be solved e.g using
Non-Negative Least Squares.

When $m=2$, we obtain a system of two linear equations. Apply Cramer’s rule to
obtain:

$\displaystyle\frac{\gamma}{\delta}$
$\displaystyle=\frac{\frac{\lambda^{*}(p_{2})}{(1-p_{1})^{2}}-\frac{\lambda^{*}(p_{1})}{(1-p_{2})^{2}}}{\frac{1}{(1-p_{1})(1-p_{2})^{2}}-\frac{1}{(1-p_{1})^{2}(1-p_{2})}}=\frac{(1-p_{2})^{2}\lambda^{*}(p_{2})-(1-p_{1})^{2}\lambda^{*}(p_{1})}{p_{2}-p_{1}}$
(11) $\displaystyle\frac{\alpha}{\beta}\frac{\gamma}{\delta}$
$\displaystyle=\frac{\frac{\lambda^{*}(p_{2})}{(1-p_{1})}-\frac{\lambda^{*}(p_{1})}{(1-p_{2})}}{\frac{1}{(1-p_{1})(1-p_{2})^{2}}-\frac{1}{(1-p_{1})^{2}(1-p_{2})}}=(1-p_{1})(1-p_{2})\frac{(1-p_{2})\lambda^{*}(p_{2})-(1-p_{1})\lambda^{*}(p_{1})}{p_{2}-p_{1}}$
(12)

And therefore $\frac{\alpha}{\beta}$ is given by:

$\displaystyle\frac{\alpha}{\beta}$
$\displaystyle=\frac{\frac{\lambda^{*}(p_{2})}{(1-p_{1})}-\frac{\lambda^{*}(p_{1})}{(1-p_{2})}}{\frac{\lambda^{*}(p_{2})}{(1-p_{1})^{2}}-\frac{\lambda^{*}(p_{1})}{(1-p_{2})^{2}}}=(1-p_{1})(1-p_{2})\frac{(1-p_{2})\lambda^{*}(p_{2})-(1-p_{1})\lambda^{*}(p_{1})}{(1-p_{2})^{2}\lambda^{*}(p_{2})-(1-p_{1})^{2}\lambda^{*}(p_{1})}$

∎

###  A.2 Model fitting from engagement predictions

#### Notations.

In this section only, we use the common notation $q=1-p$ to denote
complementary probabilities.

######  Definition 2 (Empirical value of $\alpha/\beta$).

For single-channel experiments with forced-break probabilities $p_{1},p_{2}$,
denote $\lambda_{i}=\lambda^{*}(p_{i})$, $f_{i}=f_{p_{i}}(u)$,
$q_{i}=1-p_{i}$. The empirical value of the $\frac{\alpha}{\beta}$ parameter
is given by the following formula:

$\displaystyle\hat{\frac{\alpha}{\beta}}=\frac{q_{1}q_{2}\left(q_{1}f_{1}-q_{2}f_{2}\right)}{q_{1}^{2}f_{1}-q_{2}^{2}f_{2}}$

######  Proposition 4 ($\alpha/\beta$ estimation error from prediction
errors).

Given a single-channel Lokta-Volterra system with parameter
$\frac{\alpha}{\beta}\geq 1$. Let
$p_{1},p_{2}\in\left[1,\frac{\alpha}{\beta}\right]$, denote
$\lambda^{*}_{i}=\lambda^{*}(p_{i})\in\mathbb{R}_{+}$, and let
$f_{i}=\lambda^{*}_{i}+\varepsilon_{i}$ be the predicted engagement rates
corresponding to $p_{1},p_{2}$. When
${\left|{\varepsilon_{1}}\right|},{\left|{\varepsilon_{2}}\right|}\leq\varepsilon\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}$,
the estimation error is bounded by:

${\left|{\frac{\alpha}{\beta}-\hat{\frac{\alpha}{\beta}}}\right|}\leq\frac{\varepsilon}{{\left|{p_{1}-p_{2}}\right|}}\frac{\beta\delta}{\alpha\gamma}$

###### Proof.

denote $q_{i}=1-p_{i}$. The value of $\frac{\alpha}{\beta}$ is given by
Proposition 3:

$\frac{\alpha}{\beta}=\frac{q_{1}q_{2}\left(q_{1}\lambda^{*}_{1}-q_{2}\lambda^{*}_{2}\right)}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}}$

And the estimator for $\frac{\alpha}{\beta}$ is obtained by replacing the true
value with their predictions:

$\displaystyle\hat{\frac{\alpha}{\beta}}$
$\displaystyle=\frac{q_{1}q_{2}\left(q_{1}f_{1}-q_{2}f_{2}\right)}{q_{1}^{2}f_{1}-q_{2}^{2}f_{2}}$
$\displaystyle=\frac{q_{1}q_{2}\left(q_{1}(\lambda^{*}_{1}+\varepsilon_{1})-q_{2}(\lambda^{*}_{2}+\varepsilon_{2})\right)}{q_{1}^{2}(\lambda^{*}_{1}+\varepsilon_{1})-q_{2}^{2}(\lambda^{*}_{2}+\varepsilon_{2})}$

The estimation error is given by:

$\displaystyle{\left|{\frac{\alpha}{\beta}-\hat{\frac{\alpha}{\beta}}}\right|}$
$\displaystyle={\left|{\frac{q_{1}^{2}q_{2}^{2}(q_{1}-q_{2})(\varepsilon_{2}\lambda^{*}_{1}-\varepsilon_{1}\lambda^{*}_{2})}{(q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2})(q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}-(q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2}))}}\right|}$
$\displaystyle=\underbrace{\left(q_{1}q_{2}\right)^{2}}_{\equiv\text{(i)}}\underbrace{{\left|{\frac{q_{1}-q_{2}}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}}}\right|}}_{\equiv\text{(ii)}}\underbrace{{\left|{\varepsilon_{2}\lambda^{*}_{1}-\varepsilon_{1}\lambda^{*}_{2}}\right|}}_{\equiv\text{(iii)}}\underbrace{{\left|{\frac{1}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}-(q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2})}}\right|}}_{\equiv\text{(iv)}}$

We now proceed to bound each factor:

  * •

For (i), the term $\left(q_{1}q_{2}\right)^{2}$ is bounded by $1$ since
$q_{1},q_{2}\in[0,1]$.

  * •

For (ii), the term
${\left|{\frac{q_{1}-q_{2}}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}}}\right|}$
is equal to $\left(\frac{\gamma}{\delta}\right)^{-1}$ by Eq. (11).

  * •

For (iii), from Proposition 2 we obtain the bound
$0\leq\lambda^{*}_{i}\leq\frac{\beta\gamma}{4\alpha\delta}$, and therefore the
term
${\left|{\varepsilon_{2}\lambda^{*}_{1}-\varepsilon_{1}\lambda^{*}_{2}}\right|}$
is bounded by
$2\left(\frac{\beta\gamma}{4\alpha\delta}\right)\varepsilon=\frac{\beta\gamma}{2\alpha\delta}\varepsilon$.

  * •

For (iv), the term
${\left|{\frac{1}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}-(q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2})}}\right|}$
is equal to:

(iv)
$\displaystyle\equiv{\left|{\frac{1}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}-(q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2})}}\right|}$
$\displaystyle=\frac{1}{{\left|{p_{1}-p_{2}}\right|}}{\left|{\frac{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}-(q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2})}{p_{1}-p_{2}}}\right|}^{-1}$
$\displaystyle=\frac{1}{{\left|{p_{1}-p_{2}}\right|}}{\left|{\underbrace{\frac{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}}{p_{1}-p_{2}}}_{\text{Eq.~{}(\ref{eq:lv_one_channel_gamma_delta_estimator})}}-\frac{q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2}}{p_{1}-p_{2}}}\right|}^{-1}$
$\displaystyle=\frac{1}{{\left|{p_{1}-p_{2}}\right|}}{\left|{\frac{\gamma}{\delta}-\frac{q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2}}{p_{1}-p_{2}}}\right|}^{-1}$

Note that
${\left|{\frac{q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2}}{p_{1}-p_{2}}}\right|}\leq\frac{2\varepsilon}{{\left|{p_{1}-p_{2}}\right|}}$.
When $\varepsilon$ is small enough, and specifically when the bound
$\varepsilon\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}$
holds, we obtain:

$\displaystyle{\left|{\frac{\gamma}{\delta}-\frac{q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2}}{p_{1}-p_{2}}}\right|}^{-1}$
$\displaystyle\leq\frac{\delta}{\gamma}{\left|{1-\frac{1}{2}}\right|}^{-1}\leq
2\frac{\delta}{\gamma}$

and therefore:

(iv)
$\displaystyle\leq\frac{2}{{\left|{p_{1}-p_{2}}\right|}}\frac{\delta}{\gamma}$

Aggregating results (i)-(iv) above, we obtain the overall bound:

$\displaystyle{\left|{\frac{\alpha}{\beta}-\hat{\frac{\alpha}{\beta}}}\right|}$
$\displaystyle=\underbrace{\left(q_{1}q_{2}\right)^{2}}_{\leq
1}\underbrace{{\left|{\frac{q_{1}-q_{2}}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}}}\right|}}_{=\frac{\delta}{\gamma}}\underbrace{{\left|{\varepsilon_{2}\lambda^{*}_{1}-\varepsilon_{1}\lambda^{*}_{2}}\right|}}_{\leq\frac{\beta\gamma}{2\alpha\delta}\varepsilon}\underbrace{{\left|{\frac{1}{q_{1}^{2}\lambda^{*}_{1}-q_{2}^{2}\lambda^{*}_{2}-(q_{1}^{2}\varepsilon_{1}-q_{2}^{2}\varepsilon_{2})}}\right|}}_{\leq\frac{2}{{\left|{p_{1}-p_{2}}\right|}}\frac{\delta}{\gamma}}$
$\displaystyle\leq\frac{\varepsilon}{{\left|{p_{1}-p_{2}}\right|}}\frac{\beta\delta}{\alpha\gamma}$

∎

######  Proposition 5 (Cost of $\alpha/\beta$ estimation error).

Let $\frac{\alpha}{\beta}$ be the engagement ratio parameter of a one-channel
Lotka-Volterra system, and let $\hat{\left(\frac{\alpha}{\beta}\right)}$ be an
estimate of these parameters. Let $\lambda^{*}_{\mathrm{opt}}$ be the
engagement rate of the optimal static policy, and denote
$\lambda^{*}(x)=\lambda^{*}\left(\hat{p}(x)\right)$. When
${\left|{\frac{\alpha}{\beta}-\hat{\left(\frac{\alpha}{\beta}\right)}}\right|}\leq\min{\left\\{{\frac{\alpha}{2\beta},1}\right\\}}$
The price of estimation error is bounded by:

$\lambda^{*}_{\mathrm{opt}}-\lambda^{*}\left(\hat{\left(\frac{\alpha}{\beta}\right)}\right)\leq\left(\frac{\gamma}{\delta}\right)\min{\left\\{{\left(2\frac{\alpha}{\beta}\right)^{-2}{\left|{\frac{\alpha}{\beta}-\hat{\left(\frac{\alpha}{\beta}\right)}}\right|},\left(4\frac{\alpha}{\beta}\right)^{-1}}\right\\}}$

###### Proof.

Denote $r=\frac{\alpha}{\beta},x=\hat{\left(\frac{\alpha}{\beta}\right)}$, and
assume without loss of generality that $\frac{\gamma}{\delta}=1$ and $r\leq
1$. The optimal equilibrium engagement rate is given by:

$\lambda^{*}_{\mathrm{opt}}=\begin{cases}\frac{1}{4r}&r\in\left(0,\frac{1}{2}\right]\\\
1-r&r\in\left(\frac{1}{2},1\right]\end{cases}$

The chosen policy $\hat{p}(x)$ is given by:

$\hat{p}(x)=\begin{cases}1-2x&x\in\left[0,\frac{1}{2}\right]\\\
0&\mathrm{otherwise}\end{cases}$

Assume without loss of generality that $x\in\left[0,\frac{1}{2}\right]$, as
values of $x$ outside the interval can be clipped to its edges without
affecting the result. The equilibrium engagement rate of the selected policy
is given by:

$\lambda^{*}(x)=\lambda^{*}\left(\hat{p}(x)\right)=\begin{cases}0&x\in\left[0,\frac{r}{2}\right]\\\
\frac{1}{2x}\left(1-\frac{r}{2x}\right)&x\in\left(\frac{r}{2},\frac{1}{2}\right]\end{cases}$

Denote $\Delta(x)=\lambda^{*}_{\mathrm{opt}}-\lambda^{*}(x)$. We obtain:

$\Delta(x)=\lambda^{*}_{\mathrm{opt}}-\lambda^{*}(x)=\begin{cases}\frac{1}{4r}&r\in\left(0,\frac{1}{2}\right],x\in\left[0,\frac{r}{2}\right]\\\
\frac{\left(x-r\right)^{2}}{4x^{2}r}&r\in\left(0,\frac{1}{2}\right],x\in\left(\frac{r}{2},\frac{1}{2}\right]\\\
(1-r)&r\in\left(\frac{1}{2},1\right],x\in\left[0,\frac{r}{2}\right]\\\
(1-r)-\frac{1}{2x}\left(1-\frac{r}{2x}\right)&r\in\left(\frac{1}{2},1\right],x\in\left(\frac{r}{2},\frac{1}{2}\right]\\\
\end{cases}$

Observe that $\frac{1}{4r}\geq 1-r$ for all $r\in(0,1]$, and therefore we
obtain for all $x,r$:

$\Delta(x)\leq\frac{1}{4r}$ (13)

From the convexity of $\Delta(x)$ in the region around $x=r$ we obtain:

$\Delta(x)\leq\frac{1}{2r^{2}}{\left|{x-r}\right|}$ (14)

Finally, combining the two bounds yields the final result. A geometric
interpretation of this claim is illustrated in Figure 4. ∎

Figure 4: Graphical illustration of Proposition 5. Cost of estimation error
for different values of $\frac{\alpha}{\beta}$, and their corresponding upper
bounds given by the claim.

###  A.3 Optimal stationary policy from engagement predictions

######  Definition 3 (Expected observable rate).

Let $u\in{\mathcal{U}}$, $p\in[0,1]$, and $T>0$. Let $p\in[0,1]$, denote the
corresponding static policy by $\pi_{p}$. The expected observable rate
$\bar{\lambda}_{u}(p;T)$ is defined as:

$\bar{\lambda}_{u}(p;u)=\mathbb{E}_{\pi}{\left[{\frac{1}{T}{\left|{\mathrm{TPP}_{\pi_{p}}(u;T)}\right|}}\right]}$

where expectation is taken over the stochastic decisions of $\pi_{p}$.

######  Definition 4 (Lokta-Volterra approximation of $\mathrm{TPP}$).

Let $u\in{\mathcal{U}}$, and $T>0$. Denote by $p^{*}$ the maximizer of
expected observable rate:

$p^{*}=\operatorname*{arg\,max}_{p\in[0,1]}\bar{\lambda}_{u}(p;u)$

The LV approximation of $\mathrm{TPP}(u;T)$ is defined as:

$\theta^{*}_{u}=\operatorname*{arg\,min}_{\theta}\max_{p\in[0,1]}{{\left|{\bar{\lambda}_{u}(p;u)-\lambda^{*}(p;\theta)}\right|}}$

such that $\operatorname*{arg\,max}_{p}\lambda^{*}(p;\theta)=p^{*}$. The
corresponding approximation error is defined as:

$\varepsilon_{\mathrm{LV},u}=\max_{p\in[0,1]}{{\left|{\bar{\lambda}_{u}(p;u)-\lambda^{*}(p;\theta^{*}_{u})}\right|}}$

#### Notations.

When $u$ is clear from the context, we denote $\theta^{*}=\theta^{*}_{u}$,
$\varepsilon_{\mathrm{LV}}=\varepsilon_{\mathrm{LV},u}$. We use
$\alpha^{*},\beta^{*},\dots$ to refer to the corresponding parts of the Lokta-
Volterra parameters vector $\theta^{*}$.

We are now ready to state and prove the main theorem for this section:

######  Theorem 1 (Regret bound for learned static policy. Formal version of
Theorem 1).

Let $p_{1},p_{2}\in[0,1]$ denote two static forced-break policies, and denote
by ${\mathcal{U}}$ the set of users, and assume they remain engaged under the
stationary policies $\pi(p_{1})$ and $\pi(p_{2})$. Assume
$S_{u}(p;T)\sim\mathrm{TPP}_{\pi_{p}\circ\psi}(u;T)$, and let
$\mu=\left(\max_{u\in{\mathcal{U}}}\frac{\bar{\gamma}_{u}}{\bar{\delta}_{u}}\right)\cdot\left(\max_{u^{\prime}\in{\mathcal{U}}}\frac{\bar{\delta}_{u^{\prime}}}{\bar{\gamma}_{u^{\prime}}}\right)$,
$\nu=\max_{u\in{\mathcal{U}}}\left(\frac{\bar{\beta}_{u}}{\bar{\alpha}_{u}}\right)$.

Let $f_{p_{1}},f_{p_{2}}:{\mathcal{U}}\to\mathbb{R}_{+}$ be functions
predicting
${\tfrac{1}{T}{\left|{S_{u}\left({p_{1}};T\right)}\right|}},{\tfrac{1}{T}{\left|{S_{u}\left({p_{2}};T\right)}\right|}}$,
respectively. Denote the learned policy by $\hat{p}$, and the optimal policy
by $p^{*}$.

If (i) the expected RMSE of $f_{p_{1}},f_{p_{2}}$ is bounded by
$\varepsilon_{\mathrm{pred}}$, (ii) the average absolute deviation of
$\frac{1}{T}{\left|{\mathrm{TPP}(u;T)}\right|}$ is bounded by
$\varepsilon_{\mathrm{dev}}$, and (iii) the expected LV approximation error of
the system is bounded by $\varepsilon_{\mathrm{LV}}$, then the learned policy
$\hat{p}$ has bounded regret:

$\mathbb{E}_{u,\pi}{\left[{{\left|{{\tfrac{1}{T}{\left|{S_{u}\left({p^{*}};T\right)}\right|}}-{\tfrac{1}{T}{\left|{S_{u}\left({\hat{p}};T\right)}\right|}}}\right|}}\right]}\leq\frac{\eta_{\mathrm{TPP}}}{{\left|{p_{1}-p_{2}}\right|}}(\varepsilon_{\mathrm{pred}}+\varepsilon_{\mathrm{dev}}+\varepsilon_{\mathrm{LV}})$

where expectation is taken over stochastic choices of policies, and
$\eta_{\mathrm{TPP}}=g(\mu,\nu)\in\mathrm{poly}(\mu,\nu)$.

###### Proof.

By assumption (i), the functions $f_{p_{1}},f_{p_{2}}$ have bounded expected
RMSE:

$\mathbb{E}_{u}{\left[{\left(f_{p_{i}}(u)-{\tfrac{1}{T}{\left|{S_{u}\left({p_{i}};T\right)}\right|}}\right)^{2}}\right]}\leq\varepsilon_{\mathrm{pred}}^{2}$
(15)

Applying Jensen’s inequality with the convex function $\varphi(x)=x^{2}$
yields:

$\left(\mathbb{E}_{u}{\left[{{\left|{f_{p_{i}}(u)-{\tfrac{1}{T}{\left|{S_{u}\left({p_{i}};T\right)}\right|}}}\right|}}\right]}\right)^{2}\leq\mathbb{E}_{u}{\left[{\left(f_{p_{i}}(u)-{\tfrac{1}{T}{\left|{S_{u}\left({p_{i}};T\right)}\right|}}\right)^{2}}\right]}$

Combining with Eq. (15) and taking the square root, we obtain an upper bound
on the expected absolute error:

$\mathbb{E}_{u}{\left[{{\left|{f_{p_{i}}(u)-{\tfrac{1}{T}{\left|{S_{u}\left({p_{i}};T\right)}\right|}}}\right|}}\right]}\leq\varepsilon_{\mathrm{pred}}$
(16)

Let $\Delta_{f}={\left|{f_{p_{i}}(u)-\lambda^{*}(p_{i})}\right|}$ apply the
triangle inequality to obtain:

$\displaystyle\Delta_{f}$
$\displaystyle={\left|{f_{p_{i}}(u)-\lambda^{*}(p_{i})}\right|}$
$\displaystyle\leq{\left|{f_{p_{i}}(u)-{\tfrac{1}{T}{\left|{S_{u}\left({u};T\right)}\right|}}}\right|}+{\left|{{\tfrac{1}{T}{\left|{S_{u}\left({u};T\right)}\right|}}-\bar{\lambda}(p_{i};u)}\right|}+{\left|{\bar{\lambda}(p_{i};u)-\lambda^{*}(p_{i})}\right|}$

Denote
$\varepsilon_{f}=\varepsilon_{\mathrm{pred}}+\varepsilon_{\mathrm{dev}}+\varepsilon_{\mathrm{LV}}$.
Applying the triangle inequality and using the bounds in Eq. (16) together
with assumptions (ii), (iii), we obtain:

$\displaystyle\mathbb{E}_{u,\pi}{\left[{\Delta_{f}}\right]}\leq$
$\displaystyle\mathbb{E}_{u}{\left[{{\left|{f_{p_{i}}(u)-{\tfrac{1}{T}{\left|{S_{u}\left({u};T\right)}\right|}}}\right|}}\right]}$
$\displaystyle+\mathbb{E}_{u,\pi}{\left[{{\left|{{\tfrac{1}{T}{\left|{S_{u}\left({u};T\right)}\right|}}-\bar{\lambda}(p_{i};u)}\right|}}\right]}$
$\displaystyle+\mathbb{E}_{u}{\left[{{\left|{\bar{\lambda}(p_{i};u)-\lambda^{*}(p_{i};\theta^{*}_{u})}\right|}}\right]}$
$\displaystyle\leq$
$\displaystyle\varepsilon_{\mathrm{pred}}+\varepsilon_{\mathrm{dev}}+\varepsilon_{\mathrm{LV}}=\varepsilon_{f}$
(17)

Denote $\theta^{*}_{u}=(\alpha,\beta,\gamma,\delta)$. The empirical value
$\hat{\left(\frac{\alpha}{\beta}\right)}$ of
$\left(\frac{\alpha}{\beta}\right)$ is given by Definition 2. Denote the
estimation error by
$\Delta_{\frac{\alpha}{\beta}}={\left|{\hat{\left(\frac{\alpha}{\beta}\right)}-{\left(\frac{\alpha}{\beta}\right)}}\right|}$.

By Proposition 4, the following pointwise upper bound on
$\Delta_{\frac{\alpha}{\beta}}$ applies when
$\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}$:

$\Delta_{\frac{\alpha}{\beta}}\leq\frac{\Delta_{f}}{{\left|{p_{1}-p_{2}}\right|}}\frac{\beta\delta}{\alpha\gamma}$
(18)

Plugging in the bound on the expected value of $\Delta_{f}$ into Eq. (18), we
obtain in expectation:

$\displaystyle\mathbb{E}_{u,\pi}{\left[{\Delta_{\frac{\alpha}{\beta}}\mid\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$
$\displaystyle\leq\mathbb{E}_{u,\pi}{\left[{\frac{\Delta_{f}}{{\left|{p_{1}-p_{2}}\right|}}\frac{\beta\delta}{\alpha\gamma}\mid\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$
$\displaystyle\leq\frac{\varepsilon_{f}}{{\left|{p_{1}-p_{2}}\right|}}\max_{u}\frac{\beta\delta}{\alpha\gamma}$
(19)

Next, we apply Proposition 5. Denote
$\Delta_{\lambda^{*}}=\lambda^{*}(p^{*})-\lambda^{*}(\hat{p})$, and define the
following probability event:

$A=\left(\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}\right)\text{
and }\left(\Delta_{\frac{\alpha}{\beta}}\leq\frac{1}{2\nu}\right)$

Note that the bound in Proposition 5 is represented as a minimum between two
functions, one linear in $\varepsilon$ and one constant. To leverage this
property, apply the law of total expectation:

$\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}}\right]}=\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid
A}\right]}\mathbb{P}{\left[{A}\right]}+\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid\bar{A}}\right]}\mathbb{P}{\left[{\bar{A}}\right]}$
(20)

Under $A$, the first term in Eq. (20) can be bounded by the linear term in
Proposition 5. Taking $\mathbb{P}{\left[{A}\right]}\leq 1$ and combining with
equation Eq. (18):

$\displaystyle\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid
A}\right]}\mathbb{P}{\left[{A}\right]}$
$\displaystyle\leq\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid
A}\right]}$
$\displaystyle\leq\mathbb{E}_{u,\pi}{\left[{\frac{\beta^{2}\gamma}{2\alpha^{2}\delta}\Delta_{\frac{\alpha}{\beta}}\mid
A}\right]}$
$\displaystyle\leq\mathbb{E}_{u,\pi}{\left[{\frac{\beta^{2}\gamma}{2\alpha^{2}\delta}\frac{\Delta_{f}}{{\left|{p_{1}-p_{2}}\right|}}\frac{\beta\delta}{\alpha\gamma}\mid
A}\right]}$
$\displaystyle\leq\frac{\nu^{3}}{2{\left|{p_{1}-p_{2}}\right|}}\varepsilon_{f}$
(21)

The expectation factor in the second term of Eq. (20) can be bounded by the
constant term in Proposition 5:

$\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid\bar{A}}\right]}\leq\frac{1}{4}\max_{u}\frac{\beta\gamma}{\alpha\delta}\leq\frac{\nu}{4}\max_{u}\frac{\gamma}{\delta}$
(22)

Decompose the probability factor $\mathbb{P}{\left[{\bar{A}}\right]}$ using
the law of total probability:

$\displaystyle\mathbb{P}{\left[{\bar{A}}\right]}$
$\displaystyle=\mathbb{P}{\left[{\Delta_{f}>\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}+\mathbb{P}{\left[{\Delta_{\frac{\alpha}{\beta}}>\frac{1}{2\nu}\mid\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}\mathbb{P}{\left[{\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$
$\displaystyle\leq\mathbb{P}{\left[{\Delta_{f}>\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}+\mathbb{P}{\left[{\Delta_{\frac{\alpha}{\beta}}>\frac{1}{2\nu}\mid\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$

Apply Markov’s inequality $\mathbb{P}{\left[{{\left|{X}\right|}\geq
a}\right]}\leq\frac{\mathbb{E}{\left[{{\left|{X}\right|}}\right]}}{a}$ on the
probabilities to obtain:

$\displaystyle\mathbb{P}{\left[{\Delta_{f}>\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$
$\displaystyle\leq\mathbb{E}_{u,\pi}{\left[{\Delta_{f}}\right]}\left(\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}\right)^{-1}$
$\displaystyle\underset{\text{by
Eq.~{}(\ref{eq:total_alpha_beta_estimation_error})}}{\leq}\varepsilon_{f}\frac{4}{{\left|{p_{1}-p_{2}}\right|}}\max_{u}\frac{\delta}{\gamma}$
(23)
$\displaystyle\mathbb{P}{\left[{\Delta_{\frac{\alpha}{\beta}}>\frac{1}{2\nu}\mid\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$
$\displaystyle\leq\mathbb{E}_{u,\pi}{\left[{\Delta_{\frac{\alpha}{\beta}}\mid\Delta_{f}\leq\frac{\gamma}{\delta}\frac{{\left|{p_{1}-p_{2}}\right|}}{4}}\right]}$
$\displaystyle\underset{\text{by
Eq.~{}(\ref{eq:alpha_beta_error_small_epsilon_upper_bound_expectation})}}{\leq}\frac{\varepsilon_{f}}{{\left|{p_{1}-p_{2}}\right|}}\max_{u}\frac{\beta\delta}{\alpha\gamma}$
$\displaystyle\leq\frac{\varepsilon_{f}}{{\left|{p_{1}-p_{2}}\right|}}\nu\max_{u}\frac{\delta}{\gamma}$
(24)

Plugging back equations Eq. (21), Eq. (22),Eq. (23),Eq. (24) into equation Eq.
(20), we obtain bounds for each term:

$\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}}\right]}=\underbrace{\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid
A}\right]}\mathbb{P}{\left[{A}\right]}}_{\text{by
Eq.~{}(\ref{eq:delta_lambda_star_upper_bound_linear_term})}}+\underbrace{\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}\mid\bar{A}}\right]}}_{\text{by
Eq.~{}(\ref{eq:expected_cost_of_estimation_error_large_deviation_upper_bound})}}\underbrace{\mathbb{P}{\left[{\bar{A}}\right]}}_{\text{by
Eq.~{}(\ref{eq:large_delta_f_probability_bound}),Eq.~{}(\ref{eq:large_delta_alpha_beta_probability_bound})}}$
(25)

we obtain:

$\mathbb{E}_{u,\pi}{\left[{\Delta_{\lambda^{*}}}\right]}\leq\frac{\varepsilon_{f}}{{\left|{p_{1}-p_{2}}\right|}}\left(\frac{\nu^{3}}{2}+\left(\nu+\frac{\nu^{2}}{4}\right)\mu\right)=\varepsilon_{\lambda^{*}}$

To obtain the regret bound on the empirical rates, we apply assumptions (ii),
(iii) once again to bound the expected difference between $\lambda^{*}(p)$ and
${\tfrac{1}{T}{\left|{S_{u}\left({p};T\right)}\right|}}$, and apply the
triangle inequality:

$\displaystyle\mathbb{E}_{u,\pi}{\left[{{\left|{{\tfrac{1}{T}{\left|{S_{u}\left({p^{*}};T\right)}\right|}}-{\tfrac{1}{T}{\left|{S_{u}\left({\hat{p}};T\right)}\right|}}}\right|}}\right]}$
$\displaystyle\leq\varepsilon_{\lambda^{*}}+2(\varepsilon_{\mathrm{dev}}+\varepsilon_{\mathrm{LV}})$

Note that $\frac{\nu}{{\left|{p_{1}-p_{2}}\right|}}>1$, as
$\frac{\beta}{\alpha}\geq 1$ since all the users are assumed to remain engaged
in the long term, and ${\left|{p_{1}-p_{2}}\right|}\leq 1$ as
$p_{1},p_{2}\in[0,1]$. Therefore, the function
$\eta_{\mathrm{TPP}}=g(\mu,\nu)=\left(\frac{\nu^{3}}{2}+\left(\nu+\frac{\nu^{2}}{4}\right)\eta+2\nu\right)$
satisfies:

$\mathbb{E}_{u,\pi}{\left[{{\left|{{\tfrac{1}{T}{\left|{S_{u}\left({p^{*}};T\right)}\right|}}-{\tfrac{1}{T}{\left|{S_{u}\left({\hat{p}};T\right)}\right|}}}\right|}}\right]}\leq\frac{\eta_{\mathrm{TPP}}}{{\left|{p_{1}-p_{2}}\right|}}(\varepsilon_{\mathrm{pred}}+\varepsilon_{\mathrm{dev}}+\varepsilon_{\mathrm{LV}})$

∎

##  Appendix B Experimental details

###  B.1 Data

#### MovieLens-1M

We base our main experimental environment on the MovieLens-1M dataset, which
is a standard benchmark dataset used widely in recommendation system research
(Harper and Konstan,, 2015). The dataset includes 1,000,209 ratings provided
by 6,040 users and for 3,706 movies. Ratings are in the range
$\\{1,\dots,5\\}$, and all users in the dataset have at least 20 reported
ratings. The dataset is publicly available at:
https://grouplens.org/datasets/movielens/1m/.

#### Data partitioning.

To learn latent user and item features, 30% of all ratings were drawn at
random. Stratified sampling was applied to ensure that all users and items
were covered, and so that each users have roughly the same proportion of
ratings used for this step. These ratings were only used only for learning a
CF model, and were discarded afterwards. The remaining 70% data points were
used for training and testing. For these, we first randomly sampled 1,000
users to form the test set. Then, the remaining users were partitioned into
the main train set ${\cal{S}}$, which included 70% ($\approx$3,528) of these
users, and the experimental treatment sets ${\cal{S}}^{(j)}$, each including
10% ($\approx$504) users for $N=3$. This procedure was repeated 10 times with
different random seeds. We report average results, together with 95%
t-distribution confidence intervals representing variation between runs.

###  B.2 Implementation details

  * •

Hardware: All experiments were run on a single laptop, with 16GB of RAM, M1
Pro processor, and with no GPU support.

  * •

Runtime: A single run consisting the entire pipeline (data loading and
partitioning, collaborative filtering, training classifiers, simulating
dynamics, learning policies, measuring and comparing performance) takes
roughly 23 minutes. The main bottleneck is the discrete LV simulation, taking
roughly 70% of runtime to compute, mostly due to bookkeeping necessary for the
non-stationary baselines. Simulation code was optimized using the numba jit
compiler, which improves runtime.

  * •

Optimization packages:

    * –

Collaborative filtering (CF): We use the surprise package (Hug,, 2020), which
includes an implementation of the SVD algorithm for CF. All parameters were
set to default values.

    * –

Regression: We use the scikit-learn implementation of linear regression for
predicting long-term engagement from user features (i.e the prediction models
$f_{j}(u)$ in Eq. (8)). All parameters were set to default values.

    * –

Non-Negative Least Squares (NNLS): We use the scipy.optimize implementation of
NNLS. The algorithm was used with its default parameters.

  * •

Code: Code for reproducing all of our figures and experiments is available in
the following repository: https://github.com/edensaig/take-a-break.

###  B.3 Other baselines

  * •

Safety: In each step of the TPP simulation, look $k$ step back, and calculate
the empirical rate $\tilde{\lambda}_{i}=\frac{k}{t_{i}-t_{i-k}}$. If this rate
exceeds the threshold $\tilde{\lambda}_{i}>\tau$, the policy enters a ‘cool-
down’ policy state, serving only forced breaks until the next time period. In
our experiments, we used thresholds $\tau\in{\left\\{{14,16}\right\\}}$,
$k=10$ look-behind steps, and defined the cool-down period as $0.5$ time
units.

  * •

Oracle: To estimate the effect of perfect predictions, we implement an oracle
predictor $f^{\mathrm{oracle}}_{p}(u)$ which has access to the latent user
parameters. For a given $u$ and for each $p$, the predictor outputs the
infinite-horizon LV equilibrium for $u$, namely
$f^{\mathrm{oracle}}_{p}(u)=\lambda^{*}(p;\tilde{\theta}_{u})$. We define
$\tilde{\theta}_{u}=(\alpha_{u},\tilde{\beta}_{u},\gamma_{u},\delta_{u})$,
where $\alpha_{u},\gamma_{u},\delta_{u}$ are the unobserved parameters for the
given user, and $\tilde{\beta}_{u}$ is the expected value of $\beta_{ux}$
induced by the distribution over recommended items $x$ induced by the
recommendation policy $\psi$. We view $\tilde{\theta}_{u}$ as a useful proxy
for the otherwise unattainable $\bar{\theta}_{u}$.

###  B.4 Hyperparameters

  * •

Collaborative filtering: We used $d=8$ latent factors and enabled bias terms,
which ensured performance is close to the benchmark of $\mathrm{RMSE}=0.873$
reported in the surprise documentation. We used the vanilla SVD solver, with
all hyperparameters set to their default values.

  * •

Recommendation policy: Softmax temperature was set to 0.5.

  * •

Prediction: We trained regressors $f(u)$ on input feature vectors consisting
of three components:

$u=(\tilde{v}_{u},b_{u},\hat{\rho}_{u})\in\mathbb{R}^{d+2}$ (26)

The three components are: (i) SVD latent user factors
$\tilde{v}_{u}\in\mathbb{R}^{d}$, (ii) SVD user bias term
$b_{u}\in\mathbb{R}$, (iii) an additional feature consisting of the average
predicted ratings for unseen items $\hat{r}_{u}$ weighted by recommendation
probability, which we found to slightly improve predictive performance:

$\hat{\rho}_{u}=\sum_{x\in\mathrm{holdout}(u)}\hat{r}_{ux}\cdot\mathrm{softmax}_{x}(\hat{r}_{u})$
(27)

Where $\mathrm{holdout}(u)$ is the set of unseen items corresponding to user
$u$, $\hat{r}_{ux}\in[1,5]$ are the predicted ratings, and
$\hat{r}_{u}\in\left[1,5\right]^{\left|{\mathrm{holdout}(u)}\right|}$ is the
vector of all predicted ratings used for softmax recommendation as described
in section 5. We chose to focus on linear models since the treatment datasets
are relatively small (each $|{\cal{S}}^{(j)}|\approx 500$), and since other
model classes (including boosted trees and MLPs) did not perform significantly
better.

  * •

Discrete TPP: Interaction sequences for each user were generated according to
an LV discretization scheme $\mathrm{TPP}^{\mathrm{LV}}(u;T)$, described in
detail in the next section. Latent sates were initialized randomly with
relative uniform noise around the theoretical LV equilibrium point
$(\lambda_{0},q_{0})=((1+\xi_{\lambda})\lambda^{*},(1+\xi_{q})q^{*})$, where
$\xi_{\lambda},\xi_{q}\sim\mathrm{Uniform}(-0.1,0.1)$. Latent states were
updated each $B=10$ recommendations to stabilize noise (see Figure 5). When
$x$ is recommended to $u$ at time $t$, latent states and $\Delta t$ are set
according to $\beta_{u}(t)$, which depends on ratings $r_{ux}$ (true or mixed
with predictions $u^{\top}x$ via $\kappa$). Specifically, we use
$\beta_{u}(t)=r_{ux}^{2}/5\in\\{0.2,0.8,1.8,3.2,5\\}$, which is convex, to
accentuate the role of low ratings since they are underrepresented in the
data. For $B\geq 1$, we take the effective $\beta_{u}(t)$ to be the average
over the $B$ items recommended in that step. We set $\alpha=1.3$, and chose
$\gamma=0.2,\delta=0.01$ (which together determine scale) so that typical
values for engagement rate $\frac{1}{T}|S_{u}|$ are on the order of $\approx
10$ for the chosen $T=100$.

Figure 5:  Example discrete sequence
$S_{u}\sim\mathrm{TPP}^{\mathrm{LV}}(u;T)$, compared to continuous LV
dynamics. Our TPP produces discrete sequences that are qualitatively different
from their continuous-time analogs (blue lines), Nonetheless, it captures the
general properties of our proposed behavioral model: note how cumulative
averaging behavior (orange dashes) exhibits ‘habit formation’, which our
equilibrium approach targets (blue dots). For the same initial conditions
$\lambda(0),q(0)$, the figure shows how varying the number of recommended
items per step ($B$) ‘smooths’ the discrete behavior (left: $B=1$, center:
$B=10$). For fixed $\beta_{u}(t)=\beta_{u}$, when $B\rightarrow\infty$, and
when $\Delta t\rightarrow 0$, TPP sequences approach a continuous LV
trajectory; in general, and particularly when $\beta_{u}(t)$ varies by step
and per recommended items—this is not the case.

###  B.5 Discrete TPP for Lokta-Volterra simulation

The Temporal Point Process (TPP) we use for simulating user interaction
sequences $S_{u}$ is based on a discretization of the LV system described in
Eq. (2), using the forward Euler method with variable step sizes. We denote
this process by $\mathrm{TPP}^{\mathrm{LV}}(u;T)$, and present the sampling
procedure in Algorithm 1.

Each user is associated with discrete latent states $\lambda_{i},q_{i}$, and
parameters $\alpha_{u},\gamma_{u},\delta_{u}$. Initial states
$\lambda_{0},q_{0}$ are set randomly. At each step, and in time $t_{i}$, the
system recommends $x_{i}=x(t_{i})$, which triggers updates in latent states,
and determines the next time of interaction $t_{i+1}$. As noted, these update
depend on item-specific parameters $\beta_{u,x_{i}}$.

Under stationary policy $\pi(p)$, the system recommends an item with
probability $(1-p)$, and suggests a break with probability $p$. The simulator
considers $B$ recommendation opportunities at each step. For each
$k\in{\left\\{{1,\dots,B}\right\\}}$, denote by
$I_{k}\in{\left\\{{0,1}\right\\}}$ the break indicator, equal to $0$ when a
break is recommended at the $k$-th slot in the batch. Denote by $x\sim\psi$
the item recommended by the underlying policy $\psi$, and by
$\beta(x)=r_{ux}^{2}/5$ the corresponding LV hyperparameter as defined above.

Input: Break probability $p\in[0,1]$
Stationary content recommendation probability $\psi$
Lotka-Volterra parameters $\theta_{u}=\left(\alpha,\beta,\gamma,\delta\right)$
Time horizon $T>0$

Output: Interaction sequence
$S_{u}\sim\mathrm{TPP}^{\mathrm{LV}}_{\pi(p)\circ\psi}(u;T)$

Initialize $i=0$, $t_{0}=0$, $S_{u}={\left\\{{}\right\\}}$;

while  _$t_{i} <T$_ do

foreach  _$k\in{\left\\{{1,\dots,B}\right\\}}$_ do

$I_{k}\sim\mathrm{Bernoulli}(1-p)$;

$x_{k}\sim\psi$;

$\beta_{k}\longleftarrow\beta(r_{ux_{k}})$;

end foreach

$\Delta t_{i}\longleftarrow\lambda_{i}^{-1}$ ;

$\lambda_{i+1}\longleftarrow\lambda_{i}+\left(-\alpha+\tfrac{\sum_{k=1}^{B}I_{k}\beta_{k}}{B}q_{i}\right)\lambda_{i}\Delta
t_{i}$ ;

$q_{i+1}\longleftarrow
q_{i}+\left(\gamma(1-q_{i})-\tfrac{\sum_{k=1}^{B}I_{k}\delta}{B}\lambda_{i}\right)q_{i}\Delta
t_{i}$ ;

$t_{i+1}\longleftarrow t_{i}+\lambda_{i+1}^{-1}$ ;

$S_{u}\longleftarrow
S_{u}\cup{\left\\{{\left(t_{i},(x_{1},\dots,x_{B}),(I_{1},\dots,I_{B})\right)}\right\\}}$
;

$i\longleftarrow i+1$ ;

end while

Algorithm 1 Sample from $\mathrm{TPP}^{\mathrm{LV}}(u;T)$

##  Appendix C Additional empirical evaluation

In this section, we provide additional empirical results as further indicators
of potential applicability:

  * •

In subsection C.1, we repeat the experiments presented in section 5 using a
different dataset, showing that qualitative results remain consistent across
recommendation modalities.

  * •

In subsection C.2, we demonstrate how to leverage additional problem structure
commonly available in practice, and develop adaptive breaking policies which
improve over time.

  * •

Finally, in subsection C.3 we present a stateless consumption model which is
unrelated to Lotka-Volterra and does not promote breaks. Evaluation of the LV
policy optimization method under this model shows that the optimization
procedure does not degrade performance when breaks are not beneficial, thus
extending Corollary 2 beyond the realizable case.

###  C.1 Additional dataset

#### The Goodreads dataset.

The Goodreads book recommendations data set is a common benchmark dataset used
in recommendation systems research (Wan and McAuley,, 2018; Wan et al.,,
2019). Ratings are in the range $\\{1,\dots,5\\}$, and the dataset is filtered
to only include users with at least 20 reported ratings. We use the official
comic-books genre subset of the dataset, which includes 3,679,076 ratings
provided by 41,932 users and for 87,565 books after pre-processing. The
dataset is publicly available at:
https://sites.google.com/eng.ucsd.edu/ucsdbookgraph/. Pre-processing code is
available in our code repository: https://github.com/edensaig/take-a-break.

#### Evaluation.

For evaluation, we follow the same procedure described in Appendix B,
maintaining identical data processing procedures, constants, and
hyperparameters. This procedure was repeated 10 times with different random
seeds. We report average results, together with 95% t-distribution confidence
intervals.

#### Results.

Figure 6 shows results for the Goodreads experiment, in the same format as
Figure 3. Results exhibit performance and trends that are qualitatively
similar to the MovieLens experiment in section 5. The LV policy optimization
method achieved better performance on this dataset (right pane): The
performance of the LV is closer to oracle (-0.526% in Goodreads compared to
-0.918% in ML1M), and the gap from the best-of method is slightly larger
(+5.77% in Goodreads compared to +2.19% in ML1M). Varying user types (center
pane) shows less variation across values of $\kappa$, indicating that
predictors achieve satisfactory performance even when the $\kappa$ is low and
prediction is harder (see subsection 5.1). Varying treatments (right pane)
also coincides with the observations: LV policy optimization on the Goodreads
dataset exhibits less performance degradation as $p_{1}\to 0$, suggesting that
less data may be sufficient for optimization. We attribute these results to
the larger size of the dataset (approximately 3.6M interactions, compared to
1M interactions in ML1M), and to the possibility that a stronger structure may
exist in this recommendation modality compared to general movie
recommendation.

Figure 6:  Results on the Goodreads comic-books dataset (compare to Figure 3).
(Left) Performance gain of different approaches (relative to default policy).
(Center) Performance of LV by user group, partitioned by learned policies
$\hat{p}_{u}$. (Right) Sensitivity to an increasingly aggressive experimental
$p_{1}$ ($N=2,p_{0}=0$).

###  C.2 Leveraging additional structure using adaptive policies

One benefit of our approach in section 4 is that it requires only predictions
of long-term engagement. In reality, however, other sources of information may
also be available to the learner, which may be useful for improving decisions
regarding when to recommend breaks. Here we consider additional information in
the form of explicit user feedback, collected by the system over the course of
interaction. We model users as providing to the system feedback regarding item
quality, namely true ratings $r_{ux}$, for some of the recommended items $x$
they consume. For simplicity, we assume users report ratings with probability
$p_{r}\in[0,1]$, independently for each item, where we vary $p_{r}$ across
experimental conditions. Intuitively, we would expect that, over time, user-
reported ratings help improve our learned breaking policy; however, over-
relying on few data and at and early stage my hinder performance. Here we
examine when and how such feedback should be utilized.

#### Adaptive policy optimization.

To extend our approach to incorporate ratings-based feedback, we propose an
adaptive explore-then-exploit approach which can leverage sparse explicit
feedback. Fixing an _adaptation time_ $T_{0}\in[0,T]$ (a hyperparameter), the
method applies the standard learned breaking policy $\hat{\pi}$ (Equation 9)
in the time frame $t\leq T_{0}$, during which it also collects and stores
user-reported ratings. Then, at time $t=T_{0}$, the method updates the
learning policy on the basis of the new ratings data, and applies this policy
until the eventual time $T$. In particular, ratings are used to improve the
estimated component $\hat{\rho}_{u}$ in the user feature vector $u^{\prime}$
(described in Equation 27). In this way, the method uses the existing
engagement predictors $f_{p}(u)$ without requiring any retraining.

A formal description of this method is provided in Algorithm 2. Here we denote
the set of ratings collected until time $t$ by
$r_{[0,t]}\in{\left\\{{1,\dots,5}\right\\}}^{*}$; the average rating at time
$t$ by $\bar{\rho}_{(0,t)}=\tfrac{1}{{\left|{r_{[0,t]}}\right|}}\sum_{r\in
r_{[0,t]}}r$; and the set of long-term engagement predictions for a feature
vector $u$ by
${\bm{f}}(u)={\left\\{{\left(p_{i},f_{p_{i}}(u)\right)}\right\\}}$ (as in
subsection 4.1).

Input: Initial break probability $p\in[0,1]$
Time horizon $T>0$
Adaptation time $T_{0}\in[0,T]$
User feature vector $u=(\tilde{v},b,\hat{\rho})$

1\. Collect ratings data $r_{[0,T_{0}]}$ with breaking policy $\pi(p)$ until
time $T_{0}$ ;

2\. Construct updated feature vector
$u^{\prime}=(\tilde{v},b,\bar{\rho}_{[0,T_{0}]})$ using the average rating
$\bar{\rho}_{[0,T_{0}]}$ ;

3\. Compute updated long-term engagement predictions ${\bm{f}}(u^{\prime})$ ;

4\. Use the LV policy optimization method to obtain updated policy
$p^{\prime}=p^{*}\left({\bm{f}}(u^{\prime})\right)$ ;

5\. Use breaking policy $\pi(p^{\prime})$ for the remaining time
$t\in[T_{0},T]$ ;

Algorithm 2 Adaptive policy optimization using sparse rating signals

#### Evaluation.

We evaluate the adaptive policies using the MovieLens-1M experimental setup,
as described in section 5 and appendix B. We make use of the same linear
regression predictors $f_{p}(u)$ from the main part of the experiment, and
maintain the same time horizon $T=100{}$. We vary the ratings density
parameter $p_{r}$ in the range $[0,1]$, and vary the adaptation time $T_{0}$
across three distinct values $T_{0}\in{\left\\{{0.5,5,50}\right\\}}$.

Figure 7:  Adaptive policy evaluation for varying rating density
$p_{r}\in[0,1]$. Solid blue line represents the adaptive policy described in
Algorithm 2, with adaptation time $T_{0}=5$. Dashed lines represent the
performance of adaptive policies under different choices of $T_{0}$. Cyan
horizontal line represents LV-true-ratings, a non-adaptive method with oracle
access to the true average rating. The remaining horizontal lines represent
the performance of selected non-adaptive policies, as presented in Figure 3
(left pane).

#### Results.

Results are presented in Figure 7. In the results, we observe that LV-adaptive
with $T_{0}=5$ represents an optimal point within the hyperparameter space.
For this choice, results show significant gains over the non-adaptive LV
method even under moderate rating densities, and quick convergence towards the
LV-true-ratings upper bound. In contrast, LV-adaptive with $T_{0}=0.5$
represents a “premature optimization” scenario, attempting to adapt the policy
without allowing the estimated average $\bar{\rho}_{[0,T_{0}]}$ enough time to
converge to its expected value. Similarly, $T_{0}=50$ represents the result of
“late optimization”, which has accurate estimation of $\bar{\rho}_{[0,T_{0}]}$
but not enough time to benefit from the updated policy.

###  C.3 Distinct behavioral model

In this work, we propose the LV model as a behavioral hypothesis class for
counterfactual prediction of long-term engagement. As such, using the LV model
within the learning-to-break optimization framework is a design choice, to be
made at the discretion of the learner; our LV model would be a good choice if
it fits the data better than alternative model classes, given the amount of
available data. This relation is made precise in our error bound (Theorem 1),
which bounds the error when using the LV model for _any_ underlying
$\mathrm{TPP}$ (i.e., we make no assumptions about the true underlying data
generation process). Our main experiments evaluate our approach on data that
is not LV dynamics, but nonetheless, bear some resemblance. To complement
these results, here we run an additional experiment in which we empirically
evaluate our LV-based approach on data that is generated by a behavioral model
that is entirely distinct. In particular, we consider a user model in which
consumption decisions only depend on the quality of recommended items, without
any dependence on internal states—i.e., it is _stateless_. This conforms to
behavioral models which are implicitly assumed in conventional recommendation
methods such as collaborative filtering.

#### Stateless content consumption.

The data generation process is formalized in Algorithm 3. As a means of
capturing stateless behavior, we define a “close-range” temporal point process
$\mathrm{TPP}^{\mathrm{CR}}(u;T)$ which generates sequences of user
interactions based solely on the average rating of recommended items in a
batch. Here we relate rating with utility, and assume that items having higher
utility (and therefore higher ratings) induce more frequent interactions; in
the same way, we consider break prompts as items having zero utility, and
hence zero rating. At time $t_{i}$, and given a batch of size $B$ with $k\leq
B$ recommended items and $B-k\geq 0$ breaks prompts, users acting according to
the stateless behavioral will consume the next batch of content at time:

$t_{i+1}=t_{i}+\frac{1}{\tau}\left(\frac{1}{B}\sum_{j=1}^{k}r_{u{x_{j}}}\right)^{-1}$
(28)

where
$\left(r_{u{x_{1}}},\dots,r_{u{x_{k}}}\right)\in{\left\\{{1,\dots,5}\right\\}}^{k}$
are the ratings of the items recommended at time $k$, and $\tau>0$ is a
constant latent parameter to be learned from data. The breaking policy $\pi$
decides on the number of items $k$ to recommend on each step. Since
$r_{ux}\geq 1$ for all user-item pairs, the time difference $\Delta
t_{i}=t_{i+1}-t_{i}$ in Equation 28 is minimized by taking $k=B$. This shows
that the optimal breaking policy under this behavioral model is the default
one, which does not prompt the user to break.

Input: Break probability $p\in[0,1]$
Stationary content recommendation probability $\psi$
Scalar parameter $\theta_{u}=\tau>0$
Time horizon $T>0$

Output: Interaction sequence
$S_{u}\sim\mathrm{TPP}^{\mathrm{CR}}_{\pi(p)\circ\psi}(u;T)$

Initialize $i=0$, $t_{0}=0$, $S_{u}={\left\\{{}\right\\}}$;

while  _$t_{i} <T$_ do

foreach  _$k\in{\left\\{{1,\dots,B}\right\\}}$_ do

$I_{k}\sim\mathrm{Bernoulli}(1-p)$;

$x_{k}\sim\psi$;

end foreach

$t_{i+1}\longleftarrow
t_{i}+\frac{1}{\tau}\left(\frac{1}{B}\sum_{k=1}^{B}I_{k}r_{ux_{k}}\right)^{-1}$
;

$S_{u}\longleftarrow
S_{u}\cup{\left\\{{\left(t_{i},(x_{1},\dots,x_{B}),(I_{1},\dots,I_{B})\right)}\right\\}}$
;

$i\longleftarrow i+1$ ;

end while

Algorithm 3 Sample from $\mathrm{TPP}^{\mathrm{CR}}(u;T)$

#### Evaluation.

We evaluate the stateless behavioral model using the MovieLens-1M experimental
setup, as described in section 5. We set $\tau=4$ for all users, utilize
linear regression for engagement prediction, and maintain all hyperparameters
without change. Data processing steps are performed as described in appendix
B, and the chosen breaking policies are evaluated.

#### Results.

Despite the difference between the true $\mathrm{TPP}$ and our choice of model
class, the LV policy optimization method successfully learned the optimal no-
breaks—which in this case, is the policy $p=0$ for all users. Further detail
is provided by Figure 8, which illustrates the policy optimization steps under
$\mathrm{TPP}^{\mathrm{CR}}(u;T)$ for typical users with unbiased and biased
engagement predictions. In both examples, the optimal points of both curves
coincide, and the optimal policy is selected despite poor point-wise fit.
Combined, these results show that our approach is “safe”, in the sense that
when breaking is sub-optimal, the learned breaking policy does not override
the default policy.

Figure 8:  Fitting an LV equilibrium curve on the empirical consumption rates
of $\mathrm{TPP}^{\mathrm{CR}}(u;T)$. The plots are a realization of the
schematic diagram in Figure 2 using data from the MovieLens-1M experiment.
(Left) Prediction with unbiased engagement predictions. (Right) User with
extremely low engagement, for which the system tends to over-predict. In both
cases, the LV policy optimization method selects the optimal policy $p=0$.