# Personalization for Web-based Services
using Offline Reinforcement Learning

Pavlos Athanasios Apostolopoulos<EMAIL_ADDRESS>The University of New
Mexico, ECE  Albuquerque, NM  and  Zehui Wang, Hanson Wang, Chad Zhou,
Kittipat Virochsiri, Norm Zhou, Igor L. Markov
<EMAIL_ADDRESS>Facebook Inc.,
Menlo Park, CA

(2021)

###### Abstract.

Large-scale Web-based services present opportunities for improving UI policies
based on observed user interactions. We address challenges of learning such
policies through model-free offline Reinforcement Learning (RL) with off-
policy training. Deployed in a production system for user authentication in a
major social network, it significantly improves long-term objectives. We
articulate practical challenges, compare several ML techniques, provide
insights on training and evaluation of RL models, and discuss generalizations.

††copyright: acmcopyright††journalyear: 2021††doi:
10.1145/1122445.1122456††copyright: none

##  1\. Introduction

For Web-based services with numerous customers, such as social networks, UI
decisions impact top-line metrics, such as user engagement, costs and
revenues. Machine learning, especially Supervised Learning, can optimize these
decisions but long-term cumulative objectives makes it challenging to label
each decision for training. As an illustration, consider online user
authentication. When a user mistypes or forgets her password, the service can
loop back to the login prompt or offer another login channel, authorization
code via SMS (Short Message Service), etc. The user’s context is a part of a
personalized configuration. The scale of the social network makes monetary
cost (service fees for authentication) significant. Some users get their
password right on the second try or look it up but others give up, and this is
reflected in daily/monthly user engagement. Deciding when to authenticate
requires real-time personalization with consideration for cost and engagement
metrics, but the impact on end metrics is delayed. Our work addresses these
challenges with ML, describes insights on model selection and training, and
reports a production deployment. We also explain how our general approach and
our infrastructure generalize to multiple challenges beyond the didactic
application.

Prior efforts on personalized configuration systems covers a wide range of
content-serving applications, including recommendation systems (Garcin et al.,
2013; Garcin and Faltings, 2013), ad targeting (Toğuç and Kuzu, 2020; Agarwal
et al., 2014), and personalized medical healthcare (Ho et al., 2019). Recent
advances in Machine Learning brought us expressive deep neural networks with
effective generalization, as well as infrastructure for building real-time
systems powered by ML. A common theme is to provide a personalized
configuration and optimize desired end metrics given user’s real-time context.
Supervised Learning methods have shown promising results (Wang et al., 2018;
Hidasi et al., 2016), but they only focus on optimizing immediate metrics such
as the click-through rate (Li et al., 2017) and conversion rate (Pradel et
al., 2011). This limitation is important for long-term user engagement
(daily/monthly active users and monetary cost), as shown by the authentication
example.

Reinforcement Learning (RL) (Sutton and Barto, 2018) seeks an optimal policy
to maximize a long-term reward, and thus helps drive real-time personalized
decisions (Tang et al., 2019). Here, the RL agent’s environment is an
individual online user, and the personalized serving procedure is modeled via
sequential agent-environment interactions. Therefore, the RL agent alternates
between policy improvement and interaction with the environment. In practice,
RL agents need many interactions to learn good policies (Dulac-Arnold et al.,
2019). To this end, training RL agents online may undermine user experience
and/or incur large costs. Hence, policies are trained offline using logged
interactions from any type of prior policies (Offline RL in Section 3).

In this work, we use Offline RL to improve personalized authentication for a
Web-based service. We formalize the problem as a Markov decision process (MDP)
(Puterman, 2014), where the RL agent learns a personalized policy, i.e., when
to send an authentication message to the user after a failed login attempt.
The training process optimizes long-term user engagement and the service’s
authentication costs. We avoid the pitfalls of online trial-and-error by
training on prior experiences logged for different policies through off-policy
learning. To ensure effective offline training, we use several training
heuristics, then evaluate performance of the learned candidate policies via an
unbiased off-policy estimator. The best learned policy is chosen and evaluated
in online A/B tests (Kohavi and Longbotham, 2017) on live data.

The remaining part of the paper is organized as follows. We review
Reinforcement Learning in Section 2 focusing on off-policy learning. Section 3
discusses the need for offline learning and the challenges of deploying such
applications. A representative application amenable to reinforcement learning
is covered in Section 4, and problem formalization in Section 5. Our
contributions are introduced in Section 6 where we describe an industry
application of Offline RL to user authentication. Section 7 details how our
Offline RL model was deployed to a major social network and compares it to a
supervised-learning baseline. Section 8 puts our work in perspective and
discusses personalization more broadly.

##  2\. Background

In this section, we review the background and establish notation that will be
used later. Section 2.1 reviews Reinforcement Learning theory (Sutton and
Barto, 2018) and Section 2.2 focuses on off-policy learning.

###  2.1. Preliminaries on Reinforcement Learning

Reinforcement learning seeks to control an interactive dynamic system. At each
discrete time step, $t=0,1,\cdots$, the agent observes the environment’s state
$s_{t}$ and responds with action $a_{t}$, while the environment responds with
an associated reward $r_{t+1}$ and transitions into the next state $s_{t+1}$.
To this end, the environment can be defined by a Markov Decision Process (MDP)
$(\mathcal{S},\mathcal{A},r,p,\gamma)$ (Puterman, 2014), where $\mathcal{S}$
is the state space with $s_{t}\in\mathcal{S}$, $\mathcal{A}$ is the action
space with $a_{t}\in\mathcal{A}$, $r_{a}^{s}$ is the reward function,
$p(\cdot|s,a)$ is a conditional probability distribution of the form
$p(s_{t+1}|s_{t},a_{t})$ (environment’s dynamics), and $\gamma\in(0,1]$ is a
scalar discount factor.

A Reinforcement Learning agent seeks a policy $\pi^{*}(a_{t}|s_{t})$ that for
each state defines a distribution over possible actions. The policy should
maximize cumulative reward over time, i.e., an expectation under the
environment’s dynamics:

(1)
$J(\pi)=\operatorname{\mathbb{E}}\sum_{t=0}^{\infty}{\gamma^{t}r_{a_{t}}^{s_{t}}}$
$s_{0}=s,a_{0}=a,s_{t}\sim p(\cdot|s_{t-1},a_{t-1}),a_{t}\sim\pi(\cdot|s_{t})$

Based on Equation 1, the Reinforcement Learning problem can be viewed as an
optimization over the space of policies $\pi$:

(2)
$\begin{split}\pi^{*}=\operatorname*{arg\,max}_{\pi}\>\operatorname{\mathbb{E}}\sum_{t=0}^{\infty}{\gamma^{t}r_{a_{t}}^{s_{t}}}\end{split}$

For policies $\pi_{\theta}$ smoothly parameterized by weights $\theta$,
Equation 1 is amenable to gradient descent in terms of
$\nabla_{\theta}J(\pi_{\theta})$, i.e., policy gradients (Sutton et al., 1999;
Schulman et al., 2015). Functional dependency of $\pi$ on $s_{t},a_{t}$ makes
it easier to work with stochastic policies.

Additionally, the Reinforcement Learning objective in Equation 1 can be
optimized by accurately estimating a state-action value function (Sutton and
Barto, 2018), and then using that value function to derive an optimal policy
$\pi^{*}$. The state-action value function of policy $\pi$ is

(3)
$Q^{\pi}(s_{t},a_{t})=\displaystyle\operatorname{\mathbb{E}}_{\begin{subarray}{c}s_{t}\sim
p(s_{t}|s_{t-1},a_{t-1})\\\
a_{t}\sim\pi(a_{t}|s_{t})\end{subarray}}\left[\sum_{t=0}^{\infty}{\gamma^{t}r_{a_{t}}^{s_{t}}}\right],$

and we can express $Q^{\pi}(s_{t},a_{t})$ as follows:

(4)
$Q^{\pi}(s_{t},a_{t})=r_{a_{t}}^{s_{t}}+\gamma\operatorname{\mathbb{E}}_{\begin{subarray}{c}s_{t+1}\sim
p(s_{t+1}|s_{t},a_{t})\\\
a_{t+1}\sim\pi(a_{t+1}|s_{t+1})\end{subarray}}\left[Q^{\pi}(s_{t+1},a_{t+1})\right]$

A policy can be defined for a state-action value function by

(5)
$\pi(a_{t}|s_{t})=\delta(a_{t}=\operatorname*{arg\,max}_{a_{t}\in\mathcal{A}}\>Q(s_{t},a_{t})),$

and by substituting Equation 5 into Equation 4, we obtain the Bellman
optimality equations (Bellman, 1957) that characterize optimal policies in
terms of optimal Q-functions $Q^{*}=Q^{\pi^{*}}$:

(6)
$Q^{*}(s_{t},a_{t})=r_{a_{t}}^{s_{t}}+\gamma\operatorname{\mathbb{E}}_{s_{t+1}\sim
p(s_{t+1}|s_{t},a_{t})}\left[\underset{a_{t+1}}{\max}\>Q^{*}(s_{t+1,a_{t+1}})\right]$

Therefore, Reinforcement Learning (Equation 2) seeks an optimal policy
$\pi^{*}$ such that $Q^{\pi^{*}}(s_{t},a_{t})\geq Q^{\pi}(s_{t},a_{t})$,
$\forall\pi,s_{t}\in\mathcal{S},a_{t}\in\mathcal{A}$.

###  2.2. Off-Policy Learning

In the discussion below, we focus on policies defined by state-action value
functions per Equation 5, and $\varepsilon$-greedy policies (Sutton and Barto,
2018) that perform random exploration with a small probability. This focus
simplifies the narrative, and we found such policies successful in practice.
Sections 5 and 7 also discuss parameterized policies that draw actions from
state-conditional probability distributions and are trained to maximize
cumulative rewards.

Striving for an optimal Q-function $Q^{*}$ and therefore an optimal policy
$\pi^{*}=\delta(a_{t}=\operatorname*{arg\,max}\>Q^{*}(s_{t},a_{t})),$
Q-learning (Watkins and Dayan, 1992) alternates between two phases. First, it
improves an approximate estimate $Q(s_{t},a_{t};\theta)$ of $Q^{*}$ by
repeatedly regressing Equation 6 with respect to parameters $\theta$. In off-
policy Q-learning, an action with the best Q-value is considered Bellman-
optimal per Equation 6 and used in gradient calculations. In the second phase,
the agent explores the environment, typically following a stochastic policy
based on $Q(s_{t},a_{t};\theta)$, e.g., an $\epsilon$-greedy (Sutton and
Barto, 2018) version of
$\pi(s_{t})=\delta(a_{t}=\operatorname*{arg\,max}\>Q(s_{t},a_{t};\theta))$.
Q-learning takes a single gradient step per training iteration to minimize the
difference between the left-hand and right-hand side of Equation 6. Among the
many variants of this learning procedure, the most common variant called DQN
(Mnih et al., 2013) utilizes a replay buffer
$\mathcal{D}=\\{(s^{i}_{t},a^{i}_{t},s^{i}_{t+1},r^{i}_{t+1})\\}$ for storing
the agent’s interaction with the environment and alternates between data
collection and gradient steps with respect to the Temporal Difference loss
function $\mathcal{L}_{i}(\theta_{i})$, defined as follows via Equation 6:

(7) $\mathcal{L}_{i}(\theta_{i})=\operatorname{\mathbb{E}}_{s_{t},a_{t}\sim
D}\left[(y_{i}-Q(s_{t},a_{t};\theta_{i}))^{2}\right]$

Here $y_{i}=\operatorname{\mathbb{E}}_{s_{t+1}\sim
D}\left[r(s_{t},a_{t})+\gamma\underset{a}{\max}\>Q(s_{t+1},a;\theta^{-}_{i}))\right]$,
$\theta_{i}$ are the parameters of the approximate Q-function at the iteration
$i$ of the learning procedure, and for better stability in the learning
procedure the $\theta_{i}^{-}$ are frozen parameters that are used for
estimating the target values, i.e., $y_{i}$. The $\theta^{-}$ parameters are
periodically updated to the current parameters $\theta$ of the parametric
Q-function. Variants of DQN decouple the selection from action evaluation
(Double DQN (van Hasselt et al., 2015)), or approximate the Q-function through
dueling network architectures (Dueling DQN (Wang et al., 2016)) to address the
overestimation of Q-values in the DQN setting.

Q-learning and its aforementioned deep reinforcement learning variants are
characterized as off-policy algorithms (Sutton and Barto, 2018), since the
learning target values $y_{i}$ can be computed without any consideration of
the policy that was used to generate the experiences of the replay buffer
i.e., $\mathcal{D}=\\{(s^{i}_{t},a^{i}_{t},s^{i}_{t+1},r^{i}_{t+1})\\}$.

##  3\. Offline Reinforcement Learning

Offline Reinforcement Learning seeks policy $\pi^{*}$ to maximize cumulative
reward (Equation 1), but avoids live interactions with the environment during
training. It is used when such interactions can be harmful (Gauci et al.,
2018). Hence, a static dataset of state-action transitions
$\mathcal{D_{\pi_{\beta}}}=\\{(s^{i}_{t},a^{i}_{t},s^{i}_{t+1},r^{i}_{t+1})\\}$
logged via behavioral policy $\pi_{\beta}$ is used. Web-based services
routinely log each user session, but offline RL based on Equation 7 requires
redundant logs, where each row includes the previous state $s_{t}$ and next
state $s_{t+1}$ (Gauci et al., 2018). Such rows can be used independently and
batch-sampled for training.

Offline RL trains on state-action transitions batch-sampled from training set
$\mathcal{D_{\pi_{\beta}}}$ through off-policy learning (Section 2.2).
Learning relies on RL mechanisms such as optimal state-action values (e.g.,
Equation 7), but generalization results from supervised learning should apply.
After the learned policy $\pi^{*}$ is evaluated offline and tuned, online
evaluation uses live interaction with the environment.

Off-policy Reinforcement Learning algorithms (DQN, Double DQN, and Dueling
DQN) that estimate the state-action value function $Q(s_{t},a_{t};\theta)$ can
be directly used offline. However, the online regime has the benefit of
additional data collection via environment interaction (exploration) that
helps the agent refine its $Q(s_{t},a_{t};\theta)$ estimates for high-reward
actions. Although off-policy learning has shown promising results in offline
RL (Kalashnikov et al., 2018; Agarwal et al., 2020), the lack of exploration
limits agent’s learning (Kumar et al., 2019) due to distributional shift
phenomena.

State distributional shift affects offline RL algorithms during the agent’s
exposure on the real-world environment (deployment phase) (Kumar et al.,
2019). The latter occurs as the agent’s learned policy $\pi^{*}$ may follow a
systematically different state visitation frequency compared to the one of the
training set, i.e., $\mathcal{D}_{\pi_{\beta}}$, that is induced by the
behavior policy $\pi_{\beta}$. In other words, since the agent’s goal is to
find the best policy $\pi^{*}$ offline by utilizing the static dataset
$\mathcal{D}_{\pi_{\beta}}$, its learned policy can diverge from the behavior
policy $\pi_{\beta}$, invoking unreasonable actions in out-of-
distribution/unseen states.

Action distributional shift affects off-policy learning algorithms, which are
estimating the Q-function, i.e., $Q(s_{t},a_{t};\theta)$, during the training
process as well (Wu et al., 2019). In principle, the accuracy of the
regression in Equation 7 depends on the estimate of the Q-function for actions
that may be outside of the distribution of actions that the Q-function was
ever trained on, i.e., right-hand side of Eq. 6 . The action distribution
shift is exacerbated by the differences between the agent’s learned policy and
the behavioral policy $\pi_{\beta}$ during training. If the agent’s
parameterized Q-function produces large, erroneous values for out-of-
distribution actions, it will further learn to do so.

Standard RL methods address the distributional shift via ongoing exploration
by revising the state-action value function $Q(s_{t},a_{t};\theta)$. However,
offline RL lacks such a feedback loop, and inaccuracies in the state-action
value function accumulate during learning.

Prior research (Schulman et al., 2015; Siegel et al., 2020) focuses on
mitigating distributional shift in offline RL by limiting how much the learned
optimal policy $\pi^{*}$ may deviate from the behavior policy $\pi_{\beta}$.
This forces the agent’s learned policy to stay close to the behavior policy
and reduces state distributional shift. Additionally, the latter reduces
action distributional shift during offline training as well, as most of the
states and actions fed into the right-hand side of Equation 7 are in-
distribution with respect to the training set. In that case, the action
distributional shift errors should not accumulate. As another example, the
work in (Jaques et al., 2019) regularizes the agent’s learned policy towards
the behavioral policy by using the Kullback-Leibler (KL) divergence with a
fixed regularization weight, while the Maximum Mean Discrepancy with an
adaptively trained regularization weight is used in (Kumar et al., 2019).
Moreover, the work in (Wu et al., 2019; Jaques et al., 2019) defines target
values ($y_{i}$) to regularize the agent’s learned policy towards the
behavioral policy and avoid actions inconsistent with the behavioral policy.

To guide our design decisions for the application in Section 4, in Section 5.1
we introduce a simplified, intuitive problem environment. In Section 6.1, we
use this environment to illustrate important insights on distributional shift
in Offline RL and the impact of behavioral policy’s exploration on the quality
of trained models.

##  4\. Application: User Authentication

We illustrate personalization in Web-based services with a self-contained
application that is representative of Web-based software and widespread. It
optimizes long-term rewards and favors reinforcement learning. We formalize
the problem in Section 5.

User authentication starts a typical Web-based session; thus failures can
limit user engagement. Due to forgotten passwords, such failures are common
when different passwords are used for each Web-based service. The
authentication UI may respond to failures by offering password recovery
(Action A) or by prompting another log-in attempt (Action B), see Figure 1.
Password recovery verifies the user’s identity by sending a code to a pre-
registered mobile phone or another user-owned device. Common options include

  * •

One-Time Password (OTP) sent via SMS and email

  * •

Time-based One-Time Password (TOTP) authentication (Ledesma, 2020).

Such authentication is considered safe unless the user’s device is compromised
or the phone number is hijacked (Sharma and Nene, 2020).111Such authentication
can also support two-factor authentication (2FA) (Awasthi, 2015; Rockwell,
2016). In our application, we authenticate via OTP.

Figure 1. User authentication UI.

The decision mechanism that choses between available actions may affect long-
term objectives, for example user engagement captured by how many service
users are active per day or per month. To this end, Action A may typically
succeed and improve user engagement but incurs OTP costs. Action B may succeed
without such costs, and if it fails again, Action A can still be invoked. A
nascent Web-based service may strive to grow user engagement at greater cost,
but mature services may lean towards lower cost.

Training the decision mechanism on user activity assumes trends or patterns,
as well as sufficient features to learn them. When different user cohorts
develop different trends, separating them simplifies the learning task. For
example, new users may exhibit different behaviors than long-term users or
have less historical data to use for inferences. This paper focuses on users
with prior login history and additional user features.

Additional aspects of the problem are worth noting. Compared to multiplayer
games, the lack of typical-case adversarial behavior simplifies successful
policies and makes them easier to learn from logged data without simulated
interactions. On the other hand, training a policy from scratch on user
interactions in a live production system may adversely affect user engagement
until the policy is optimized. Compared to the (contextual) bandit setup, this
problem exhibits sequential depth.

Extensions of this problem include state-dependent and larger action spaces,
and substantial generalizations covered in Section 8.

##  5\. Problem formalization

The application introduced in Section 4 deals with sequences of actions, e.g.,
Action B can be tried several times before falling back on Action A. The
sequential nature of this application calls for models based on states and
state transitions. First, we introduce a simplified, intuitive example used
later in the paper and then outline our full-fledged state model.

###  5.1. A motivational example

Consider the user-authentication application with two possible actions at a
time, introduced in Section 4. It is easy to check whether Action A (password
recovery) is more beneficial than Action B (another log-in attempt) on
average, marginalizing all other information. This generalizes to short
sequences, e.g., which action is preferable after one Action B? After two of
them? (marginalizing everything else). By capturing such sequences up to some
depth, one can optimize per-action rewards using either supervised or
reinforcement learning. Figure 2 illustrates possible system actions from the
initial failed-login state and subsequent states, as well as implied state
transitions. The feedback to system actions is captured by rewards only (which
include OTP costs as per Equation 8), and successful logins bring higher
rewards than failures. State transitions depend entirely on the actions chosen
by the system. In particular, after a successful login, the sequence can
continue upon a future login failure. This self-contained state-reward model
can capture the most basic sequential preferences but lacks user
personalization, which is key to our work.

Figure 2.  State transitions during user authentication. All states are
distinct and indexed with depth (subscripts) and the history of actions
(encoded in binary in superscripts).

A user can be characterized by a number of features that cover, e.g., time
since account creation and local time, the amount of online activity, etc. In
other words, each user at each step is represented by a multidimensional
vector such that proximal vectors represent similar users. Compared to the
earlier few-state model that represents an average user, personalization
requires states that cannot be limited to a small predefined set. We now
interpret Figure 2 as showing possible transitions and traces for one user,
wherein each user state $s_{t}$ in each trace is represented by a finite-
dimensional feature vector (user state vector).

The semantics of features are not important for our motivational example.
Therefore, for each user we represent the initial state $s_{0}$ (the first
failed log-in attempt) with a state vector by drawing it at random from a
personalized multivariate Gaussian distribution with mean and covariance
selected randomly per user. Subsequent user states $s_{t}$ with $t=1,\dots$
that correspond to future failed login attempts are generated randomly and as
part of state transitions. Specifically, a base (user) feature vector is drawn
from a multivariate Gaussian that is defined per user, per discrete time step
$t$, and per action $a_{t}\in\\{A,B\\}$ that was prompted by the
authentication UI on user state $s_{t-1}$. This base vector and the vector
representing user state $s_{t-1}$ are then aggregated to construct the vector
for the next user state $s_{t}$. The user engagement and corresponding action
costs on each user transition $(s_{t},a_{t},s_{t+1})$ are represented by a
single normalized scalar value-reward $r_{t+1}$. The latter is drawn from a
normal distribution with randomly selected mean and variance, defined per
user, per time step, per action.

The simulated environment we introduced for system actions during user
authentication is used as a testbed for evaluating ML techniques in Section 6.
Under our formulation, the trained decision mechanism that chooses an action
for each transition aims to optimize and/or balance long-term objectives,
i.e., cumulative rewards over a time horizon (Section 2.1).

###  5.2. Our practical state model and rewards

The problem formalization we use in practice differs from the motivational
example in two aspects: (1) user state vectors are based on actual user
features rather than drawn at random, (2) rewards are crafted to approximate
long-term objectives.

User features for state representation include real-time (contextual) and
historical features. On-device real-time features capture the app platform,
local time of day, day of the week, login time, etc. Historical features focus
on past events, such as the total number of user password recoveries sent in
the past, time since last login, etc. Historical features may also aggregate
user data specific to the authentication process, e.g., the number of
authentication attempts in the last 30 days and their outcomes, which of
Action A and Action B were used, and how the user reacted.

Rewards are formulated to help optimize long-term objectives that are affected
by the authentication decision mechanism. Specifically, we aim to optimize
user engagement and monetary cost. In our Web-based service, user engagement
is reported and expressed as a binary value that determines if the user was
active during the day. Monetary cost represents the amount of money spent on
password recovery (Action A) after failed login attempts. Ideally, we would
like to avoid harming user engagement and not block users from logging in,
while reducing charges for Action A. To this end, we formulate the reward as a
linear combination of user engagement (UE) and monetary cost (Cost), thus

(8)
$\vspace{-1mm}r=w_{u}\cdot\mathrm{UE}-w_{c}\cdot\mathrm{Cost},~{}\mathrm{where}~{}w_{u},w_{c}\in\mathbb{R}$

Varying the weights helps explore the Pareto curve of multiobjective tradeoffs
relevant to the application. We assign rewards to the last action on each day
(Figure 3) because system metrics for user engagement are tallied at the end
of each day. During RL training, rewards are propagated back via the Bellman
equation.

Figure 3.  Rewards are set daily per user, to the last action.  Figure 4.
Exploration diversity improves state coverage. To visualize distributions of
visited states in the feature space, we use two principal components via PCA.
P$(A)=0.5$ exhibits the greatest state coverage (blue), whereas P$(A)=0.9$
exhibits limited coverage (red). The blue pointset is cloned.

##  6\. Applying Reinforcement Learning

The problem formalization in Section 5 facilitates established ML techniques,
including supervised and reinforcement learning. In this section we discuss
the pros and cons of competing ML approaches, introduce our proposed RL model,
then explain how training data is extracted and augmented.

###  6.1. Competing ML approaches

For a given state vector in the feature space, an ML model can ($i$) predict
rewards for each action via Supervised Learning or ($ii$) estimate Q-values
via offline Reinforcement Learning (Section 3). Generally, Reinforcement
Learning offers two important advantages:

  * •

RL exploration policies are linked to exploitation and optimized to improve
state coverage.

  * •

Long-term cumulative objectives, e.g., user engagement, can be optimized by RL
even when they are not faithfully represented by immediate rewards for each
action (Figure 3).

The impact of the exploration level of behavioral policies in offline RL is
illustrated in Figure 4 that is produced for the motivational example from
Section 5.1. Here we compare (behavioral) exploration policies defined by the
complementary probabilities of Actions A and B. Specifically, using the
induced state transitions and corresponding rewards as a static dataset, we
train offline an RL model for each behavioral policy (Section 3) through an
established off-policy algorithm (DQN). Intuitively, balanced exploration with
equal probabilities should provide better state coverage. Indeed, this is
observed in our simulation. In the figure, we project state vectors from the
feature space onto two principal components (via PCA). The yellow (70:30) and
red (90:10) point clouds are narrower than the blue (50:50) point cloud.
Furthermore, Figure 5 shows that exploration diversity in the behavioral
policy helps train a better model. In other words, during the evaluation
phase, we can expect higher perceived returns (cumulative rewards) for the RL
model trained offline with balanced exploration. Such evaluation is performed
on users not seen in the training dataset and by using the trained RL model as
a decision mechanism that drives the state transitions. The impact of
behavioral policies’ exploration is related to the distributional shift
phenomemon discussed in Section 3. The higher the state coverage in the
training dataset, the less significant the state distributional shift during
evaluation and action distributional shift during training for an Offline RL
model.

Modeling forward sequential depth (i.e., the future impact of current actions)
is difficult for supervised learning. Modeling backward sequential depth (the
impact of past actions) is somewhat easier, using additional features that
summarize past states and actions. Furthermore, reinforcement learning often
suffers from high variance in rewards and statistical bias in explored states.

Figure 5. The impact of exploration diversity on learning returns. Between
Actions A and B, the former is chosen with probability P$(A)$. The plot shows
average cumulative rewards over trajectories of different length. The best
results are observed for balanced exploration P$(A)=0.5$.

Despite obvious limitations, supervised learning (SL) remains a viable
competitor to RL because

  * •

SL often facilitates more mature optimization methods

  * •

SL can train higher-quality models faster due to smaller variances and greater
learning rates.

Therefore, to empirically evaluate our proposed RL method, Section 7 compares
it to a production system trained with supervised learning. Among different RL
techniques, model-based RL (tree search, etc), runs into difficulties
predicting state transitions based on actions and corresponding rewards
(environment dynamics).

###  6.2. RL decision models

Given the large state space in our application, we model the Q-function by a
neural net. The values of the trained $Q(s_{t},a_{t};\theta)$ express
accumulated rewards (Equation 8) for Actions A, B based on user features for a
given state. We perform training offline on a static dataset of state
transitions using the DQN algorithm. As a result, the task of learning a near-
optimal Q-function is transformed into a familiar regression problem for
Equation 7. A learned near-optimal Q-function supports a decision mechanism
that selects Action A or B based on current user features to maximize
accumulated rewards. The use of Temporal Difference (Equation 7) as the loss
function is a key distinction from supervised learning. The implied use of the
Bellman equation is what allows this approach to track long-term rewards.

Our implementation (Section 7) makes it easy to try more sophisticated RL
models and algorithms with the same state model and rewards (Section 5.2). In
fact, we have tried several improvements to DQN (such as DDQN, Dueling DQN)
and concluded that DQN does not leave much room for improvement despite being
simpler. However, more techniques that parameterize both the policy and the
Q-function are generally worth trying. Whether or not this improves long-term
objectives is unclear a priori, and we therefore implement a recent technique
in this category.

Critic-Regularized Regression (CRR) (Wang et al., 2020a) is an off-policy
method that discourages sampling low-quality actions from the training
dataset. As it is typical for actor-critic algorithms, CRR parameterizes both
the policy and the Q-function (Konda and Tsitsiklis, 2000). It additionally
transforms Q-values in the objective (Equation 1) with a monotonically
increasing function, such as $\exp(\cdot)$, to emphasize higher Q-values. This
way, during policy update, $\pi$ more often samples high-quality actions
within the training distribution. The critic is trained using distributional
Q-learning (Wang et al., 2020a; Barth-Maron et al., 2018). The distributional
representation of the returns translates well into stochastic behavioral
policies.

###  6.3. Preparing data for training

Each row of our training data includes user features, actual system actions,
and rewards.

Data extraction. Motivated by the impact of the exploration level of
behavioral policies in Offline RL (Section 6.1) we choose equal probabilities
for Actions A and B. Thus, upon a failed login attempt, an action is chosen at
random. Another handler starts extracting and computing user features
simultaneously. The two asynchronous events are tagged with the same unique ID
and joined by a stream processing system for logging in the training table.
The user engagement metric and the sum of OTP fees are computed by the end of
the day. These rewards are joined with the training table based on an
anonymized user ID column, where rewards are attributed to the last login
failure event of the day (see Figure 3). We logged data using the described
exploratory behavioral policy for a time period of three weeks.

Data augmentation. Following the standard Markov Decision Process (MDP)
framework, RL models are trained on consecutive pairs of state/action tuples
that correspond to state transitions in user sequences (Figure 2). We use the
open-source applied Reinforcement Learning platform ReAgent (Horizon) (Gauci
et al., 2018) to transform logged state-action-reward data to the following
row format:

  * •

MDP ID: a unique id, e.g., anonymized user ID, for the Markov Decision Process
chain.

  * •

Sequence Number: a number representing the depth of the state in the MDP,
e.g., the timestamp of the state.

  * •

State Features: the user features for state representation of the current
step.

  * •

Action: the actual system action.

  * •

Action Probability: the probability that the current system took the action
logged.

  * •

Metrics: user engagement metric and the sum of OTP fees.

  * •

Possible Actions: an array of possible actions at the current step.

  * •

Next State Features: the user features for state representation of the
subsequent step.

  * •

Next Action: the actual system action at the next step.

  * •

Sequence Number Ordinal: a number representing the depth of the state in the
MDP after converting the Sequence Number to an ordinal number.

  * •

Time Diff: a number representing the time difference between the current and
next state.

  * •

Possible Next Actions: an array of actions that were possible at the next
step.

The metrics map enables reward shaping for Equation 8 by tuning $w_{u},w_{c}$
during training. Offline RL training is enabled by preprocessing logic for the
row format above.

##  7\. Empirical evaluation

To evaluate ML techniques from Section 6, our workflow first assesses quality
of RL models trained offline. Hyperparameter tuning is performed based on
offline metrics. For the best seen RL model, online evaluation on live data
helps us compare to a baseline production solution that uses SL. Our
evaluation methodology and especially the metrics are applicable beyond this
work.

###  7.1. Offline training and evaluation for RL

To support decision models from Section 6.2, we train our DQN model using the
ReAgent (Horizon) platform (Gauci et al., 2018) and its default neural network
architecture. We set the discount factor to $\gamma=1.0$ because most MDP
sequences are short (80% of them have a single step). Equation 8 is used for
the reward function.

Offline RL training minimizes Temporal Difference loss (Equation 7) over the
provided static dataset of transitions (Section 6.3). Figure 6 shows how loss
values for the DQN RL model (Section 6.2) change over training epochs that
consist of mini-batches (iterations). To avoid overfitting, we split the
static dataset of transitions into a training and test set, then compare the
average Q-values of the same Actions A, B between the training and test sets,
and keep the differences below $10\%$.

Figure 6. Temporal Difference loss (MSE) with respect to learning epochs for
the DQN RL model (Section 6.2).

Offline evaluation avoids the dangers of poorly trained policies in
production. Additionally, it uses a static dataset for consistent
experimentation and performance tuning. To watch out for deficient training
and distributional shifts (Section 3), our evaluation methodology pays
particular attention to training quality metrics.

  * •

The Temporal Difference Ratio addresses a pitfall in using raw Temporal
Difference (TD) loss (Equation 7), which is sensitive to the magnitude of
Q-values. A poor RL model will exhibit low TD values when Q-values are low.
Instead, we evaluate the metric
$\frac{\mathrm{TD}}{\min\\{\bar{Q}_{A},\bar{Q}_{B}\\}}$, where
$\bar{Q}_{A},\bar{Q}_{B}$ are the average Q-values of Actions A, B over the
training dataset.

  * •

Distributional stability during training is vital to effective learning. The
action distribution of the trained offline RL model should not shift too far
from the behavioral policy, which we measure as the KL-divergence (Jaques et
al., 2019) between the RL model’s learned policy and the behavioral policy
function. The action distribution of our DQN model in Figure 7 stabilizes over
training. The change of the action distribution over a time window ($25$
training iterations) is a good proxy for the distributional stability for
offline RL models.

Figure 7. Changes in action distributions for Actions A, B over learning
epochs for the DQN RL model (Section 6.2).

Counterfactual Policy Evaluation (CPE) with respect to perceived accumulated
reward is the second tier of our evaluation methodology. CPE is performed
offline (because our RL model is trained offline) and off-policy because it
uses a static dataset of transitions $\mathcal{D_{\pi_{\beta}}}$, logged by
the behavioral policy $\pi_{\beta}$. The key idea is to answer counterfactual
questions of the form: “how probable would it be for the offline policy to
encounter/perceive same state transitions/rewards as the logged ones?”. To
this end, CPE provides a safe way of ranking different offline RL models
before they are tested in production. Off-policy evaluation can be based on
ordinary importance sampling (Precup et al., 2000), but at risk of high
variance which grows exponentially with the number of state transitions. More
sophisticated methods (Thomas and Brunskill, 2016) provide controlled bias-
variance tradeoffs by utilizing additional information, e.g., the learned
Q-function. In our application, relatively short MDP sequences make it
practical to use the sequential doubly robust estimator (Jiang and Li, 2016)
to produce unbiased performance estimates for policies trained offline.

Hyperparameter optimization for Offline RL is challenging (Paine et al.,
2020), as there is no access to the environment. Hyperparameters can be
optimized toward the metrics in our two-tier offline evaluation above, after
which the best seen models can be evaluated online. For example, parameters
for training, such as the learning rate, can be optimized via the first tier,
whereas the second tier can compare different RL algorithms. Our DQN model
(Figures 6 and 7) exhibited the best results in offline evaluation, was
deployed online and evaluated vs. SL baseline production system (Section 7.2).

###  7.2. Empirical comparisons

Our baseline using supervised learning is a prior production model with a
tree-based meta-learner that estimates the Conditional Average Treatment
Effects (Künzel et al., 2019). The treatment effects are represented by a
linear combination of the rewards defined in Equation 8. The meta-learner
decomposes the decision process into two steps by first estimating the
conditional reward expectations, $U=\operatorname{\mathbb{E}}[\mathrm{UE}\mid
a_{t},s=s_{t}]$, $C=\operatorname{\mathbb{E}}[\mathrm{Cost}\mid
a_{t},s=s_{t}]$, where $a_{t}\in A$. $U$ and $C$ are computed using Gradient
Boosted Decision Trees whose parameters, including learning rate, tree depth
and tree leaves, are swept in a large range to minimize mean squared errors.
Then the learner takes the differences between the estimates and chooses the
action that gives the highest reward:

(9) $a_{t}=\operatorname*{arg\,max}_{a_{t}\in
A}\operatorname{\mathbb{E}}[r_{t}|a_{t},s=s_{t}]$

After deriving $r_{t}$ using Equation 8, the final action is given by

(10) $a_{t}=\operatorname*{arg\,max}_{a_{t}\in A}(w_{u}\cdot U-w_{c}\cdot C)$

The reward weights $w_{u}$ and $w_{c}$ are chosen through online experiments
to ensure compelling multiobjective tradeoffs.

Table 1.  SL and RL policies are compared online to fixed policies that always perform Action A or Action B. RL provides the most attractive tradeoffs, shown in bold.  | User engagement | OTP cost
---|---|---
Policy | Daily | Monthly |
B | - | - | -
A | +1.49% $\pm$ 0.273% | +2.81% $\pm$ 0.279% | +120% $\pm$ 4.16%
SL | +1.35% $\pm$ 0.212% | +2.58% $\pm$ 0.116% | +81.1% $\pm$ 3.13%
RL | +1.50% $\pm$ 0.213% | +2.55% $\pm$ 0.183% | +70.3% $\pm$ 2.96%
Table 2.  Action distributions (% of Action A) for behavioral policies trained with SL and RL by user cohort. | SL | RL
---|---|---
All users | 55.88% | 22.75%
Single-login users | 60.16% | 36.41%
Multiple-login users | 49.40% | 11.19%

Online evaluation is performed using one-month’s data. Table 1 demonstrates
the first round of results at a 95% confidence level, where our RL model is
trained on randomized data as per Section 6.3. The RL model is compared to
fixed policies (the same action repeated always) and a prior production model
based on SL. The RL model significantly reduces OTP costs while exhibiting
competitive daily and monthly user engagement results. We estimate Return On
Investment (ROI) by dividing total cost by total monthly engagement. RL
outperforms SL by $5.93\%\pm 0.782\%$. To collect data for recurrent training,
we deploy our RL agent by extending the deterministic policy to an
$\epsilon$-greedy policy with $\epsilon=0.1$.

When ML is applied to practical problems, it commonly optimizes surrogate
objectives, and one has to empirically check that practical objectives are
improved as a result. Moreover, sophisticated applications often track
multiple performance metrics whose regressions may block new optimization. In
our case, one such metric counts notification disavow events (NDEs) for
password reset, where a user turns off notifications, perhaps because there
were too many. Comparing to SL, we observe that RL reduces NDEs by 50%. To
this end, Table 2 shows that the RL agent trades off Action A for Action B and
thus reduces authentication messages, while maintaining neutral engagement
metrics. We also group users into cohorts — single-login users and multiple-
login users (within the evaluation period). Intuitively, multiple-login users
are more likely to enter their password correctly without help from our
authentication messages. From Table 2, we see that both RL and SL make use of
this user attribute but RL exploits it more efficiently

Figure 8.  Daily user engagement of a recurrent CRR model

Recurrent training and stability evaluation. When user behaviors shift over
time, ML models must be refreshed using the most recent data (recurrent
training). The initial RL model trained on randomized data shows sizable
improvements. However, continued collection of randomized data is risky in
terms of direct costs and user experience. The ”Rand” policy (Section 6.1) in
Table 3 suggests that its deployment limits the benefits of RL models. Thus,
we refresh the model using behavioral data collected from our RL agent.
Additionally, to check if recurrent training stabilizes in the long term
compared with the initial RL model, we perform a second round of online
evaluation to study recurrent training using RL behavioral data and model
stability in the long term.

We train a DQN and a CRR model (Section 6.2) with the same parameters and
rewards as the initial RL model, but collect their training data via policies
from the deployed RL agent. Results summarized in Table 3 use the same
settings as before (a 95% confidence level, one-month’s testing window, and an
$\epsilon$-greedy policy on DQN and CRR with 10% exploration rate). Both
recurrent DQN and CRR give neutral results compared with the initial RL model,
which indicates stable performance in recurrent training. Figure 8
additionally shows daily user engagement for the CRR recurrent model vs. the
initial RL model, and this also shows consistent performance.

Table 3.  Recurrent training results compared with the initial RL agent trained on randomized data (the ”Rand” policy issues Action A and Action B with equal probability).  | User engagement | OTP cost
---|---|---
Policy | Daily | Monthly |
DQN | +0.0023% $\pm$0.106% | +0.0576% $\pm$0.0841% | -0.166% $\pm$0.439%
CRR | +0.116% $\pm$0.240% | +0.107% $\pm$0.140% | +1.27% $\pm$2.87%
Rand | -0.336% $\pm$0.266% | -0.693% $\pm$0.162% | +3.78% $\pm$3.29%

##  8\. Conclusions

In this work we show how to apply reinforcement learning (RL) to personalize
user authentication in a Web-based system and compare RL to a competing
approach based on supervised learning. Working with industry data, using
offline RL avoids releasing poorly trained agents in production. However, this
approach requires careful extraction and augmentation of training data to
ensure that off-policy learning does not succumb to distributional shift. In
practice, RL is often susceptible to high variance and high bias at several of
its stages, especially when operating on large-scale live data. Fortunately,
our method is sufficiently robust for production deployment using the ReAgent
(Horizon) platform (Gauci et al., 2018). Starting from state modeling and data
collection, we articulate obstacles and milestones toward model training and
demonstrate practical improvement in end-to-end system-level metrics at a
large-scale deployment at Facebook. During development, we use a simplified
problem environment to test intuition without sensitive data.

Broader applications. Our self-contained didactic application is not only
critical to many Web-based systems, but also generalizes well (e.g., to more
than two actions at a time) and illustrates how other aspects of Web-based
systems can be personalized and enhanced via ML. Relevant optimizations
explored previously include

  * •

personalized individual user notifications (Gauci et al., 2018),

  * •

page prefetching to optimize user experience (Wang et al., 2020b).

User notifications and product delivery must balance utility with distraction,
while page prefetches improve access latency at the cost of increased network
bandwidth. These applications bear structural similarity to our work and share
several salient aspects.

  1. (1)

At each step, the system chooses from several actions. Future decisions are
improved based on user feedback.

  2. (2)

One must optimize and/or balance long-term cumulative objectives, some of
which do not reduce to rewards for individual actions that can be handled by
supervised learning.

  3. (3)

Personalization is based on a number of user features and can be supported by
ML models in the feature space.

More general RL methods. Our straightforward RL model (DQN) is facilitated by
the small action space, which ($i$) is less demanding in terms of learning
robustness, and ($ii$) allows for simpler neural-net representations where
separate outputs produce $Q(s,a)$ values for different actions $a$. However,
the overall approach generalizes to more sophisticated settings and can be
extrapolated to other personalized enhancements for Web-based systems as
follows.

  * •

We use model-free RL without predicting future states and action rewards, but
such predictions can be leveraged.

  * •

Our extraction and handling of user features, as well as offline model-free
training and the evaluation methodology are not tied to specific RL models.

  * •

Our use of the open-source ReAgent (Horizon) platform (Gauci et al., 2018)
makes it easy to employ models such as Double DQN (van Hasselt et al., 2015)
(DDQN) and Dueling DQN (Wang et al., 2016) that could provide a more stable
learning for larger action spaces.

For very large or continuous action spaces, $Q(s,a)$ can be modeled as a
function of both $s$ and $a$ (Lillicrap et al., 2019). Continuous action
spaces can be addressed using policy gradient methods (Sutton et al., 1999),
available in ReAgent.

## References

  * (1)
  * Agarwal et al. (2014) Deepak Agarwal, Bo Long, Jonathan Traupman, Doris Xin, and Liang Zhang. 2014.  LASER: A Scalable Response Prediction Platform for Online Advertising. In _Proc. 7th ACM International Conference on Web Search and Data Mining_ (New York, NY) _(WSDM ’14)_. ACM, New York, NY, 173–182.  https://doi.org/10.1145/2556195.2556252
  * Agarwal et al. (2020) Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. 2020\.  An Optimistic Perspective on Offline Reinforcement Learning. In _Proc. 37th International Conference on Machine Learning_ _(Proc. Machine Learning Research, Vol. 119)_ , Hal Daumé III and Aarti Singh (Eds.). PMLR, Virtual, 104–114.  http://proceedings.mlr.press/v119/agarwal20c.html
  * Awasthi (2015) Akshay Awasthi. 2015\.  Reducing Identity Theft Using One-Time Passwords and SMS.  _EDPACS_ 52, 5 (2015), 9–19.
  * Barth-Maron et al. (2018) Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, TB Dhruva, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. 2018\.  Distributed Distributional Deterministic Policy Gradients.
  * Bellman (1957) Richard Bellman. 1957\.  _Dynamic Programming_ (1 ed.).  Princeton University Press, Princeton, NJ, USA.
  * Dulac-Arnold et al. (2019) Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. 2019\.  Challenges of Real-World Reinforcement Learning.  arXiv:1904.12901 [cs.LG]
  * Garcin et al. (2013) Florent Garcin, Christos Dimitrakakis, and Boi Faltings. 2013\.  Personalized News Recommendation with Context Trees. In _Proc. 7th ACM Conference on Recommender Systems_ (Hong Kong, China) _(RecSys ’13)_. ACM, New York, NY, 105–112.  https://doi.org/10.1145/2507157.2507166
  * Garcin and Faltings (2013) Florent Garcin and Boi Faltings. 2013.  PEN Recsys: A Personalized News Recommender Systems Framework. In _Proc. 2013 International News Recommender Systems Workshop and Challenge_ (Kowloon, Hong Kong) _(NRS ’13)_. ACM, New York, NY, 3–9.  https://doi.org/10.1145/2516641.2516642
  * Gauci et al. (2018) Jason Gauci, Edoardo Conti, Yitao Liang, Kittipat Virochsiri, Yuchen He, Zachary Kaden, Vivek Narayanan, and Xiaohui Ye. 2018\.  Horizon: Facebook’s Open Source Applied Reinforcement Learning Platform.  _CoRR_ abs/1811.00260 (2018), 10pp.  arXiv:1811.00260 http://arxiv.org/abs/1811.00260
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based Recommendations with Recurrent Neural Networks.  arXiv:1511.06939 [cs.LG]
  * Ho et al. (2019) Daniel Sik Wai Ho, William Schierding, Melissa Wake, Richard Saffery, and Justin O’Sullivan. 2019\.  Machine learning SNP based prediction for precision medicine.  _Frontiers in Genetics_ 10 (2019), 267.
  * Jaques et al. (2019) Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019\.  Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog.  arXiv:1907.00456 [cs.LG]
  * Jiang and Li (2016) Nan Jiang and Lihong Li. 2016.  Doubly Robust Off-policy Value Evaluation for Reinforcement Learning. In _Proceedings of The 33rd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, NYA, 652–661.  http://proceedings.mlr.press/v48/jiang16.html
  * Kalashnikov et al. (2018) Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine. 2018.  Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation. In _Proc. 2nd Conference on Robot Learning_ _(Proc. Machine Learning Research, Vol. 87)_ , Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto (Eds.). PMLR, Virtual, 651–673.  http://proceedings.mlr.press/v87/kalashnikov18a.html
  * Kohavi and Longbotham (2017) Ron Kohavi and Roger Longbotham. 2017.  Online Controlled Experiments and A/B Testing.  _Encyclopedia of Machine Learning and Data Mining_ 7, 8 (2017), 922–929.
  * Konda and Tsitsiklis (2000) Vijay Konda and John Tsitsiklis. 2000.  Actor-Critic Algorithms. In _Advances in Neural Information Processing Systems_ , S. Solla, T. Leen, and K. Müller (Eds.), Vol. 12. MIT Press, Denver, CO, 1008–1014.  https://proceedings.neurips.cc/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf
  * Kumar et al. (2019) Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. 2019.  Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction. In _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., Virtual, 11784–11794.  https://proceedings.neurips.cc/paper/2019/file/c2073ffa77b5357a498057413bb09d3a-Paper.pdf
  * Künzel et al. (2019) Sören R Künzel, Jasjeet S Sekhon, Peter J Bickel, and Bin Yu. 2019.  Metalearners for estimating heterogeneous treatment effects using machine learning.  _Proceedings of the National Academy of Sciences_ 116, 10 (2019), 4156–4165.
  * Ledesma (2020) Ricardo Margarito Ledesma. 2020\.  Systems and methods for one-time password authentication.  US Patent App. 16/918,742.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural Attentive Session-Based Recommendation. In _Proc. 2017 ACM on Conference on Information and Knowledge Management_ (Singapore, Singapore) _(CIKM ’17)_. ACM, New York, NY, 1419–1428.  https://doi.org/10.1145/3132847.3132926
  * Lillicrap et al. (2019) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2019.  Continuous control with deep reinforcement learning.  arXiv:1509.02971 [cs.LG]
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing Atari with Deep Reinforcement Learning.  arXiv:1312.5602 [cs.LG]
  * Paine et al. (2020) Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. 2020\.  Hyperparameter Selection for Offline Reinforcement Learning.  arXiv:2007.09055 [cs.LG]
  * Pradel et al. (2011) Bruno Pradel, Savaneary Sean, Julien Delporte, Sébastien Guérif, Céline Rouveirol, Nicolas Usunier, Françoise Fogelman-Soulié, and Frédéric Dufau-Joel. 2011\.  A Case Study in a Recommender System Based on Purchase Data. In _Proc. 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (San Diego, California, USA) _(KDD ’11)_. ACM, New York, NY, 377–385.  https://doi.org/10.1145/2020408.2020470
  * Precup et al. (2000) Doina Precup, Richard S. Sutton, and Satinder P. Singh. 2000\.  Eligibility Traces for Off-Policy Policy Evaluation. In _Proceedings of the Seventeenth International Conference on Machine Learning_ _(ICML ’00)_. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 759–766.
  * Puterman (2014) Martin L Puterman. 2014\.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons, Virtual.
  * Rockwell (2016) Paul Rockwell. 2016\.  Two factor authentication using a one-time password.  US Patent 9,378,356.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015.  Trust Region Policy Optimization. In _Proc. 32nd International Conference on Machine Learning_ _(Proc. Machine Learning Research, Vol. 37)_ , Francis Bach and David Blei (Eds.). PMLR, Lille, France, 1889–1897.  http://proceedings.mlr.press/v37/schulman15.html
  * Sharma and Nene (2020) Mohit Kr Sharma and Manisha J Nene. 2020.  Two-factor authentication using biometric based quantum operations.  _Security and Privacy_ 3, 3 (2020), e102.
  * Siegel et al. (2020) Noah Y. Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. 2020\.  Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning.  arXiv:2002.08396 [cs.LG]
  * Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto. 2018.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA.
  * Sutton et al. (1999) Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999.  Policy Gradient Methods for Reinforcement Learning with Function Approximation. In _Proc. 12th International Conference on Neural Information Processing Systems_ (Denver, CO) _(NIPS’99)_. MIT Press, Cambridge, MA, USA, 1057–1063.
  * Tang et al. (2019) Xueying Tang, Yunxiao Chen, Xiaoou Li, Jingchen Liu, and Zhiliang Ying. 2019.  A reinforcement learning approach to personalized learning recommendation systems.  _Brit. J. Math. Statist. Psych._ 72, 1 (2019), 108–135.
  * Thomas and Brunskill (2016) Philip Thomas and Emma Brunskill. 2016.  Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning. In _Proceedings of The 33rd International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, NY, 2139–2148.  http://proceedings.mlr.press/v48/thomasa16.html
  * Toğuç and Kuzu (2020) H. Toğuç and R. S. Kuzu. 2020.  Hybrid Models of Factorization Machines with Neural Networks and Their Ensembles for Click-through Rate Prediction. In _2020 5th International Conf. on Computer Science and Engineering (UBMK)_. ICEECS, Hangzhou, China, 31–36.  https://doi.org/10.1109/UBMK50275.2020.9219371
  * van Hasselt et al. (2015) Hado van Hasselt, Arthur Guez, and David Silver. 2015\.  Deep Reinforcement Learning with Double Q-learning.  arXiv:1509.06461 [cs.LG]
  * Wang et al. (2020b) Hanson Wang, Zehui Wang, and Yuanyuan Ma. 2020b.  Predictive Precompute with Recurrent Neural Networks. In _Proc. Machine Learning and Systems_ , I. Dhillon, D. Papailiopoulos, and V. Sze (Eds.), Vol. 2. mlsys.org, San Jose, CA, 470–480.  https://proceedings.mlsys.org/paper/2020/file/8f53295a73878494e9bc8dd6c3c7104f-Paper.pdf
  * Wang et al. (2018) Zihan Wang, Ziheng Jiang, Zhaochun Ren, Jiliang Tang, and Dawei Yin. 2018.  A Path-Constrained Framework for Discriminating Substitutable and Complementary Products in E-Commerce. In _Proc. Eleventh ACM International Conference on Web Search and Data Mining_ (Marina Del Rey, CA, USA) _(WSDM ’18)_. ACM, New York, NY, 619–627.  https://doi.org/10.1145/3159652.3159710
  * Wang et al. (2020a) Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. 2020a.  Critic Regularized Regression.  arXiv:2006.15134 [cs.LG]
  * Wang et al. (2016) Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016\.  Dueling Network Architectures for Deep Reinforcement Learning. In _Proc. 33rd International Conference on Machine Learning_ _(Proc. Machine Learning Research, Vol. 48)_ , Maria Florina Balcan and Kilian Q. Weinberger (Eds.). PMLR, New York, NY, 1995–2003.  http://proceedings.mlr.press/v48/wangf16.html
  * Watkins and Dayan (1992) Christopher JCH Watkins and Peter Dayan. 1992.  Q-learning.  _Machine learning_ 8, 3-4 (1992), 279–292.
  * Wu et al. (2019) Yifan Wu, George Tucker, and Ofir Nachum. 2019.  Behavior Regularized Offline Reinforcement Learning.  arXiv:1911.11361 [cs.LG]