# Towards Validating Long-Term User Feedbacks
in Interactive Recommendation Systems

Hojoon Lee<EMAIL_ADDRESS>KAIST ,  Dongyoon Hwang
<EMAIL_ADDRESS>KAIST ,  Kyushik Min<EMAIL_ADDRESS>KAKAO Enterprise and  Jaegul Choo<EMAIL_ADDRESS>KAIST

(2022)

###### Abstract.

Interactive Recommender Systems (IRSs) have attracted a lot of attention, due
to their ability to model interactive processes between users and recommender
systems. Numerous approaches have adopted Reinforcement Learning (RL)
algorithms, as these can directly maximize users’ cumulative rewards. In IRS,
researchers commonly utilize publicly available review datasets to compare and
evaluate algorithms. However, user feedback provided in public datasets merely
includes instant responses (e.g., a rating), with no inclusion of delayed
responses (e.g., the dwell time and the lifetime value). Thus, the question
remains whether these review datasets are an appropriate choice to evaluate
the long-term effects in IRS. In this work, we revisited experiments on IRS
with review datasets and compared RL-based models with a simple reward model
that greedily recommends the item with the highest one-step reward. Following
extensive analysis, we can reveal three main findings: First, a simple greedy
reward model consistently outperforms RL-based models in maximizing cumulative
rewards. Second, applying higher weighting to long-term rewards leads to
degradation of recommendation performance. Third, user feedbacks have mere
long-term effects in the benchmark datasets. Based on our findings, we
conclude that a dataset has to be carefully verified and that a simple greedy
baseline should be included for a proper evaluation of RL-based IRS
approaches.

Interactive Recommender System, Reinforcement Learning

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain††booktitle: Proceedings
of the 45th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††price:
15.00††doi: 10.1145/3477495.3531869††isbn: 978-1-4503-8732-3/22/07††ccs:
Information systems Recommender systems

##  1\. Introduction

Over the past decades, we have witnessed a huge growth in recommender systems
in various e-commerce platforms and social network services (Koren et al.,
2009; Sarwar et al., 2001; Kang and McAuley, 2018; Sun et al., 2019; Covington
et al., 2016). However, conventional recommender systems regard recommendation
as a one-step decision-making task and do not consider future changes in user
preferences. This prohibits the recommender system from further optimizing
users’ long-term satisfaction. As a response, Interactive Recommender Systems
(IRSs) have been proposed to reformulate the recommendation scenario as a
multi-step decision-making process. In contrast to a conventional
recommendation scenario, the goal of an IRS is to provide recommendations that
can maximize a user’s cumulative satisfaction.

Reinforcement learning (RL) algorithms, whose objective is to optimize long-
term rewards, have naturally attracted much interest in the interactive
recommendation scenarios and have been successfully applied in real-world
applications (Chen et al., 2019a; Zhao et al., 2020, 2018b). Not only have
they been able to maximize the immediate user feedbacks (e.g., clicks), they
have also been able to maximize the delayed feedbacks (e.g., lifetime value)
which is not apparent at first, but which can be profitable in the long run.
However, since the experiments are conducted on their private dataset,
researchers commonly rely on public review datasets (e.g., MovieLens) to
compare and develop the algorithms. Unlike the private dataset, public review
datasets only contains instant feedbacks (e.g., movie rating) without any
inclusion of the delayed feedbacks (e.g., lifetime value) from the users.
Therefore, this raises questions around the significance of long-term effects
in the review datasets. If the long-term effect is insignificant, a simple
recommender system that maximizes the one-step reward will be sufficient,
making the dataset inappropriate to benchmark the IRS.

In this work, we first revisited experiments involving RL-based IRSs to verify
the significance of long-term effects on public datasets. First, we compared
RL-based models with a simple recommender system that greedily recommends the
item with the highest one-step reward. We found that this greedy method
outperforms RL models. Second, we ablated through a discount factor to compare
the performance of RL models according to different weights on future rewards.
Here, we observed a gradual degradation in the performance when more weights
were applied to future rewards. Finally, by comparing the approximated optimal
policy and the simple greedy policy, we were able to conclude that user
feedbacks in the public review datasets have insignificant long-term effects.

From these experimental results, we believe that current public review
datasets may not be an appropriate choice to benchmark the RL-based IRS.
Therefore, for a proper evaluation, we suggest to carefully analyze the
dataset and include a simple greedy baseline to verify the effectiveness of an
RL-based IRS.

##  2\. Related Work

Interactive recommender system (IRS) is designed to model the sequential
interaction between a user and the recommender system. Traditionally,
contextual bandits (Li et al., 2010; Qin et al., 2014; Wu et al., 2016; Wang
et al., 2017) are used to learn the empirical utilities from the online
interaction while handling the explore/exploit dilemma of the recommendations.
However, learning directly from the online experience is expensive and may
hurt user experience. Therefore, the focus has been shifted to learning the
recommender system through the user’s logged behavior data.

Recent works on IRS has adopted RL algorithms to identify optimal
recommendation strategies from the logged behavior data. These approaches have
shown great success in real-world e-commerce and social media / networking
platforms such as Alibaba (Zhao et al., 2018b), TikTok (Zhao et al., 2020),
and YouTube (Chen et al., 2019a). However, such work utilizes a private
dataset, making them inaccessible for the research community. As an
alternative, researchers commonly rely on public review datasets to compare
and develop RL-based recommendation algorithms. The RL-based IRS can be mainly
categorized into two groups, policy-based and value-based methods. Policy-
based methods attempt to learn an optimal recommendation policy by applying
policy gradient to a parameterized policy (Liu et al., 2018; Chen et al.,
2019a, b). On the other hand, value-based methods aim to learn the Q-value for
each state and perform the action with the highest Q-value (Zhao et al.,
2018b; Zou et al., 2020). Actor-critic algorithms (Xin et al., 2020; Zhao et
al., 2018a), which integrates policy- and value-based methods have also been
studied. There, the recommendation policy serves as an “actor” and is trained
to maximize the value from the trained “critic”.

##  3\. Problem Formulation

We first illustrate how the recommender system and a user jointly build an
interactive process. As the user enters the service or platform, the
recommender system constructs the user’s profile based on the items they have
interacted with and corresponding feedback. By inferring the user’s latent
interests behind the interaction history, the system provides an item to the
user and receives feedback on the recommended item. The feedback can be either
explicitly provided such as via ratings, or inferred from an implicit reaction
such as views or clicks. After receiving the feedback, the system updates the
user’s profile and continues to recommend the next item. This interaction loop
proceeds until the user leaves the platform. The goal of the recommender
system is to maximize the cumulative ”rewards” during the interaction, which
is the numerical representation of the feedback.

Following previous work (Wang et al., 2020; Zou et al., 2020; Zhao et al.,
2018a; Zhou et al., 2020; Chen et al., 2019b; Zhao et al., 2018b), this
interactive process is formulated as a Markov Decision Process (MDP).
Formally, MDP consists of a tuple of
$(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma)$ as follows:

  * $\bullet$

$\mathcal{S}:$ a continuous state space to describe the user’s profile. A
state $s_{t}^{u}\in\mathcal{S}$ is defined as the previous interaction history
of the user $u$ before time-step $t$. For each time-step $t$, the interaction
consists of the recommended item $i_{t}^{u}$, and its corresponding feedback
$f_{t}^{u}$.

(1)
$s_{t}^{u}=\\{(i_{1}^{u},f_{1}^{u}),(i_{2}^{u},f_{2}^{u}),...,(i_{t-1}^{u},f_{t-1}^{u})\\}$

In this work, we consider the user’s provided rating for the recommended item
$i_{t}^{u}$ as the feedback.

  * $\bullet$

$\mathcal{A}:$ a discrete action space with candidate items to recommend. An
action $a_{t}^{u}\in\mathcal{A}$ denotes the recommended item by the
recommender system at time-step $t$ for the user $u$.

  * $\bullet$

$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward
function where $r(s_{t}^{u},a_{t}^{u})$ denotes the immediate reward from user
$u$ at state $s_{t}^{u}$ by taking action $a_{t}^{u}$. The flexibility of
designing the reward function allows the integration and optimization of
user’s diverse feedbacks. Following from (Zou et al., 2020; Zhou et al.,
2020), we set the provided rating as the reward.

  * $\bullet$

$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$
is a state transition probability.

  * $\bullet$

$\gamma$ is a discount factor that controls the weights for future rewards.

Without loss of generality, we aim to learn the recommendation policy
$\pi:\mathcal{S}\rightarrow\mathcal{A}$ which maximizes the cumulative rewards
as:

(2)
$\pi^{*}=\operatorname*{argmax}_{\pi}\mathbb{E}[\sum_{u=1}^{U}\sum_{t=1}^{T}r(s_{t}^{u},a_{t}^{u})]$

where $T$ denotes the total number of steps of the interaction process and $U$
denotes the number of all users.

##  4\. Revisiting IRS Experiments

In this section, we revisit the experiments of the IRS papers (Zou et al.,
2020; Zhou et al., 2020; Chen et al., 2019b; Zhao et al., 2018a) that utilized
RL algorithms into recommender system. Although these RL-based recommendation
models directly optimize the long-term satisfaction of the users, we show that
a simple greedy recommendation model yields competitive or even better
results.

###  4.1. Experimental Setup

Datasets We evaluate our model using four standard recommendation review
datasets in IRS: EachMovie, Movielens-1M, -20M, and Netflix, where explicit
ratings are available for each interaction. For each dataset, interacted items
are grouped by users and ordered by timestamps. Following (Kang and McAuley,
2018), we only kept users and items that have at least five interactions.
Statistical deatails are outlined in Table 1.

Table 1. Statistics of the processed dataset. The Avg.Int denotes the average number of interacted items per user. Dataset | # Users | # Items | # Int | Avg.Int
---|---|---|---|---
EachMovie | 56,071 | 1,613 | 2,798,088 | 49.90
ML-1M | 6,040 | 3,416 | 999,611 | 165.50
ML-20M | 138,493 | 18,345 | 19,984,024 | 144.30
Netflix | 472,987 | 17,769 | 100,461,928 | 212.40
Table 2. Performance of different models on the interactive recommender
system. The results are averaged over 10 random seeds. Bold scores indicate
the best model for each metric and underlined scores indicate the second best
model.

|  |  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---
Methods | EachMovie | ML-1M | ML-20M | Netflix
RW@40 |  PR@40 |  RC@40 |  RW@40 |  PR@40 |  RC@40 |  RW@40 |  PR@40 |  RC@40 |  RW@40 |  PR@40 |  RC@40
|  |  |  |  |  |  |  |  |  |  |  |
Random |  3.167 |  0.0203 |  0.0246 |  3.241 |  0.0900 |  0.0094 |  2.885 |  0.0450 |  0.0010 |  2.949 |  0.0668 |  0.0011
POP |  4.021 |  0.0723 |  0.1620 |  4.353 |  0.8521 |  0.1415 |  4.002 |  0.5445 |  0.0802 |  3.680 |  0.3757 |  0.0294
SASRec (Kang and McAuley, 2018) |  3.449 |  0.0872 |  0.0677 |  3.879 |  0.5243 |  0.0896 |  3.455 |  0.2235 |  0.0124 |  3.648 |  0.3399 |  0.0195
|  |  |  |  |  |  |  |  |  |  |  |
DQNR (Zhao et al., 2018b) |  5.079 |  0.6420 |  0.6552 |  4.689 |  0.9950 |  0.1597 |  4.483 |  0.9475 |  0.1269 |  4.608 |  0.9530 |  0.1226
NICF (Zou et al., 2020) |  5.075 |  0.6289 |  0.6434 |  4.677 |  0.9947 |  0.1662 |  4.478 |  0.9421 |  0.1262 |  4.655 |  0.9548 |  0.1230
SQN (Xin et al., 2020) |  4.521 |  0.3734 |  0.4468 |  4.134 |  0.6285 |  0.0662 |  3.973 |  0.5048 |  0.0557 |  3.497 |  0.2374 |  0.0105
SAC (Xin et al., 2020) |  4.225 |  0.1104 |  0.2352 |  3.961 |  0.4943 |  0.0728 |  3.845 |  0.4047 |  0.0463 |  3.589 |  0.2601 |  0.0119
DDPGR (Liu et al., 2018) |  4.063 |  0.0681 |  0.0917 |  3.750 |  0.3696 |  0.0683 |  3.673 |  0.2370 |  0.0310 |  3.588 |  0.0730 |  0.01389
|  |  |  |  |  |  |  |  |  |  |  |
GreedyRM |  5.116 |  0.6690 |  0.6578 |  4.680 |  0.9948 |  0.1695 |  4.536 |  0.9712 |  0.1318 |  4.689 |  0.9596 |  0.1244

Models We compared the following representative baselines:

  * $\bullet$

Random: A model that randomly recommends the items.

  * $\bullet$

POP: A model that recommends the most popular item.

  * $\bullet$

SASRec (Kang and McAuley, 2018): A uni-directional Transformer model that is
trained to predict the user’s next interacted item.

  * $\bullet$

DQNR (Zhao et al., 2018b): A DQN (Mnih et al., 2015)-based model that
estimates Q-value for each action and recommends the item with the highest
Q-value.

  * $\bullet$

NICF (Zou et al., 2020): A model similar to DQNR where the target Q-value is
computed within the user’s previous interacted items.

  * $\bullet$

SQN (Xin et al., 2020): A model that jointly optimizes the Q-value and the
probability of interaction for each item.

  * $\bullet$

SAC (Xin et al., 2020): The actor-critic version of SQN where the interaction
probability is weighted by the estimated Q-value.

  * $\bullet$

DDPGR (Zhao et al., 2018a): A DDPG (Lillicrap et al., 2015)-based model that
addresses the large discrete action space by learning a ranking vector.

  * $\bullet$

GreedyRM: Our proposed simple baseline that estimates the one-step reward for
each action and greedily recommends the item with the highest reward. (i.e.,
DQNR with $\gamma=0$).

Architecture For a fair comparison between the learning algorithms, we unified
the network architecture with a uni-directional Transformer as SASRec (Kang
and McAuley, 2018) to encode the states .

Figure 1. Unified architecture for the recommendation models. Each model only
differs in the Predictor Head.

Given a state $s_{t}$, the input embedding is constructed by summing up the
embedded items $[i_{1},\text{…},i_{t}]$, feedbacks $[f_{1},\text{…},f_{t}]$,
and pre-defined sinusoidal positional embedding. Then, the input embedding is
fed into the self-attention block to construct the hidden representation
$[h_{1},\text{…},h_{t}]$. The last hidden representation $h_{t}$ is passed
onto the prediction head with two fully-connected layers and projected onto
the item embedding space. For each item, the projection score indicates the
reward for GreedyRM, Q-value for DQNR, NICF, and un-normalized log probability
in SASRec. SQN, SAC, and DDPGR have two separate prediction heads, one for
policy and another for Q-value.

Training We re-implemented the aforementioned baselines by following the
details of each paper. Following (Kang and McAuley, 2018; Zou et al., 2020),
we set the number of layers as $N=2$, hidden dimension size as $d=64$, maximum
sequence length as $s=200$, and set the discount factor $\gamma$ as $0.95$.
For training, we used an Adam optimizer with $\beta_{1}=0.9$,
$\beta_{2}=0.999$ with gradient clipping when its $l_{2}$ norm exceeds $5$ and
set the batch size to $256$.

For all models, we tune the hyper-parameters based on the performance by
varying the learning rate $lr$ from $\\{0.01,0.001,0.0001\\}$ and $l_{2}$
regularization weight $wd$ from $\\{0.001,0.0001,0.00001\\}$.

###  4.2. Evaluation Protocol

Following (Zou et al., 2020; Zhou et al., 2020; Chen et al., 2019b), the
datasets were split into three disjoint sets: 85% of users for training, 5% of
users for validation, and 10% for testing. In the evaluation, the user profile
was constructed based on their initial $40$ steps of interaction history and
the interactive recommendation process was run through for $40$ time-steps to
measure the recommendation performance.

However, unless one happens to own a service or a platform, obtaining true
feedback from the test user is infeasible and is also prohibitively expensive.
Therefore, a common evaluation protocol is to build an environment simulator
that mimics the user’s behavior and measures the performance within the
learned simulator. Often, matrix factorization is used as the simulator to
predict the user’s behavior (Zhou et al., 2020; Hu et al., 2018; Liu et al.,
2018). However, matrix-factorization is limited in modeling the dynamic
changes of the user’s preferences, hence produce predictions that deviate from
the actual user’s behaviors. Therefore to accurately model the user’s
feedback, we construct the simulator based on the self-attentive architecture
as Figure 1. We report the RMSE value of our simulator for each dataset in
Table 3 to assess the effectiveness of the constructed simulator,

Table 3. RMSE value of the environment simulators.

Methods | EachMovie | ML-1M | ML-20M | Netflix
---|---|---|---|---
Matrix Factorization | 1.402 | 0.990 | 0.979 | 1.073
Transformer | 1.184 | 0.826 | 0.949 | 0.852

Metrics We use three standard evaluation metrics to measure the recommendation
performance during $T$ time-steps of interactions: average reward (RW@T)
(i.e., cumulative rewards divided by $T$), precision (PR@T), and recall
(RC@T). Following (Zou et al., 2020; Zhou et al., 2020), the precision and
recall were computed by setting the positive items as the ground truth labels.
We define an item to be positive if the rating provided from the simulator
exceeded $5$-points in EachMovie and $4$-points in the other datasets.

###  4.3. Experimental Results

Table 2 summarizes the performances of all baseline models for each benchmark
dataset. Here, we observe the followings:

Out of all RL-based models, DQNR and NICF consistently outperformed
traditional recommendation models (i.e., Random, POP, SASRec), which do not
consider the long-term utilities of the users. By observing the poor
performance of SQN and SAC, we found that jointly optimizing the next item
prediction task is not beneficial. In addition, DDPGR also struggled to learn
an effective recommendation policy due to the inaccessibility to online
interactions. Surprisingly, our simple baseline model, GreedyRM, achieved the
best performance for all metrics for the EachMovie, ML-20M, and Netflix
datasets. Though DQNR showed best performance in the ML-1M dataset, GreedyRM
achieved competitive results compared with DQNR. GreedyRM aims to recommend
items that maximize the immediate one-step reward and is analogous to DQNR
when its discount factor $\gamma$ is 0. Therefore, this naturally raises the
question of whether putting more weighting on future rewards harms the
recommendation quality.

###  4.4. Influence of Future Rewards

To further examine the influence of accounting for future rewards, we
conducted an ablation study referring to the discount factor $\gamma$. If the
long-term effects of user feedback are significant, assigning a higher weight
(i.e., large discount factor) will be beneficial. However, if the long-term
effect is insignificant, assigning greater weight on future rewards may serve
as extra noise rather than as a beneficial learning signal. For all
experiments, we varied the discount factor from 0 to 0.99 and fixed the
remaining hyper-parameters to each model’s optimal configurations.

$0$$0.2$$0.4$$0.6$$0.8$$1$$4$$4.5$$5$$\gamma$RW@40EachMovie$0$$0.2$$0.4$$0.6$$0.8$$1$$3.8$$4$$4.2$$4.4$$4.6$$\gamma$RW@40ML-1M$0$$0.2$$0.4$$0.6$$0.8$$1$$3.8$$4$$4.2$$4.4$$4.6$$\gamma$RW@40ML-20M$0$$0.2$$0.4$$0.6$$0.8$$1$$3.5$$4$$4.5$$\gamma$RW@40NetflixGreedyRMDQNRNICFSQNSAC
Figure 2. Performance comparison of RL-based recommendation models with
varying discount factor $\gamma$.

Figure 2 displays the average reward (i.e., RW@40) with respect to each
discount factor. We observe that RL-based models display a gradual decrease in
cumulative rewards as the models put more weight on future rewards. Thus, we
suspect that long-term effects merely exist between the user feedbacks in the
benchmark review datasets. If this were true, a simple greedy one-step
recommendation model will be able to directly maximize the cumulative rewards.
Therefore, we further investigate whether a greedy algorithm can maximize the
cumulative rewards in the review datasets.

###  4.5. Analyzing Long-Term Effects in Datasets

Here, we investigated the significance of the long-term effects in the public
review dataset. By comparing the performance between the greedy- and optimal-
recommendation policies, we were able to verify whether the greedy algorithm
could maximize the cumulative rewards (i.e., no long-term effects in the
dataset). The performance of the optimal policy can be measured by searching
across all actions with the simulator until the last time-step and selecting
the best action sequence that maximizes the cumulative rewards. However, this
measurement is computationally infeasible since it requires a search across
all possible actions for 40 time steps (i.e., $|A|^{40}$).

Therefore, to approximate the optimal recommendation performance using the
simulator, we adopt beam search (Reddy, 1977), a commonly used decoding
algorithm to generate the sequence that considers the long-term effects
(Freitag and Al-Onaizan, 2017). A beam search maintains a beam of $k$ possible
trajectories, updating them incrementally by ranking their extensions via the
objective score. Since the memory requirements for beam search is proportional
to the action size and the review dataset has a large number of actions (e.g.,
17,769 for ML-20M), we use beam size $k\in\\{1,10\\}$. Here, we note that the
beam search ($k=1$) is identical to the greedy policy.

Table 4. Relative performance of the greedy policy against the performance of the beam search at $k=10$. Beam size | EachMovie | ML-1M | ML-20M | Netflix
---|---|---|---|---
$k=1$ | 0.9960 | 0.9844 | 0.9981 | 0.9974

Table 4 reports the relative recommendation performance of the greedy search
(i.e., beam search at $k=1$) with respect to the beam search performance at
$k=10$. We found that the greedy search matches the performance of the beam
search by more than 99.5% for EachMovie, ML-20M, and Netflix dataset. This
implies that the benefits of considering the long-term effects are marginal
and provides an explanation for the high performance of GreedyRM for these
datasets in Table 2. We also observed a slight performance gap between the
greedy search and the beam search in ML-1M. This indicates that considering
the long-term effects can be slightly beneficial for ML-1M, which matches with
our experimental findings in Table 2 where DQNR achieved the best performance
in ML-1M.

##  5\. Conclusion and Discussion

Recently, a lot of community effort in IRS has been devoted to develop the RL-
based recommendation algorithms to effectively model the long-term effects
between the recommendations. However, our findings imply that the benefits of
accounting for the long-term effects can be marginal in the public review
dataset. Therefore, to accurately benchmark the performance of the RL-based
recommendation algorithms, it is crucial to validate the significance of the
long-term effects prior to the evaluation. We suggest that an evaluation
protocol should (i) perform a dataset validation procedure by comparing beam-
and greedy-search performance to verify the existence of beneficial long-term
rewards; and (ii) include a simple reward model that greedily selects items as
a baseline. We will make our code publicly available to ensure the
reproducibility of our work.

## Acknowledgments

This work was supported by Institute of Information & communications
Technology Planning $\&$ Evaluation (IITP) grant funded by the Korea
government(MSIT) (No. 2019-0-00075, Artificial Intelligence Graduate School
Program(KAIST), and No. 2020-0-00368, A NeuralSymbolic Model for Knowledge
Acquisition and Inference Techniques).

## References

  * (1)
  * Chen et al. (2019b) Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu. 2019b.  Large-scale interactive recommendation with tree-structured policy gradient. In _Proc. the AAAI Conference on Artificial Intelligence (AAAI)_ , Vol. 33. 3312–3320.
  * Chen et al. (2019a) Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019a.  Top-k off-policy correction for a REINFORCE recommender system. In _Proc. of the Web Search and Data Mining (WSDM)_. 456–464.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep Neural Networks for YouTube Recommendations. In _Proceedings of the 10th ACM Conference on Recommender Systems_.
  * Freitag and Al-Onaizan (2017) Markus Freitag and Yaser Al-Onaizan. 2017.  Beam Search Strategies for Neural Machine Translation.  _Proc. the Annual Meeting of the Association for Computational Linguistics (ACL)_ (2017), 56\.
  * Hu et al. (2018) Yujing Hu, Qing Da, Anxiang Zeng, Yang Yu, and Yinghui Xu. 2018.  Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application. In _KDD_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _Proc. the IEEE International Conference on Data Mining (ICDM)_. 197–206.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix Factorization Techniques for Recommender Systems.  _Computer_ (2009).
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010\.  A Contextual-Bandit Approach to Personalized News Article Recommendation. In _WWW_.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ (2015).
  * Liu et al. (2018) Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, and Yuzhou Zhang. 2018\.  Deep reinforcement learning based recommendation with explicit user-item interactions modeling.  _arXiv preprint arXiv:1810.12027_ (2018).
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ (2015).
  * Qin et al. (2014) Lijing Qin, Shouyuan Chen, and Xiaoyan Zhu. 2014.  _Contextual Combinatorial Bandit and its Application on Diversified Online Recommendation_.
  * Reddy (1977) Raj Reddy. 1977\.  Speech understanding systems: A summary of results of the five-year research effort. In _Carnegie Mellon University_.
  * Sarwar et al. (2001) Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001.  Item-Based Collaborative Filtering Recommendation Algorithms. In _Proceedings of the 10th International Conference on World Wide Web_. Association for Computing Machinery, New York, NY, USA, 285–295.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In _Proc. the ACM Conference on Information and Knowledge Management (CIKM)_.
  * Wang et al. (2020) Fan Wang, Xiaomin Fang, Lihang Liu, Hao Tian, and Zhiming Peng. 2020.  MBCAL: Sample Efficient and Variance Reduced Reinforcement Learning for Recommender Systems.  _Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_ (2020).
  * Wang et al. (2017) Huazheng Wang, Qingyun Wu, and Hongning Wang. 2017.  Factorization Bandits for Interactive Recommendation. In _AAAI_.
  * Wu et al. (2016) Qingyun Wu, Huazheng Wang, Quanquan Gu, and Hongning Wang. 2016\.  Contextual Bandits in a Collaborative Environment. In _SIGIR_.
  * Xin et al. (2020) Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020.  Self-supervised reinforcement learning for recommender systems. In _Proc. the ACM Conference on Research and Development in Information Retrieval (SIGIR)_. 931–940.
  * Zhao et al. (2018a) Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018a.  Deep reinforcement learning for page-wise recommendations. In _Proc. the ACM Conference on Recommender systems (RecSys)_. 95–103.
  * Zhao et al. (2018b) Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018b.  Recommendations with negative feedback via pairwise deep reinforcement learning. In _Proc. the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_. 1040–1048.
  * Zhao et al. (2020) Xiangyu Zhao, Xudong Zheng, Xiwang Yang, Xiaobing Liu, and Jiliang Tang. 2020.  Jointly learning to recommend and advertise. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3319–3327.
  * Zhou et al. (2020) Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang, Xiuqiang He, and Yong Yu. 2020\.  Interactive recommender system via knowledge graph-enhanced reinforcement learning. In _Proc. the ACM Conference on Research and Development in Information Retrieval (SIGIR)_. 179–188.
  * Zou et al. (2020) Lixin Zou, Long Xia, Yulong Gu, Xiangyu Zhao, Weidong Liu, Jimmy Xiangji Huang, and Dawei Yin. 2020.  Neural interactive collaborative filtering. In _Proc. the ACM Conference on Research and Development in Information Retrieval (SIGIR)_. 749–758.