# Exploration and Regularization of the Latent Action Space in Recommendation

Shuchang Liu  Kuaishou TechnologyBeijingChina<EMAIL_ADDRESS>,
Qingpeng Cai  Kuaishou TechnologyBeijingChina caiqingpeng@@kuaishou.com ,
Bowen Sun  Peking UniversityBeijingChina<EMAIL_ADDRESS>,  Yuhao Wang  City
University of Hong KongHong KongChina<EMAIL_ADDRESS>,  Ji Jiang
Peking UniversityBeijingChina<EMAIL_ADDRESS>,  Dong Zheng  Kuaishou
TechnologyBeijingChina<EMAIL_ADDRESS>,  Kun Gai
UnaffiliatedBeijingChina<EMAIL_ADDRESS>,  Peng Jiang† Kuaishou
TechnologyBeijingChina<EMAIL_ADDRESS>,  Xiangyu Zhao  City University
of Hong KongHong KongChina<EMAIL_ADDRESS>and  Yongfeng Zhang  Rutgers
UniversityNew Brunswick, NJUSA<EMAIL_ADDRESS>
(2023)

###### Abstract.

In recommender systems, reinforcement learning solutions have effectively
boosted recommendation performance because of their ability to capture long-
term user-system interaction. However, the action space of the recommendation
policy is a list of items, which could be extremely large with a dynamic
candidate item pool. To overcome this challenge, we propose a hyper-actor and
critic learning framework where the policy decomposes the item list generation
process into a hyper-action inference step and an effect-action selection
step. The first step maps the given state space into a vectorized hyper-action
space, and the second step selects the item list based on the hyper-action. In
order to regulate the discrepancy between the two action spaces, we design an
alignment module along with a kernel mapping function for items to ensure
inference accuracy and include a supervision module to stabilize the learning
process. We build simulated environments on public datasets and empirically
show that our framework is superior in recommendation compared to standard RL
baselines.

Recommender Systems, Reinforcement Learning, Representation Learning

$\dagger$ Corresponding author

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
ACM Web Conference 2023; May 1–5, 2023; Austin, TX, USA††booktitle:
Proceedings of the ACM Web Conference 2023 (WWW ’23), May 1–5, 2023, Austin,
TX, USA††price: 15.00††doi: 10.1145/3543507.3583244††isbn:
978-1-4503-9416-1/23/04††ccs: Information systems Recommender systems††ccs:
Computing methodologies Sequential decision making

##  1\. Introduction

Recommender Systems (RS) serve as one of the fundamental components for a wide
range of web services including e-commerce, social media, news, and
advertising. In recent years, studies have shown that the long-term
interactions between users and the RS formulate a Markov Decision Process
(MDP), where Reinforcement Learning (RL) methods can be used to further
improve the predictive performances (Afsar et al., 2021) compared to
traditional learning-to-rank solutions (Liu et al., 2009). Rather than
optimizing the immediate user response, the key insight behind RL is to
maximize the cumulative reward over all the interactions over time.

When adopting this formulation in practice, the recommendation problem
distinguishes itself from other RL tasks by the challenge of large dynamic
discrete action space. For example, in an e-commerce system or video
recommendation system, it may need to select for each request a recommendation
list from a pool of millions of products, and the candidate pool grows every
day. This means that it would be unrealistic for tabular-based methods (e.g.
Q-Learning, SARSA, Policy Iteration) that favor small action spaces and
methods designed for fixed action spaces. Though efforts have been made to
alleviate this issue by decomposition of the recommendation list (Ie et al.,
2019a) into item-wise sub actions, learning a policy gradient over the entire
candidate item pool is still an challenging task.

Fortunately, this challenge has already been solved in early-age non-RL models
like latent factor models (Koren et al., 2009) and two-tower collaborative
filtering method (Yi et al., 2019). They learn a common latent space that can
represent both the user requests and items (or item lists), so that the
learned latent space can accommodate arbitrary number of items and is agnostic
to the dynamic changes of the item pool. In this work, we combine this insight
into the RL-methods and focus on the list recommendation problem. The latent
representations of recommendation lists are generalized as hyper-actions as
shown in Figure 1. In a forward inference, the policy first propose a
vectorized hyper-action, then this latent vector will serve as a deterministic
function that rank, and finally select a list of items, denoted as effect-
action, from the candidate pool.

Note that this extra implicit inference step also induces new challenges to
the RL solution: Most importantly, it introduces inconsistency between the two
actions spaces. Specifically, we want to apply efficient end-to-end training
on the hyper-actions but it is the effect-actions that actually interact with
the users. On the other hand, the most accurate latent representation of the
discrete effect-action may not be exactly the same as that of the proposed
hyper-action that selects it, so the inference accuracy is not guaranteed.
Additionally, it also introduces an extra exploration stage and it is unknown
whether one should explore hyper-actions on the latent space or explore the
discrete effect-action space. All of these add up to the instability and
uncertainty of the learning, so we need a solution that can regulate the two
action spaces and stabilize the RL process.

Figure 1. MDP formulation with latent hyper-action

To solve the aforementioned challenges, we propose a general Hyper-Actor
Critic (HAC) learning framework that contains four components: the user state
and hyper-action generator; the scoring function that maps hyper-actions into
effect-actions (i.e. recommendation lists); the critic network that evaluates
both the hyper-action space and the effect-action space; and an inverse
mapping module that infers the hyper-action back based on the effect action.
During training, the backbone actor-critic learning paradigm is augmented with
an alignment module that ensures consistency between two action spaces and a
supervision module that improves the stability and effectiveness of RL. The
resulting framework generalizes many existing solutions to recommendation
tasks like DDPG and Online/Offline Supervise Learning (SL).

We summarize our contribution as follows:

  * •

We propose a practical and efficient RL framework that learns to recommend a
list of items from a large item pool through a latent hyper-action.

  * •

We build online simulators based on public datasets and empirically show that
the resulting framework achieves better performance compared to standard RL
and SL solutions.

  * •

We also point out that providing supervision and regulating the consistency
between the hyper-action space and the effect-action space are helpful for
improving the sampling efficiency and inference accuracy.

##  2\. Method

###  2.1. Problem Formulation

Here we consider the session-based recommendation scenario where the system
considers an item pool of size $N$, denoted as $\mathcal{I}$, and for each
user we observe the static user features $\bm{u}$ and the interaction history
of a session $x_{1:t}=\\{x_{1},\dots,x_{t}\\}$ where
$x_{i}\in\mathcal{I}(1\leq i\leq t)$. And the goal is to interactively
recommend lists of items to users that maximizes user cumulative reward over
the session of interactions. As we have described in section 1, we emphasize
the existence of the latent action space and formulate the recommendation task
as a modified Markov Decision Process (MDP) as captured in Figure 1. Then, the
MDP components with the latent action space become:

  * •

$\mathcal{S}$: the continuous representation space of user state.

  * •

$\mathcal{A}$: the final effect-action space corresponds to the possible
recommendation lists. Without loss of generality, we consider the list of
fixed size $k$ so the action space is $\mathcal{A}=\mathcal{I}^{k}$.

  * •

$\mathcal{Z}$: the latent hyper-action space which is a continuous vector
space that encodes how the effect-action will be selected. We assume a many-
to-one mapping function $f:\mathcal{Z}\rightarrow\mathcal{A}$.

  * •

$\mathcal{R}$: The cumulative reward function that estimates the user feedback
in the user session, and the immediate reward $r(s,a)$ captures the single
step reward when taking an action $a\in\mathcal{A}$ on state
$s\in\mathcal{S}$.

Note that there is an implicit transition model
$\mathcal{P}(s_{t}^{\prime}|s_{t},a_{t})$ that describes the probability of
reaching a certain new state $s_{t}^{\prime}$ after taking action $a_{t}$ on
state $s_{t}$. In RS, this transition function is integrated into the user
state encoder and is usually modeled by a sequential model that takes the user
interaction sequence as input.

At each interaction step in a user session, given a user’s current state
$s_{t}\in\mathcal{S}$ (e.g. portrait and history interactions), the
recommendation policy $\pi_{\theta}(a_{t}|s_{t})$ first infer a hyper-action
representation $Z_{t}$ and then generates a list of items as the effect action
$a_{t}$. The user’s feedback along with the updated user state is returned
from the user environment and the reward function $r(s_{t},a_{t})$ is assumed
as given so is regarded as part of the environment. The goal is to find an
optimal recommendation policy
$\pi^{*}(a_{t}|s_{t}):\mathcal{S}\rightarrow\mathcal{A}$ that maximizes the
expected cumulative reward throughout the session:

(1)
$\mathbb{E}_{\tau\sim\pi}[\mathcal{R}(\tau)]=\mathbb{E}_{\tau\sim\pi}\Big{[}\sum_{t=0}^{|\tau|}\gamma^{t}r(s_{t},a_{t})\Big{]}$

where $\tau=[(s_{0},a_{0},r_{0}),(s_{1},a_{1},r_{1}),\dots]$ denotes the
sampled trajectories, and $\gamma\in[0,1)$ denotes the discount factor for
future reward.

###  2.2. Overall Framework

We present our framework as Figure 2, and denote it as the Hyper-Actor Critic
(HAC) learning method. The recommendation policy $\pi(a_{t}|s_{t})$ is
decomposed into an hyper-actor network $P(Z_{t}|s_{t})$ that generates a
vectorized hyper-action and a ranking scorer $P(a_{t}|Z_{t})$ that select the
final recommendation list based on the hyper-action. Then we propose to share
the critic network between action spaces so that it can evaluate either the
hyper-action or the final effect-action (i.e. the recommendation list). Our
framework uses DDPG (Lillicrap et al., 2016) as foundation, but differently,
we address the importance of using different action spaces for actor learning
and critic learning. Specifically, we optimize the critic based on effect-
actions to guarantee the accuracy of action/state evaluation, and use hyper-
actions to optimize the actor so that efficient end-to-end training and
exploration can be applied. To ensure consistency between the two different
action spaces, we also learn an inverse pooling module with item kernel
functions to infer the hyper-action back from the effect-action. This means
that the evaluation of the two action spaces will share the same critic, and
the knowledge learned from the effect-action can be transferred to hyper-
actions. To stabilize the learning process, we also include supervision on the
effect-action using immediate user responses.

Figure 2. Hyper-Actor Critic (HAC) learning framework. $\odot$ represents the
scoring function that selects items from $\mathcal{I}$,

###  2.3. User State and Hyper-Actor

In the case of RS, the observable information of a user usually includes the
up-to-date user interaction history $\bm{x}_{1:t}$ and the static demographic
features $\bm{u}$. Then one can use any sequential model to encode this
information and infer the dynamic user state:

(2) $\bm{s}_{t}=\mathrm{StateEnc}(\Phi(x_{1}),\dots,\Phi(x_{t}),\bm{u})$

where the item kernel function $\Phi$ maps the items’ raw features into a
dense embedding in the kernel space. We also use a user kernel function to map
the user features $\bm{u}$ into the same kernel space of items, then
concatenate the sequence of history items with the user embedding as the new
sequence.

To encode user features and histories, there exist several feasible sequential
models such as (Jannach and Ludewig, 2017; Kang and McAuley, 2018; Sun et al.,
2019), and we use the state encoder in SASRec (Kang and McAuley, 2018) as our
backbone since it can capture both the sequential patterns in the dynamic
history and the user-item correlations through the self-attention mechanism.
With the state encoded, a vectorized representation (i.e. the hyper-action) is
inferred by an Multi-Layer Perceptron (MLP) module:

(3) $\bm{Z}_{t}=\mathrm{MLP}(\bm{s}_{t})$

We assume that the distribution of this hyper-action follows the standard
Gaussian $\mathcal{N}(\bm{Z}_{t},\sigma_{Z}^{2})$ and we can use the
reparameterization trick to engage the end-to-end training.

###  2.4. Scoring Functions and Effect Action

Given $\bm{Z}_{t}$ that contains sufficient information of user preferences,
we can assume that the selection of items only depends on this latent vector.
In other words, we have conditional independence:

(4) $P(a_{t}|\bm{s}_{t},\bm{Z}_{t})=P(a_{t}|\bm{Z}_{t})$

Note that this setting also indicates that the inferred $\bm{Z}_{t}$ is a
hyper-action that can be considered as the parameters of the later item
selection module. As a result, the overall recommendation policy follows
$\pi(a_{t}|s_{t})\sim P(a_{t}|\bm{Z}_{t})$.

When generating the effect-action, we select items from the candidate pool
according to a scoring function parameterized by $\bm{Z}_{t}$. The scoring
function provides a ranking score for each of the candidate items in
$\mathcal{I}$, and the final recommendation list is generated through either
top-$k$ selection or categorical sampling. Taking the SASRec model as an
example, the scoring function is a dot product between the kernel item
embedding and the encoded user state:

(5) $\mathrm{score}(i|Z_{t})=\Phi(i)^{\top}\bm{Z}_{t}$

Note that the parameters of the item kernel function $\Phi$ are not considered
as part of the hyper-action $\bm{Z}_{t}$ since it is independent of the given
user state. In order to engage efficient learning and inference, one can
assume that selection of each item is conditionally independent of each other:
$P(a_{t}|Z_{t})=\prod_{i\in a_{t}}P(i|Z_{t})$, similar to the slate
decomposition in (Ie et al., 2019a). Then, we can define the selection or
sampling probability of an item as
$P(i|Z_{t})=\mathrm{softmax}_{\mathcal{I}}(\mathrm{score}(i|Z_{t}))$.

###  2.5. Shared Critic and The Inverse Module

The purpose of the critic is to accurately evaluate the long-term quality of
the state-action pair (e.g. Q function in DDPG) or the expected value of the
given state (e.g. V function in A2C) so that it can effectively guide the
actor learning and the action exploration. Compared to the standard RL
framework, the new problem setting allows us to evaluate either the hyper-
action with $Q(s_{t},\bm{Z}_{t})$ or the effect-action with $Q(s_{t},a_{t}$)
or both. In order to ensure consistent evaluation of the actions from
different spaces, we propose to transfer knowledge between $Q(s_{t},a_{t})$
and $Q(s_{t},Z_{t})$ through a shared critic network. As shown in Figure 2,
this shared critic is a mapping function
$g:\mathcal{S}\times\mathcal{Z}\rightarrow\mathbb{R}$, that takes the user
state $s_{t}$ and the action embedding in the kernel space $Z_{t}$. Or
equivalently, $Q(s_{t},Z_{t})=g(s_{t},Z_{t})$. To evaluate the effect-action,
an inverse module $h$ is introduced to infer the hyper-action back:

(6) $\hat{\bm{Z}}_{t}=h(a_{t})=\mathrm{pooling}(\Phi(i)|i\in a_{t})$

and the evaluation becomes $Q(s_{t},a_{t})=g(s_{t},\hat{\bm{Z}}_{t})$. In
practice, we found that the average pooling of the item embedding in the
kernel space generates the most stable result, though there are infinitely
many latent $\bm{Z}$ that can generate the same list. Compared to existing
solutions that infer the latent action using adjacent states like PG-RA
(Chandak et al., 2019), we believe that the effect-action in recommendation
task has sufficient information to recover the hyper-action. In addition, we
use a reconstruction loss to further regulate the consistency between the
hyper-action and the effect-action through an alignment loss function. As we
will describe in the next section, it ensures that the generated hyper-action
$\bm{Z}_{t}$ is in the valid region close to the candidate items in the kernel
space.

###  2.6. Overall Learning Framework

The overall optimization process is a modified actor-critic learning framework
that consists of a critic loss, an actor loss, a hyper-actor loss, and a
supervised loss. And a experience replay buffer $\mathcal{D}$ will collect the
sample records in the form of $(s_{t},a_{t},r(s_{t},a_{t}),s_{t+1},d)$ where
$d$ is the termination signal indicating whether the user has left. The critic
loss aims to train an accurate evaluator that captures the patterns in the
quality of actions:

(7) $\displaystyle\mathcal{L}_{\mathrm{TD}}$
$\displaystyle=\mathbb{E}_{\mathcal{D}}\Big{[}(r(s_{t},a_{t})+\gamma(1-d)Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t}))^{2}\Big{]}$

where $a_{t+1}$ is generated by the recommendation policy using greedy method
(equivalent to finding $\arg\max Q(s_{t},a_{t})$ when the Q function is
accurate). This is a standard TD error and we only calculate Q for the effect-
action when learning the critic to ensure the accuracy of evaluation. Note
that in DDPG-based methods, each actor and critic is paired with a target
network which is used to estimate future state-action pairs
$Q(s_{t+1},a_{t+1})$. These target networks adopt elastic updates from the
actor and critic so that their slow optimization can stabilize the learning.

Then, with an accurate critic as the evaluator, we can efficiently learn our
actor by maximizing the Q-value of the inferred action, which is equivalent to
minimizing the following actor loss:

(8)
$\mathcal{L}_{\mathrm{QMax}}=\mathbb{E}_{\mathcal{D}}\Big{[}Q(\bm{s}_{t},\bm{Z}_{t})\Big{]}$

where $\bm{Z}_{t}$ is inferred by the hyper-actor as described in section 2.3.
Note that the learning of critic and actor uses different action spaces, so we
need to align the two spaces to avoid the mode collapse (Srivastava et al.,
2017) of the generated hyper-action. In our solution, we use the L2
regularizer to ensure this consistency:

(9)
$\mathcal{L}_{\mathrm{Hyper}}=\mathbb{E}_{D}\Big{[}\|\bm{Z}_{t}-\hat{\bm{Z}}_{t}\|^{2}\Big{]}$

where $\bm{Z}_{t}$ is produced by the hyper-action based on state
$\bm{s}_{t}$, and $\hat{\bm{Z}}_{t}$ is generated by first greedily select the
effect action using $\bm{Z}_{t}$ as described in section 2.4 and then
reconstruct the hyper-action back using the inverse module as described in
section 2.5.

Additionally, to better stabilize the RL and exploit the detailed user
response signals on each item, we also include a supervised learning objective
based on the effect-action.

(10) $\mathcal{L}_{\mathrm{BCE}}=\mathbb{E}\Big{[}\sum_{i\in a_{t}}y_{t,i}\log
P(i|\bm{Z}_{t})+(1-y_{t,i})\log(1-P(i|\bm{Z}_{t}))\Big{]}$

which is a binary cross-entropy loss where $y_{t,i}$ is the ground truth user
response on the exposed item $i$. We remind readers that there are other
advanced supervision and regularization methods that can accommodate RL-based
models and potentially extend our framework. For example, (Fujimoto and Gu,
2021) could supervise the effect-action space for off-policy training, and
(Xin et al., 2022) is well-suited for the distance control on the hyper-action
space. As a summary, we present the resulting learning paradigm as algorithm
1. And note that the parameters of the inverse module come from the item
kernel, so both Eq.(9) and Eq.(10) update them as in line 8-9.

Algorithm 1 Hyper-Actor Critic Training

1: procedure HAC

2: Initialize all trainable parameters in the actors, critics, and the item
kernel function.

3: Initialize replay buffer $\mathcal{B}$.

4: while Not Converged, in each iteration do

5: Apply current policy in running episodes, collect and store samples to
$\mathcal{B}$.

6: Sample mini-batch of
$(\bm{s}_{t},a_{t},r(\bm{s}_{t},a_{t}),\bm{s}_{t+1},d)\sim\mathcal{B}$.

7: Update actor and critic with loss Eq.(8) and Eq.(7).

8: Update actor and the kernel with loss Eq.(9), if any action alignment.

9: Update actor and the kernel with loss Eq.(10), if any supervision.

10: end while

###  2.7. Exploration in Hyper-Action Space and Effect-Action Space

Readers may notice that the inclusion of latent hyper-action also introduces
an extra action sampling step as shown in Figure 2, so the resulting framework
allows both the sampling on the hyper-actions space (e.g. by adding Gaussian
noise) and the sampling on the effect-action space (e.g. categorical sampling
of items based on ranking scores). Theoretically, this indicates that the
sampling probability of effect-actions should be described as the following:

(11)
$P(a_{t}|\bm{s}_{t})=\int_{Z_{t}}P(a_{t}|\bm{Z}_{t})P(\bm{Z}_{t}|\bm{s}_{t})$

When the effect-action generation is deterministic, the exploration only
depends on the sampling of $\bm{Z}_{t}$, similar to that in (Chandak et al.,
2019); and if the hyper-actor is deterministic, the exploration only depends
on the effect-action sampling as in standard policy gradient methods. Note
that the variance $\sigma_{Z}^{2}$ of the hyper-action controls the
uncertainty of the inferred latent action embedding, and it is critical to
find an adequate value that can improve the exploration effectiveness in RL.
On one hand, giving a variance that is too small will limit the exploration of
new actions resulting in sub-optimal results; On the other hand, making the
variance too large will induce unstable action exploration that hardly
converges.

In general, we would like to take advantage of the efficient learning and
exploration of the hyper-action space, so it becomes critical to align the
distribution of $\bm{Z}_{t}$ and the embedded item in the kernel space, as we
mentioned in section 2.6. As we showcase in Figure 3, the item kernel function
helps increase the expressiveness of the policy by folding the action space
where exploration could be more efficient. Though we are skeptical whether
there is a guarantee for all RL solutions that explores the latent action
space, we will empirically show the effectiveness of encoding both users and
items into the same kernel space and regulating the action with the inverse
pooling module using Eq.(9) in section 3.4.

Figure 3. Exploration in Different Action Spaces

##  3\. Experiments

###  3.1. Experimental Settings

####  3.1.1. Datasets:

We include three public datasets in our experiments:
RL4RS111https://github.com/fuxiAIlab/RL4RS is a session-based dataset (Wang et
al., 2021b) that is first introduced in the BigData Cup 2021 to boost RL
research; ML1M222https://grouplens.org/datasets/movielens/1m/ is the MovieLens
data (Harper, 2015) with 1 million records which consists of user’s ratings of
movies; KuaiRand1K333https://kuairand.com/ is a recent dataset for sequential
short-video recommendation, and we use the 1K version (Gao et al., 2022) which
has irrelevant videos removed. We preprocess all these datasets into a unified
format where each record consists of (user features, user history, exposed
items, user feedback, and timestamp) in sequential order. The details of this
process are summarized in Appendix A.1, and the resulting dataset statistics
is provided in Table 1.

Dataset | $|\mathcal{U}|$ | $|\mathcal{I}|$ | #record | $k$
---|---|---|---|---
RL4RS | - | 283 | 781,367 | 9
MovieLens-1M | 6400 | 3706 | 1,000,208 | 10
KuaiRand | 986 | 11,643 | 969,831 | 10
Table 1. Dataset Summary. $k$ represents the size of the recommendation list
(i.e. the effect-action size). RL4RS dataset provides user profile features
instead of user ID so it does not have a count for the user set.

####  3.1.2. Online Environment Simulator

To simulate the online user interaction we train a user response model
$\Psi:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}^{k}$ for each of the
dataset. The user state is derived based on the static user features and the
dynamic history interactions. $\Psi$ outputs the probabilities of the user’s
positive feedback on each item in recommended $a_{t}$, and the final response
$y_{t}\in\\{0,1\\}^{k}$ (e.g. click) is uniformly sampled according to the
probabilities. We design the reward $r(s_{t},a_{t})\in[-0.2,1.0]$ as the
average of the item-wise reward. We provide details of the environment in
Appendix A.2.

####  3.1.3. Models and Baselines

We use SASRec (Kang and McAuley, 2018) as our backbone actor as described in
section 2.3, it consists of a Transformer-based state encoder and hyper-action
generator, and the dot-product scorer. We also implemented the following RL
baselines using our proposed HAC framework to better showcase how our HAC
generalizes existing methods:

  * •

Online SL: the SASRec actor directly learns from immediate user feedback
instead of the long-term commutative reward.

  * •

A2C: the synchronized version of the A3C (Mnih et al., 2016) that applies the
policy gradient on the effect-action space.

  * •

DDPG: a Deep DPG framework using the hyper-action space for both the actors
and critics (Lillicrap et al., 2016). This method is equivalent to our HAC
model without supervision.

  * •

TD3: improve the DDPG with double Q-learning so the training of critic becomes
more stable (Fujimoto et al., 2018).

  * •

DDPG-RA: the DDPG framework with the action representation learning as in
(Chandak et al., 2019). This method is closest to our work and it regulates
the effect-actions while our HAC model aligns the hyper-actions.

To better compare RL and SL methods, we also include the Offline SL that
optimizes the policy using Eq.(10) based on the offline data instead of the
online environment. The model architectures and specifications of these models
are provided in Appendix A.3.

Model | RL4RS | ML1M | KuaiRand
---|---|---|---
Total Reward | Depth | Total Reward | Depth | Total Reward | Depth
Offline SL | 6.721 | 8.163 | 18.559 | 18.717 | 14.394 | 14.982
Online SL | 9.502 | 10.571 | 18.629 | 18.780 | 13.456 | 14.147
A2C | 7.789 | 9.140 | 16.158 | 16.556 | 12.460 | 13.250
DDPG | 8.337 | 9.588 | 17.205 | 17.508 | 11.394 | 12.313
TD3 | 8.553 | 9.791 | 17.545 | 17.814 | 11.777 | 12.664
PG-RA | 8.561 | 9.728 | 18.466 | 18.633 | 10.859 | 11.814
HAC | 10.059 | 11.102 | 18.863 | 18.988 | 14.789 | 15.335
Table 2. Online Performance. The best performances in bold and second best in
Underline

####  3.1.4. Evaluation

For all datasets, we split them into the first 80% for training and the last
20% for evaluation according to record timestamps. We then pretrain the online
environment on the training set, and pretrain another online environment on
the entire dataset for later evaluation. We train our recommendation policies
in the first environment and evaluate them in the second. During training, we
set the discount of reward as $\gamma=0.9$ and limit the interaction depth to
$\leq 20$ for all experiments. We find that most RL-based methods converge and
stabilize within 50,000 iterations. For long-term evaluation metrics, we
consider the Total Reward that represents the summation of the rewards in an
entire user session and the Depth represents how many interactions the user,
and each session is observed using the simulated online environment that
interacts with the learned policies. And for both metrics, a higher value
means better performance. To evaluate the stability of the learned policies,
we include a Reward Variance metric that estimates how inconsistent a policy
would deal with different user states, so a lower value indicates a more
stable policy. Note that this metric describes the variance across states not
the variance across random seeds. In each experiment, we evaluate all
aforementioned metrics and report the average values across different user
sessions.

###  3.2. Effectiveness

For all tasks, the goal of the recommender system is to maximize the long-term
satisfaction represented by total reward and average depth. For each model, we
grid-search the hyper-parameters and pick the setting with the best results to
report in Table 2.

Main result: We can see that the proposed HAC framework consistently achieves
the best performances across all datasets on both long-term metrics: 6%
improvement on Rl4RS, 1% on ML1M, and 3% on KuaiRand over the best baselines.
This indicates the expressiveness of the proposed hyper-actions and the
effectiveness of the learning method. Note that all other RL methods can only
achieve better results than offline supervised learning in the RL4RS task, but
become worse in ML1M and KuaiRand with larger effect-action spaces. This
indicates that our HAC framework can better capture the patterns for larger
action spaces.

RL-baselines: Among RL solutions, A2C always has the worst performance and
appears to be the most unstable learning framework, but the gap between A2C
and DDPG tends to be smaller in datasets (ML1M and KuaiRand) with larger
action spaces and A2C even achieves better performances than DDPG in KuaiRand
with the largest action space. Since A2C directly optimizes the effect-action
and DDPG uses the hyper-action, this reduced gap may indicate that it may
become harder to learn consistent and accurate hyper-action representations in
larger effect-action spaces. This may also proves that ensuring consistency
between the two action spaces is critical to achieving effective RL. TD3
slightly improves the performance over DDPG but still behave in a similar way.

Action space regularization:  In addition to our method, The DDPG-RA method
also addresses the alignment of action spaces and has the closest behavior to
our method. Differently, it does not regulate the hyper-actions has HAC,
instead, it aligns the effect-action space that is not directly used in
guiding the actor. Additionally, DDPG-RA uses the hyper-action rather than the
effect-action when learning critics, so it achieves better results than A2C
and DDPG, but does not surpass our method or even supervised methods.

###  3.3. Learning HAC

Figure 4. Training curves of HAC without supervision on RL4RS. X-axis
corresponds to the number of iteration. The four losses are presented in log
scales. Figure 5. Effect of supervision loss on KuaiRand. X-axis represents
the learning rate of Eq.(10).

Supervision and stable learning: To further illustrate the training behavior
of the HAC model, we plot the learning curves of HAC in Figure 4 where we
compare a “HAC w/o supervision” that has no gradient for the supervised loss
Eq.(10). All methods saturate on the performance metric as shown in the penal
“Total Reward” and can successfully reduce all four loss functions mentioned
in section 2.6. Note that “HAC w/o supervision” can still successfully reduce
the BCE loss on each item, indicating the effectiveness of RL based on the
reward that aggregates the item-wise signals. We can also see that including
the supervision would help boost the model performance and reduce the actor
loss $\mathcal{L}_{\mathrm{QMax}}$ that helps explore better actions. Note
that HAC has a lower critic loss $\mathcal{L}_{\mathrm{TD}}$ than “HAC w/o
supervision”, which indicates a more stable learning process. We can also
verify this by observing the variance of the total reward across users since
higher reward variance indicates that the learned policy is less capable of
providing good actions under different user states. As shown in Figure 5 for
the KuaiRand environment, increasing the importance of supervised loss would
help improve the recommendation performance and reduce the variance. Yet,
assigning the supervision module with a learning rate (0.001 in KuaiRand) that
is too large may over-exploit the user’s intention and harm the performance.
We observe similar patterns in the other two datasets and provide the results
in Appendix B.

Figure 6. Training curves of HAC on RL4RS. The reward variance correspond to
the variance of total reward across different users. The four losses are
presented in log scales. $\lambda_{h}$ represents the magnitude of hyper-
action alignment. $\lambda_{h}=0$ does not include this alignment loss so has
$\mathcal{L}_{\mathrm{Hyper}}=0$. Figure 7. Effect of hyper-action alignment
on ML1M. X-axis represents the magnitude of Eq.(9), i.e. $\lambda_{h}$ in
Figure 6.

Hyper-action regulation: We conduct the same ablation experiments when
comparing different magnitudes of hyper-action alignment for loss Eq.(9) and
plot the learning curves of HAC in Figure 6 In general, including the hyper-
action alignment ($\lambda_{h}=0.1$ and $\lambda_{h}=1$) would cause the HAC
framework to learn slower than the HAC that only uses supervision
($\lambda_{h}=0$) in terms of the convergence of total reward and reward
variance. In contrast, the more consistent action space helps the model to
learn and explore better action policies. Besides, increasing the importance
of this alignment module results in worse $\mathcal{L}_{\mathrm{QMax}}$ and
better $\mathcal{L}_{\mathrm{TD}}$, indicating that critic is more accurate in
capturing the quality of actions. Note that $\lambda_{h}=1$ is more stable
than $\lambda_{h}=0.1$ but may be less effective in exploration. To better
verify this We illustrate the evaluation result in Figure 7 where the results
exhibit a best point $\lambda_{h}=0.1$ where the recommendation is higher and
more stable in reward.

###  3.4. Ablation Study

Model components: To better investigate how different learning modules in our
framework work, we compare several alternatives of our method: 1) DDPG: HAC
without supervision and action alignment, and it uses hyper-action space for
both actor learning and critic learning; 2) HAC w/o
$\mathcal{L}_{\mathrm{BCE}}$: excluding the supervision of HAC; 3) HAC w/o
$\mathcal{L}_{\mathrm{Hyper}}$: excluding the hyper-action alignment module.
We summarize the results in Figure 8 for ML1M dataset. We can see that
excluding either the supervision or the alignment module would reduce the
performance and increase the reward variance. This indicates that both modules
help improve the model’s accuracy and learning stability. Similar results are
also observed in other datasets as augmented in Appendix B. Note that DDPG
achieves relatively the same performance as HAC w/o
$\mathcal{L}_{\mathrm{BCE}}$, this indicates that using separating action
spaces for actor learning and critic learning as in HAC may reduce the
performance and simply including an action alignment module would not fill the
gap. In this sense, HAC needs both hyper-action alignment and supervision.
This also means that the inconsistency between the two action spaces is
smaller than the inconsistency between the aggregated reward and item-wise
user response signals.

Figure 8. Effect of model components on ML1M. Figure 9. Effect of HAC’s hyper-
action exploration magnitude on RL4RS dataset. X-axis correspond to the
Gaussian noise variance for hyper-actions in HAC model. Figure 10. HAC’s
effect-action exploration magnitude on RL4RS dataset. X-axis correspond to the
rate of greedy top-k selection rather than categorical sampling for effect-
actions in HAC model, and we set hyper-action noise to 0.

Exploration on different action spaces:  In terms of the exploration of HAC
model, we can apply exploration on both the hyper-action space and the effect-
action space. To compare the effect of different magnitude of hyper-action
exploration, we change the variance of the Gaussian noise during learning and
fix all other hyperparameters of HAC, and present the results in Figure 9. The
comparison of recommendation performance shows an optimal point in the middle
of the search space, indicating that one should carefully design the
exploration so that the sampling variance is not too small or too large. As we
have discussed in section 2.7, small variance may limit the exploration of new
actions and large variance may induce unstable exploration that hardly
converges. And empirically, we find that sampling on effect-actions is less
effective than exploration on hyper-actions. As the example in Figure 10,
applying top-$k$ greedy selection achieves the best result and adding
categorical sampling would make the learned policy sub-optimal.

##  4\. Related Work

###  4.1. Sequential Recommendation and Session-based Recommendation

The session-based recommendation (SBR) problem is closely related to the
sequential recommendation (SR) task (Wang et al., 2021a). An SR task aims to
learn a policy that can infer the next recommendation (item or list) based on
the given user’s historical interactions. In comparison, the SBR considers the
existence of the beginning and termination of an interaction session. We see
our setting as somewhere intersects the two notions: by setting the list size
to 1, it would be almost identical to SR except for the reward setup; yet, our
goal is to optimize the entire future reward of the user session, which is
closer to the next-partial SBR as defined in (Wang et al., 2021a). Under both
problem settings, the most adopted solution uses the Markov Chain assumption
to model the dynamic transitions of the user-system interactions. The main
challenge of this line of work is how to construct a representative user state
based on the long-term history. Early solutions to the recommendation problem
adopted the collaborative filtering techniques (Rendle et al., 2010; Cheng et
al., 2016; Liang et al., 2018; Koren et al., 2022) and later approaches
embrace deep neural networks like Recurrent Network (Jannach and Ludewig,
2017), Convolution Network (Tang and Wang, 2018), Memory Network (Chen et al.,
2018), Self-Attention (Kang and McAuley, 2018; Sun et al., 2019), GCN (Wu et
al., 2019; Chang et al., 2021), Machine Reasoning (Shi et al., 2020; Chen et
al., 2021b; Ji et al., 2023) and Foundation Models (Geng et al., 2022) to
improve the model expressiveness, so that it can better capture the abundant
and complex information from user/item features and interaction histories. The
key insight behind all these methods is to accurately encode the long-range
histories, but this paradigm does not optimize the long-term user rewards.

###  4.2. Reinforcement Learning in Recommendation

The RL-based RS (Shani et al., 2005; Sutton and Barto, 2018; Afsar et al.,
2021) also follows the MDP formulation and it emphasizes the importance of
optimizing the cumulative reward that represents long-term user satisfaction.
In the simplest setting, tabular-based methods (Mahmood and Ricci, 2007) are
used to optimize an evaluation table but only work for a fixed set of state-
action pairs. Then value-based methods (Taghipour et al., 2007; Zhao et al.,
2018b, 2021a; Pei et al., 2019) and policy gradient methods (Chen et al.,
2019a, b; Ge et al., 2021; Xian et al., 2019; Ge et al., 2022; Li et al.,
2022) are proposed to learn to evaluate and optimize the action policy based
on the sampled long-term reward. The actor-critic paradigm (Zhao et al.,
2018a; Xin et al., 2020; Zhao et al., 2020) integrates these two methods by
simultaneously learning an action evaluator and an action generator. The main
challenges of RL-based RS consist of the large combinatorial state/actions
space (Dulac-Arnold et al., 2015; Ie et al., 2019a; Liu et al., 2020),
regulating the unstable learning behavior (Bai et al., 2019; Chen et al.,
2021a), and finding optimal reward function for heterogeneous user behaviors
(Chen et al., 2021c; Cai et al., 2022). Our work focus on the action space
representation learning and the stability of RL. And we consider PG-RA
(Chandak et al., 2019) as the closest work that also emphasizes the learning
of latent action representations. As we have mentioned in section 3.1, PG-RA
aims to transfer knowledge through a shared scoring function and applies
action alignment on the effect-action space, which is not well suited for the
latent-factor decomposition for users and items. We have empirically verified
this inferior performance in 3.2. Additionally, we have illustrated the
effectiveness of our method through evaluation on different simulated
environments, but we remind readers that there is still a chance that the
actual online environment is a more complex and dynamic mechanism. In this
sense, there are works focusing on building a more realistic online user
environment for RS (Ie et al., 2019b; Zhao et al., 2019, 2021b) which could
complement our work in practice.

##  5\. Conclusion

We proposed a practical actor-critic learning framework that optimizes a
recommendation policy in both the discrete effect-action space as well as the
vectorized hyper-action space. Empirical results verified the effectiveness
and stability of our proposed method in online environments built on public
datasets. The main purpose of our design is the generalization ability that
can accommodate various existing solution frameworks including Actor-Critic,
DDPG, as well as supervised learning. And we consider the notion of hyper-
action as a preliminary attempt on applying more sophisticated scoring
functions than a simple dot product.

###### Acknowledgements.

This research was partially supported by APRC—CityU New Research Initiatives
(No.9610565, Start-up Grant for New Faculty of City University of Hong Kong),
SIRG—CityU Strategic Interdisciplinary Research Grant (No.7020046,
No.7020074), and CityU—HKIDS Early Career Research Grant (No.9360163)

## References

  * (1)
  * Afsar et al. (2021) M Mehdi Afsar, Trafford Crump, and Behrouz Far. 2021\.  Reinforcement learning based recommender systems: A survey.  _ACM Computing Surveys (CSUR)_ (2021).
  * Bai et al. (2019) Xueying Bai, Jian Guan, and Hongning Wang. 2019.  A model-based reinforcement learning with adversarial training for online recommendation.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Cai et al. (2022) Qingpeng Cai, Ruohan Zhan, Chi Zhang, Jie Zheng, Guangwei Ding, Pinghua Gong, Dong Zheng, and Peng Jiang. 2022\.  Constrained Reinforcement Learning for Short Video Recommendation.  _arXiv preprint arXiv:2205.13248_ (2022).
  * Chandak et al. (2019) Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip Thomas. 2019.  Learning action representations for reinforcement learning. In _International conference on machine learning_. PMLR, 941–950.
  * Chang et al. (2021) Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021\.  Sequential recommendation with graph neural networks. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 378–387.
  * Chen et al. (2019b) Haokun Chen, Xinyi Dai, Han Cai, Weinan Zhang, Xuejian Wang, Ruiming Tang, Yuzhou Zhang, and Yong Yu. 2019b.  Large-scale interactive recommendation with tree-structured policy gradient. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 3312–3320.
  * Chen et al. (2021b) Hanxiong Chen, Shaoyun Shi, Yunqi Li, and Yongfeng Zhang. 2021b.  Neural collaborative reasoning. In _Proceedings of the Web Conference 2021_. 1516–1527.
  * Chen et al. (2019a) Minmin Chen, Alex Beutel, Paul Covington, Sagar Jain, Francois Belletti, and Ed H Chi. 2019a.  Top-k off-policy correction for a REINFORCE recommender system. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 456–464.
  * Chen et al. (2021a) Minmin Chen, Bo Chang, Can Xu, and Ed H Chi. 2021a.  User response models to improve a reinforce recommender system. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. 121–129.
  * Chen et al. (2018) Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018.  Sequential recommendation with user memory networks. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 108–116.
  * Chen et al. (2021c) Xiaocong Chen, Lina Yao, Aixin Sun, Xianzhi Wang, Xiwei Xu, and Liming Zhu. 2021c.  Generative inverse deep reinforcement learning for online recommendation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 201–210.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Dulac-Arnold et al. (2015) Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2015.  Deep reinforcement learning in large discrete action spaces.  _arXiv preprint arXiv:1512.07679_ (2015).
  * Fujimoto and Gu (2021) Scott Fujimoto and Shixiang Shane Gu. 2021.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ 34 (2021), 20132–20145.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger. 2018.  Addressing function approximation error in actor-critic methods. In _International conference on machine learning_. PMLR, 1587–1596.
  * Gao et al. (2022) Chongming Gao, Shijun Li, Yuan Zhang, Jiawei Chen, Biao Li, Wenqiang Lei, Peng Jiang, and Xiangnan He. 2022\.  KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos. In _Proceedings of the 31st ACM International Conference on Information and Knowledge Management_ (Atlanta, GA, USA) _(CIKM ’22)_. 5 pages.  https://doi.org/10.1145/3511808.3557624
  * Ge et al. (2021) Yingqiang Ge, Shuchang Liu, Ruoyuan Gao, Yikun Xian, Yunqi Li, Xiangyu Zhao, Changhua Pei, Fei Sun, Junfeng Ge, Wenwu Ou, and Yongfeng Zhang. 2021.  Towards Long-term Fairness in Recommendation. In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_. 445–453.
  * Ge et al. (2022) Yingqiang Ge, Xiaoting Zhao, Lucia Yu, Saurabh Paul, Diane Hu, Chu-Cheng Hsieh, and Yongfeng Zhang. 2022.  Toward Pareto efficient fairness-utility trade-off in recommendation through reinforcement learning. In _Proceedings of the fifteenth ACM international conference on web search and data mining_. 316–324.
  * Geng et al. (2022) Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022.  Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In _Proceedings of the 16th ACM Conference on Recommender Systems_. 299–315.
  * Harper (2015) F Maxwell Harper. 2015\.  The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5 4 (2015) 1–19. F Maxwell Harper and Joseph A Konstan. 2015. The movielens datasets: History and context. Acm transactions on interactive intelligent systems (tiis) 5 4 (2015) 1–19.
  * Ie et al. (2019a) Eugene Ie, Vihan Jain, Jing Wang, Sanmit Narvekar, Ritesh Agarwal, Rui Wu, Heng-Tze Cheng, Tushar Chandra, and Craig Boutilier. 2019a.  SlateQ: A tractable decomposition for reinforcement learning with recommendation sets.  (2019).
  * Ie et al. (2019b) Eugene Ie, Chih wei Hsu, Martin Mladenov, Vihan Jain, Sanmit Narvekar, Jing Wang, Rui Wu, and Craig Boutilier. 2019b.  RecSim: A Configurable Simulation Platform for Recommender Systems.  (2019).  arXiv:1909.04847 [cs.LG]
  * Jannach and Ludewig (2017) Dietmar Jannach and Malte Ludewig. 2017.  When recurrent neural networks meet the neighborhood for session-based recommendation. In _Proceedings of the eleventh ACM conference on recommender systems_. 306–310.
  * Ji et al. (2023) Jianchao Ji, Zelong Li, Shuyuan Xu, Max Xiong, Juntao Tan, Yingqiang Ge, Hao Wang, and Yongfeng Zhang. 2023\.  Counterfactual Collaborative Reasoning. In _Proceedings of the 16th ACM International Conference on Web Search and Data Mining_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE international conference on data mining (ICDM)_. IEEE, 197–206.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Koren et al. (2022) Yehuda Koren, Steffen Rendle, and Robert Bell. 2022\.  Advances in collaborative filtering.  _Recommender systems handbook_ (2022), 91–142.
  * Li et al. (2022) Zelong Li, Jianchao Ji, Yingqiang Ge, and Yongfeng Zhang. 2022\.  AutoLossGen: Automatic Loss Function Generation for Recommender Systems. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1304–1315.
  * Liang et al. (2018) Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018.  Variational autoencoders for collaborative filtering. In _Proceedings of the 2018 world wide web conference_. 689–698.
  * Lillicrap et al. (2016) Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2016.  Continuous control with deep reinforcement learning.. In _ICLR (Poster)_.  http://arxiv.org/abs/1509.02971
  * Liu et al. (2020) Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, Yuzhou Zhang, and Xiuqiang He. 2020.  State representation modeling for deep reinforcement learning based recommendation.  _Knowledge-Based Systems_ 205 (2020), 106170.
  * Liu et al. (2009) Tie-Yan Liu et al. 2009\.  Learning to rank for information retrieval.  _Foundations and Trends in Information Retrieval_ 3, 3 (2009), 225–331.
  * Mahmood and Ricci (2007) Tariq Mahmood and Francesco Ricci. 2007.  Learning and adaptivity in interactive recommender systems. In _Proceedings of the ninth international conference on Electronic commerce_. 75–84.
  * Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016.  Asynchronous methods for deep reinforcement learning. In _International conference on machine learning_. PMLR, 1928–1937.
  * Pei et al. (2019) Changhua Pei, Xinru Yang, Qing Cui, Xiao Lin, Fei Sun, Peng Jiang, Wenwu Ou, and Yongfeng Zhang. 2019.  Value-aware recommendation based on reinforcement profit maximization. In _The World Wide Web Conference_. 3123–3129.
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010\.  Factorizing personalized markov chains for next-basket recommendation. In _Proceedings of the 19th international conference on World wide web_. 811–820.
  * Shani et al. (2005) Guy Shani, David Heckerman, Ronen I Brafman, and Craig Boutilier. 2005.  An MDP-based recommender system.  _Journal of Machine Learning Research_ 6, 9 (2005).
  * Shi et al. (2020) Shaoyun Shi, Hanxiong Chen, Weizhi Ma, Jiaxin Mao, Min Zhang, and Yongfeng Zhang. 2020\.  Neural logic reasoning. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 1365–1374.
  * Srivastava et al. (2017) Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. 2017\.  Veegan: Reducing mode collapse in gans using implicit variational learning.  _Advances in neural information processing systems_ 30 (2017).
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 1441–1450.
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Taghipour et al. (2007) Nima Taghipour, Ahmad Kardan, and Saeed Shiry Ghidary. 2007\.  Usage-based web recommendations: a reinforcement learning approach. In _Proceedings of the 2007 ACM conference on Recommender systems_. 113–120.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 565–573.
  * Wang et al. (2021b) Kai Wang, Zhene Zou, Yue Shang, Qilin Deng, Minghao Zhao, Runze Wu, Xudong Shen, Tangjie Lyu, and Changjie Fan. 2021b.  RL4RS: A Real-World Benchmark for Reinforcement Learning based Recommender System.  _ArXiv_ abs/2110.11073 (2021).
  * Wang et al. (2021a) Shoujin Wang, Longbing Cao, Yan Wang, Quan Z Sheng, Mehmet A Orgun, and Defu Lian. 2021a.  A survey on session-based recommender systems.  _ACM Computing Surveys (CSUR)_ 54, 7 (2021), 1–38.
  * Wu et al. (2019) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019\.  Session-based recommendation with graph neural networks. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 346–353.
  * Xian et al. (2019) Yikun Xian, Zuohui Fu, Shan Muthukrishnan, Gerard De Melo, and Yongfeng Zhang. 2019.  Reinforcement knowledge graph reasoning for explainable recommendation. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 285–294.
  * Xin et al. (2020) Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020.  Self-supervised reinforcement learning for recommender systems. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 931–940.
  * Xin et al. (2022) Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M. Jose. 2022.  Supervised Advantage Actor-Critic for Recommender Systems _(WSDM ’22)_. Association for Computing Machinery, New York, NY, USA, 1186–1196.  https://doi.org/10.1145/3488560.3498494
  * Yi et al. (2019) Xinyang Yi, Ji Yang, Lichan Hong, Derek Zhiyuan Cheng, Lukasz Heldt, Aditee Kumthekar, Zhe Zhao, Li Wei, and Ed Chi. 2019.  Sampling-bias-corrected neural modeling for large corpus item recommendations. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 269–277.
  * Zhao et al. (2021a) Xiangyu Zhao, Changsheng Gu, Haoshenglun Zhang, Xiwang Yang, Xiaobing Liu, Hui Liu, and Jiliang Tang. 2021a.  DEAR: Deep Reinforcement Learning for Online Advertising Impression in Recommender Systems. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 750–758.
  * Zhao et al. (2019) Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin. 2019\.  ” Deep reinforcement learning for search, recommendation, and online advertising: a survey” by Xiangyu Zhao, Long Xia, Jiliang Tang, and Dawei Yin with Martin Vesely as coordinator.  _ACM sigweb newsletter_ Spring (2019), 1–15.
  * Zhao et al. (2018a) Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018a.  Deep reinforcement learning for page-wise recommendations. In _Proceedings of the 12th ACM Conference on Recommender Systems_. 95–103.
  * Zhao et al. (2020) Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2020\.  Whole-chain recommendations. In _Proceedings of the 29th ACM international conference on information & knowledge management_. 1883–1891.
  * Zhao et al. (2021b) Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2021b.  UserSim: User Simulation via Supervised GenerativeAdversarial Network. In _Proceedings of the Web Conference 2021_. 3582–3589.
  * Zhao et al. (2018b) Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, and Dawei Yin. 2018b.  Recommendations with negative feedback via pairwise deep reinforcement learning. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1040–1048.

##  Appendix A Experiment Setup

###  A.1. Dataset Processing

For the ML1M dataset, we treat movies with user ratings higher than 3 as
positive samples (representing a “like”) and others as negative samples. For
the KuaiRand(Gao et al., 2022) dataset, we first apply 50-core filtering where
videos with less than 50 occurrences are removed. Videos with watch time ratio
larger than 0.8 are regarded as positive samples (representing a “valid view”
or a “comsumption”). For both ML1M and KuaiRand datasets, we segment each user
session into a sequence of item lists of length 10 chronologically. For each
segmented list, only the positive samples before this list are viewed as the
historical behavior. So each record in the final dataset consists of user id,
history behavior sequence, item lists and label lists, which is in the same
format as the RL4RS dataset.

###  A.2. Online Environment

The environment is built based on a user response model that consists of an
item embedding layer, an attention-based history encoder, and a scoring module
for item in actions. The embedding layer is responsible for converting sparse
features into dense features and it is shared by the later history encoder and
action item scoring module. The history encoder layer consists of two multi-
head-attention and output a user embedding. The first network encodes the
sequence information in the history items, and the second layer encodes both
the sequence information from previous layer and the user information. The
final scoring module calculate the dot product between the target/action item
embedding and the user embedding, and we adopt binary cross-entropy loss
during training. As we have mentioned in section 3.1, the user’s feedback is
sampled according to these final scores. After receiving a recommendation list
and responding with the user’s feedback, the positive items are used to update
the user’s history while the static features stay the same. We design the
reward $r(s_{t},a_{t})$ as the average of the item-wise reward, where a
positive response gives a reward of 1 and a negative response gives a
reward/cost of -0.2.

We also construct a temper-based user leave model to give the end-of-session
signal $d$, and the user leaves the system when the system used up the user’s
patience so there is no remaining temper to engage another round of
interaction. The user leave model maintains a users budget of temper, and the
budget decreases whenever a user received a recommendation list. The magnitude
of temper budget reduction is determined by the quality of recommendation, and
worse recommendation induces quicker temper drops. Additionally, we restrict
the maximum session depths to 20 for all environments.

###  A.3. Model Specifications

Method | Actor Learning | Critic Learning | Supervised Learning | Action Alignment
---|---|---|---|---
SL | None | None | BCE | None
A2C |  Policy gradient on $P(a|s)$ with $V(s)$ |  TD error on $V(s)$ | None | None
DDPG |  $\max Q(s,Z)$ with $P(Z|s)$ |  TD error on $Q(s,Z)$ | None | None
TD3 |  $\max\min(Q_{1}(s,Z),Q_{2}(s,Z))$ with $P(Z|s)$ |  TD error on $\min(Q_{1}(s,Z),Q_{2}(s,Z))$ | None | None
DDPG-RA |  $\max Q(s,Z)$ with $P(Z|s)$ |  TD error on $Q(s,Z)$ | None | $-\mathbb{E}\Big{[}\ln(P(a_{t}|s_{t},s_{t+1}))\Big{]}$
HAC |  $\max Q(s,Z)$ with $P(Z|s)$ |  TD error on $Q(s,a)$ | Optional |  Eq.(9)
Table 3. Method Component Comparison

In this part we present the architectures and hyperparameters of models. Our
code is implemented in PyTorch and we provide source code in
https://github.com/CharlieMat/Hyper-Actor-Critic-for-Recommendation.

SASRec Backbone Policy:  This module is shared across all methods in Table 3.
For any given item embedding (appeared in the history sequence, the exposure,
and the candidate item pool), an item encoder is used to map it to a
32-dimensional vector. Then the encoded vectors of the user history is added
with a trainable positional embedding and concatenated with the encoded user
vector (by a 2-layer with hidden dim 128). The final input sequence is then
input into a 2-layer transformer encoder with 4 heads and dropout rate 0.1.
The last embedding in the output sequence is regarded as the encoded user
state. Finally, we use a linear mapping to generate the hyper-action from the
user state. And the final effect-action is selected based on the dot-product
ranking score as described in section 2.4.

Critic:  the critic is a MLP with (encoded user state vector, hyper-action
vector) as input, two hidden layers with dimension 256 and 64, and outputs a
scalar representing the Q value. Differently, in A2C, a value network is used
which has the same hidden dimensions but only takes the user state as input.

Offline Supervise:  we use the offline training set to optimize the policy
through binary cross entropy loss Eq.(10) based on the observed exposure (a
list of recommend items) and the corresponding user feedback. We adopt batch
size 64, and searched the learning rate in [0.0005, 0.0001, 0.00005, 0.00001]
and L2 regularization with coefficient in [0.0001, 0.00005, 0.00001,
0.000005]. When evaluating the online performance during training, we consider
apply mini-batch training for each interaction step.

Online Supervise:  the online version of SL differs from the offline SL by
directly using the sampled trajectories collected by online interactions with
users rather than mini-batches from offline datasets.

Figure 11. Controlling Actor Learning Rate of HAC, X-axis represents learning
rate of $\mathcal{L}_{\mathrm{QMax}}$. Figure 12. Supervision learning rate of
Eq.(10) on ML1M and RL4RS datasets. Figure 13. Effect of hyper-action
alignment on RL4RS online environment. X-axis represents the magnitude of
Eq.(9).

For all the following RL-based methods, we use a replay buffer with size
100,000 and starts mini-batch training when the buffer has more than 2,000
samples. Each sample in the buffer consists of (user static features, current
history, next history, hyper-action embedding, effect-action the
recommendation list, user response, immediate reward, and the “done” signal).
And we simultaneously run 32 user sessions/episodes so each interaction step
will sample 32 records into the buffer. The current user state combines (user
static features, current history) and the next user state combines (user
static features, current history). The common search spaces of hyperparameters
are [0.0005, 0.0001, 0.00005, 0.00001, 0.000005, 0.000001] for actor learning
rate, [0.001, 0.0001, 0.00001] for critic learning rate, and variance
$\sigma\in[0.1,0.01,0.001]$ for the Gausian hyper-action exploration. In
addition to the model differences mentioned in Table 3, we summarize the
experiment settings as the following:

A2C:  this method combines the aforementioned SASRec backbone and the value
(critic) network. uses the policy gradient that is guided by the advantage
score of the value. The exploration of A2C does not explore on hyper-actions
so has noise variance $\sigma=0$, and the action exploration is engaged based
on categorical sampling on the effect-action probabilities.

DDPG:  this model uses the Q-network for critic and adopt Gausian sampling on
the hyper-action space.

DDPG-RA:  this model is the DDPG model with an additional effect-action
alignment module as illustrated in Table 3. For the alignment module, we
search the learning rate in [0.00005, 0.00001, 0.000005, 0.000001].

HAC:  it uses the effect-action space for critic learning and hyper-action
space for actor learning. As a result, we can apply exploration on both action
spaces. Compared to DDPG and supervised learning, this method has an
additional action alignment module where we search the magnitude in
$\lambda_{h}\in[0.001,0.01,0.1,1]$ as described in section 3.3.

##  Appendix B Auxiliary Experimental Results

Figure 14. HAC model ablation results on KuaiRand online environment

We present comparison results of actor learning rates of HAC in Figure 11. We
present comparison results of different supervision learning rates of HAC in
Figure 12 as the augmentation of experimental results in section 3.3. We
present comparison of different model ablation of HAC in Figure 14 as the
augmentation of experimental results on ablation study in section 3.4. All
results are consistent with the results in the main experimental results which
further verifies our claims.