# Augmenting Recurrent Neural Networks with High-Order User-Contextual
Preference for Session-Based Recommendation

Younghun Song  KAIST291 Daehak-ro, Yuseong-guDaejeonKorea, Republic of34141
<EMAIL_ADDRESS>and  Jae-Gil Lee  KAIST291 Daehak-ro, Yuseong-
guDaejeonKorea, Republic of34141<EMAIL_ADDRESS>
###### Abstract.

The recent adoption of recurrent neural networks (RNNs) for session modeling
has yielded substantial performance gains compared to previous approaches. In
terms of context-aware session modeling, however, the existing RNN-based
models are limited in that they are _not_ designed to explicitly model rich
static user-side contexts (e.g., age, gender, location). Therefore, in this
paper, we explore the utility of explicit user-side context modeling for RNN
session models. Specifically, we propose an _augmented RNN (ARNN)_ model that
extracts _high-order user-contextual preference_ using the product-based
neural network (PNN) in order to augment any existing RNN session model.
Evaluation results show that our proposed model outperforms the baseline RNN
session model by a large margin when rich user-side contexts are available.

Session-based Recommendation, Recurrent Neural Networks, Context-aware
Recommendation, Sequential User Modeling

††copyright: none††ccs: Information systems Recommender systems††ccs:
Information systems Data stream mining

##  1\. Introduction

Making recommendations based on session logs of user-item interactions has
been a major challenge for the recommender systems community. Utilizing
session logs for recommendations has obvious advantages: 1) they can be used
to infer user preferences to make recommendations to anonymous/fresh users,
and 2) they can provide a more personalized recommendation that matches a
user’s current interest.

Since the pioneering work of Hidasi et al.(Hidasi et al., 2016a), recurrent
neural networks (RNNs) have been the de-facto choice for modeling session-
based recommender systems, as they are capable of effectively exploiting the
sequential nature of user session data. Inspired by this success, numerous
RNN-based session models have been proposed to accommodate diverse aspects of
session-based recommender systems—e.g., by incorporating item content
features(Hidasi et al., 2016b), modeling latent user intent(Loyola et al.,
2017), or merging information from past and current user sessions(Ruocco et
al., 2017; Quadrana et al., 2017).

Another interesting application of RNNs has been their usage in context-aware
sequential recommendations(Smirnova and Vasile, 2017; Liu et al., 2016), which
can be also applied to session-based recommendation settings. However, a
majority of the research in this line focused on exploiting _dynamic_ temporal
contexts(e.g., time-of-the-day), and there have been few attempts that
explicitly consider the static user-side contexts(e.g., age, gender,
location), which are readily available in commercial systems. In terms of
session modeling, explicit modeling of static user contexts is important
because _user navigation paths_ of items during sessions are dependent on
static contexts, due to the fact that users with different static contexts
have different preferences with respect to the same items. For example, when
recommended a job position in Paris during a session, a user living in New
York will tend to look for alternative positions in New York in the following
searches, while a user living in Paris will tend to look for the similar
positions.

Therefore, in order to seamlessly integrate static user-side contexts into RNN
session models, we propose an _augmented RNN (ARNN)_ , which is an augmented
version of RNN that can improve on any existing RNN session model. The unique
feature of ARNN is that it estimates _user-contextual preference_ by modeling
high-order interaction between static user context and the previous item using
a product-based neural network (PNN)(Qu et al., 2016), which is a neural
network (NN) variant of the factorization machine (FM)(Rendle, 2010). By
integrating this contextual preference with the hidden states of an RNN
session model, the ARNN makes more personalized recommendations than the plain
RNN session models that does not consider user contexts.

We evaluated the ARNN on two datasets: a job recommendation dataset and a
standard e-commerce dataset. Through an experiment, we verified that the ARNN
successfully captured user-contextual preference when abundant user contexts
were available, which led to a substantial improvement in the recommendation
quality against an RNN baseline.

##  2\. Related Works

###  2.1. Session-based / Context-aware Recommendation with RNNs

Traditional session-based recommendation algorithms have largely been based on
item-to-item approaches(Sarwar et al., 2001; Linden et al., 2003). However,
the recent success of deep learning has led to the adoption of recurrent
neural networks (RNNs) to session-based recommendations, starting from
GRU4REC(Hidasi et al., 2016a). The motivation of GRU4REC was to mitigate the
consistently-cold-starting problem that occurs frequently in a typical
e-commerce environment, where most of the customers do not log in and thus are
anonymous. After GRU4REC, several RNN-based models were introduced to consider
various aspects of session-based recommendations (Loyola et al., 2017; Ruocco
et al., 2017; Quadrana et al., 2017; Hidasi et al., 2016b).

RNNs also started to gain momentum in the field of context-aware sequential
modeling. Some approaches used temporal contexts like time-of-the-day or time
difference between the previous and current interaction(Smirnova and Vasile,
2017; Liu et al., 2016). There was also an attempt to use the type of
interaction events as a contextual information(Smirnova and Vasile, 2017).
However, all those contexts were _dynamic_ contexts dependent on specific
transaction instances, and no models have been introduced that can equip the
powerful RNN session models with abundant static user-side contexts like age,
location, or current job title.111For convenience, we’ll use the term “user
context” to refer to “static user context” throughout this paper.

###  2.2. PNN and FM

In terms of context-aware recommendations, the factorization machine
(FM)(Rendle, 2010) and its neural network (NN) variants(Qu et al., 2016; He
and Chua, 2017) have been widely successful, particularly in applications such
as click-through-rate (CTR) prediction. The FM and its variants build the
latent embeddings for the categorical contexts in order to estimate the 2nd-
order interactions between categorical features, thereby aiming to resolve the
data sparsity inherent in recommender systems. Among the NN variants of FM,
the product-based neural network (PNN)(Qu et al., 2016) has been known to
effectively capture the higher-order interactions between the categorical
variables by applying pairwise product operations for each possible pair of
embedded categorical fields and passing the resulting vector as the input to a
NN.

##  3\. Model Description

###  3.1. Problem Setup

The goal of our work is to capture the contribution of user context
information in determining a user’s item navigation path during an online
session. Therefore, our dataset can be expressed as follows:

  * •

Session: $s_{i}=\\{(x_{t},y_{t})\\}_{t=1}^{T_{i}}$, where $y_{t}$ is the item
that the user interacted with at $t$ within the session, and $x_{t}$ is the
user contextual vector.

  * •

Training set:
$\mathcal{D}_{train}=\\{s_{1},\dots,s_{|\mathcal{D}_{train}|}\\}$

  * •

Test set:
$\mathcal{D}_{test}=\\{s^{\prime}_{1},\dots,s^{\prime}_{|\mathcal{D}_{test}|}\\}$

Notice that $x_{t}$ is included in every user session. $x_{t}$, a user
contextual vector, is a concatenation of the one-hot encodings for each
categorical fields, as in Rendle(Rendle, 2010). For instance, if we have the
context information (Gender=Female,Location=U.S.) about a user, then each
field are first encoded separately as $[1,0],[0,0,1,0,\dots,0]$ and
concatenated to produce the final input $x_{t}=[1,0,0,0,1,0,\dots,0]$. The
objective of the ARNN is, given a new sequence of (context,item) pairs in a
user session $s^{\prime}_{u}=\\{(x_{1},y_{1}),\dots,(x_{t-1},y_{t-1})\\}$
included in a test set, to predict the next item $y_{t}$ with which the user
will interact.

Given the dataset, our problem reduces to a sequential ranking task with
implicit feedback, where the input consists of the item indices $y_{t}$’s and
user contexts $x_{t}$’s, and the output consists of the ranking scores for the
positive items $r(y_{t})$ and negative items $r(y_{n})$. Now, following the
approach by Rendle et al.(Rendle et al., 2009) and introducing the time-
independence assumption for user contexts, we can set up the following joint
likelihood for a single session as the objective function:

(1)
$\mathcal{L}=\prod_{t=1}^{T}\prod_{n=1}^{N}{p(r(y_{t})>r(y_{n})|\mathbf{y}_{<t},x_{t-1})}$

where $\mathbf{y}_{<t}=(y_{t-1},\dots,y_{1})$.

###  3.2. Model Overview

The proposed model, which we call the augmented RNN (ARNN) model, augments an
RNN session model with a PNN context encoder. At the training stage, a PNN
context encoder and an RNN session model are first pretrained separately to
optimize the likelihood (1), where the negative items are defined using the
items in the session-parallel mini-batches as in GRU4REC(Hidasi et al.,
2016a). Then, a merging layer is trained on top of the pretrained PNN and RNN
to integrate the information from both networks. After training, the ARNN
scores the items conditioning on both the PNN-produced contextual preference
$c_{t}$ and the RNN hidden state $h_{t}$. The details of the PNN/RNN
components and the merging network are explained in the following sections.

###  3.3. PNN Context Encoder

The foundational element of the ARNN is the PNN context encoder that models
user-contextual preference by capturing high-order interactions between user
contexts and previous items. Similar to the original PNN(Qu et al., 2016), the
PNN context encoder is composed of three layers: embedding layer, product
layer, and FC layer, as shown in Figure 1.

Figure 1. Architecture of the PNN context encoder.

First, $x_{t-1}$ (as explained in Section 3.1) and the previous item $y_{t-1}$
are concatenated and enters the embedding layer, producing the embeddings for
each fields(including the previous item). As the original PNN, only one
position remains active per field.

Next, pairwise inner product operations are applied to each possible pair of
field embeddings to produce the pairwise signal $p_{t}$. At the same time, all
field embeddings are flattened and concatenated to produce the linear signal
$z_{t}$.

Then, $p_{t}$ and $z_{t}$ are concatenated and fed as an input to the FC
layer, followed by a rectified linear unit (ReLU) + batch normalization (Ioffe
and Szegedy, 2015). From this nonlinear transformation of both linear/pairwise
signals, the PNN can effectively model high-order interactions between the
user contexts and the previous item, obtaining a vector encoding the user-
contextual preference.

Finally, after the batch normalization layer, the user-contextual preference
$c_{t}$ goes through the softmax layer that calculates the final item scores.
The item scores are then used to calculate the ranking loss and to pretrain
the PNN, as mentioned in Section 3.2.

###  3.4. RNN Session Model

For simplicity, RNN session models are set to follow the simple gated
recurrent unit (GRU)(Cho et al., 2014) architecture proposed in GRU4REC(Hidasi
et al., 2016a). More specifically, let $y_{t}$ be the one-hot encoded item for
the $t$-th transaction and $h_{t}$ be the corresponding GRU hidden state. The
scores for the next items are then calculated by the GRU as follows:

$\displaystyle\hat{r}(y_{t}|\mathbf{y}_{<t})=softmax(h_{t})\quad
h_{t}=GRU(y_{t-1},h_{t-1})$

Similar to Section 3.3, the item scores are then used to pretrain the GRU for
the session-based ranking task. However, our model does not restrict the RNN
session model to take the proposed simple GRU form, so _any_ existing RNN
session models can be used instead of the GRU.

###  3.5. Augmenting the RNN with the PNN

Figure 2. Overall architecture of the ARNN model.

Figure 2 shows the overall architecture of the ARNN which consists of three
types of layers: the input layer, the intermediate layer, and the merging
layer. The input layer is simply where the model receives its input data. The
intermediate layer consists of the PNN/RNN that we pretrained(as decribed in
Sections 3.3 and 3.4). To use the pretrained PNN/RNN as _feature extractors_ ,
we remove the final softmax layers from both the PNN/RNN, and freeze the
parameters so that they are not further updated222However, we found that
retraining only the batch normalization layers for the PNN slightly increases
the model performance.. The merging layer $M$ is where the PNN output $c_{t}$
and the RNN hidden state $h_{t}$ are combined and used to calculate the item
scores as follows:

$\displaystyle\hat{r}(y_{t}|\mathbf{y}_{<t},x_{t-1})=softmax(M([c_{t};h_{t}]))$
$\displaystyle c_{t}=PNN(x_{t-1},y_{t-1})\quad h_{t}=GRU(y_{t-1},h_{t-1})$

In our model, $M$ is a simple FC layer followed by a ReLU + batch
normalization layer.

##  4\. Experiments

###  4.1. Dataset

To test our algorithm, we used two datasets: the XING dataset from the RecSys
2016 Challenge and the TMALL dataset from the IJCAI-15 competition. Table 1
presents the profiles of the two datasets. The XING dataset contains the user-
item interaction logs from xing.com, a social networking service specialized
for job searching. Rich user-side categorical contexts such as job roles and
career levels are included in the XING dataset. Among those contexts, we
picked only 12 attributes (e.g., job roles, career level, and country/region)
that are likely to be useful for predicting job preferences. By contrast, the
TMALL dataset is a standard e-commerce dataset in which only three types of
user-side categorical information (user ID, age, and gender) are available. We
thus used all three attributes. The purpose of employing the TMALL dataset was
to examine how the degree of abundance in user-side contexts impact the
performance of the proposed context-augmented model.

Table 1. Dataset statistics. | XING | TMALL
---|---|---
# of users | 769K | 424K
# of items | 1M | 1.1M
# of sessions | 2.1M | 6.5M
# of transactions | 7.2M | 48.6M
# of user-side categorical contexts used | 12 | 3

###  4.2. Preprocessing

####  4.2.1. Train/test split

For both datasets, we extracted the last three days for testing purposes and
trained on the preceding 27 days. Transactions including the items not in the
training set were ignored as in Hidasi et al.(Hidasi et al., 2016a).

####  4.2.2. Session ID marking

Since session IDs were missing in both XING and TMALL, we manually set time
thresholds for marking a sequence of transactions as a session. The threshold
was 1 hour for XING, and 1 day for TMALL (as in Jannach et al.(Jannach and
Ludewig, 2017)).

####  4.2.3. Sampling on the items

Due to scalability issues, only the top items that covered 50% of the
transactions were selected for modeling. In other words, the items were sorted
by popularity and a minimum item popularity threshold was set such that the
transactions including only the items above the threshold accounted for 50% of
the entire dataset. After this sampling, we were left with 13,259 items for
XING and 11,939 items for TMALL.

####  4.2.4. Encoding the categorical attributes

All categorical features were encoded as binary features that indicated the
presence of the corresponding features. For XING, there were multi-valued
categorical features that held numerous values (e.g., jobroles=[350, 891]).
Making binary features for all the multi-valued features resulted in
prohibitively high-dimensional input to the PNN, which incurred large
computational cost. Thus, similar to the process we described in Section
4.2.3, we created binary features for only the most popular categories that
covered 75% of the transactions. The least-popular 25% of the features were
mapped to a single ”unknown” attribute. After this encoding, the total number
of input fields required for the PNN was 368 for XING and 4 for TMALL,
including the field kept for embedding the previous items.

###  4.3. Optimization and Hyperparameters

####  4.3.1. Loss Function

The TOP1 loss(Hidasi et al., 2016a) was used as a ranking loss to maximize the
objective (1). Let $N_{s}$ be the number of negative samples in a session-
parallel mini-batch. In addition, let $\hat{r}_{s,i}$ be the calculated score
for the $i$-th item in the session $s$. Then, the TOP1 loss is defined as:

$TOP1_{s,i}=\frac{1}{N_{s}}\sum_{j=1}^{N_{s}}{\left(\sigma(\hat{r}_{s,j}-\hat{r}_{s,i})+\sigma(\hat{r}_{s,j}^{2})\right)}$

This loss function is intended to push the scores of the negative items to
zero, which prevents the overall item scores from exploding during
optimization.

####  4.3.2. Hyperparameters and Optimizer

Here, we provide information about the core hyperparameters and optimizer. The
hyperparameters for the GRU4REC follows that of Quadrana et al.(Quadrana et
al., 2017) for XING, and Jannach et al.(Jannach and Ludewig, 2017) for TMALL.

  * •

GRU4REC

    * –

GRU hidden size: 100 (XING), 1000 (TMALL)

    * –

Dropout: 0.2 (XING), None (TMALL)

  * •

PNN

    * –

Field embedding dimension: 10 (XING, TMALL)

    * –

FC layer hidden size: 100 (XING), 300 (TMALL)

    * –

Dropout: None (XING, TMALL)

  * •

ARNN

    * –

Merging layer hidden size: 100 (XING), 1000 (TMALL)

    * –

Dropout: None (XING, TMALL)

As an optimizer, we used Adagrad(Duchi et al., 2011) with different learning
rates and weight decays for each dataset and model. It turned out that finding
the right optimizer hyperparameters was crucial to the performance of the
model.333For more details, please visit our Github repository. The repository
will go public after the review process.

###  4.4. Evaluation

####  4.4.1. Measures

We employed two ranking measures to evaluate the model performance: Recall@K
and MRR@K. Let $N_{hits}$ be the number of times that a user chose an item
from the recommendation list and $N_{recs}$ be the number of top-K
recommendation attempts. Then, the measures can be calculated as follows:

$Recall@K=\frac{N_{hits}}{N_{recs}}\qquad
MRR@K=\frac{1}{N_{recs}}\sum_{n=1}^{N_{recs}}{\frac{1}{rank(n)}}$

where $rank(n)$ denotes the rank of the chosen item within the $n$-th top-K
recommendation list.

####  4.4.2. Baselines

We compared the ARNN with the following baselines:

  * •

Item-KNN(Linden et al., 2003): A simple yet powerful item-to-item approach
that recommends items that are similar to the previous item based on cosine
similarity.

  * •

GRU4REC(Hidasi et al., 2016a): A widely used RNN-based session model, as
described in Section 2.1. Unlike ARNN, GRU4REC uses only the information from
the item IDs.

  * •

PNN(Qu et al., 2016): A PNN context encoder, which is included to assess how
well it captures high-order context information relevant for ranking.

##  5\. Results

Table 2 shows the Recall@20 and MRR@20 of the ARNN and the baselines for the
two datasets.

Table 2. Evaluation results. | XING | TMALL
---|---|---
| Recall@20 | MRR@20 | Recall@20 | MRR@20
ItemKNN | 0.2235 | 0.0932 | 0.1541 | 0.0614
GRU4REC | 0.2597 | 0.1169 | 0.3789 | 0.1886
PNN | 0.1439 | 0.0507 | 0.1755 | 0.0694
ARNN | 0.3106 | 0.1252 | 0.3812 | 0.1953

Evaluation of our model on XING provided us with a clear evidence that
capturing user-contextual preference using a PNN is indeed helpful for RNN
session models when rich user-side context is available. Although the
performance of the PNN context encoder in itself was poor compared to GRU4REC,
ARNN as a whole outperformed both ItemKNN and GRU4REC in both the Recall@20
and MRR@20 measures.

However, the performance of the ARNN was only _marginally_ better than the
GRU4REC baseline in case of TMALL. Considering that only three user context
variables were available for TMALL(user id, age, gender), our hypothesis is
that the number of user context fields was not enough to augment new piece of
information that could not be inferred by the GRU4REC. Thus, we recommend that
ARNN should be used only when the number of user-side categorical fields is
sufficiently large.

##  6\. Discussion and Conclusions

In this study, we proposed an augmented RNN model that can easily boost an
existing RNN session model by estimating high-order user-contextual preference
using the PNN. Since PNN context encoders can handle arbitrary user-side
contextual information and build upon an existing pretrained RNN session
model, we believe that deploying our model for real-world systems would be a
handy solution that can improve the recommendation quality of a system without
considerable effort.

However, one limitation of the ARNN is that it ignores the item-side
contexts(e.g., item metadata, item content), unlike the usual factorization
machines or PNNs used for CTR prediction. We omitted the item-side contexts
because our intention was to measure the effect of incorporating _pure_ user-
side contexts(e.g., age, location, login platform) with existing RNN session
models using a FM-like approach. Therefore, designing an ARNN architecture
that can also handle item-side contexts would be an interesting research topic
for a future work.

## References

  * (1)
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In _Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. pages 1724–1734.
  * Duchi et al. (2011) John Duchi, Elad Hazan, and Yoram Singer. 2011.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.  _Journal of Machine Learning Research_ vol.12 (2011), pages 2121–2159.
  * He and Chua (2017) Xiangnan He and Tat-Seng Chua. 2017.  Neural Factorization Machines for Sparse Predictive Analytics. In _Proceedings of the 40th International ACM Special Interest Group on Information Retrieval(SIGIR)_. pages 355–364.
  * Hidasi et al. (2016a) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016a.  Session-based Recommendations with Recurrent Neural Networks. In _Proceedings of the International Conference on Learning Representations(ICLR)_.
  * Hidasi et al. (2016b) Balázs Hidasi, Massimo Quadrana, Alexandros Karatzoglou, and Domonkos Tikk. 2016b.  Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations. In _Proceedings of the 10th ACM Conference on Recommender Systems(RecSys)_.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In _Proceedings of the 32nd International Conference on Machine Learning_.
  * Jannach and Ludewig (2017) Dietmar Jannach and Malte Ludewig. 2017.  When Recurrent Neural Networks meet the Neighborhood for Session-Based Recommendation. In _Proceedings of the 11th ACM Conference on Recommender Systems(RecSys)_. pages 306–310.
  * Linden et al. (2003) Greg Linden, Brent Smith, and Jeremy York. 2003.  Amazon.com Recommendations: Item-to-Item Collaborative Filtering.  _Internet Computing, IEEE_ 7 (2003), 76–80.
  * Liu et al. (2016) Qiang Liu, Shu Wu, Diyi Wang, Zhaokang Li, and Liang Wang. 2016.  Context-aware Sequential Recommendation. In _Proceedings of the IEEE International Conference on Data Mining (ICDM)_. pages 1053–1058.
  * Loyola et al. (2017) Pablo Loyola, Chen Liu, and Yu Hirate. 2017.  Modeling User Session and Intent with an Attention-based Encoder-Decoder Architecture. In _Proceedings of the Eleventh ACM Conference on Recommender Systems(RecSys)_. pages 147–151.
  * Qu et al. (2016) Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, and Jun Wang. 2016.  Product-based Neural Networks for User Response Prediction. In _Proceedings of the 16th International Conference on Data Mining(ICDM)_. pages 1149–1154.
  * Quadrana et al. (2017) Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017.  Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks. In _Proceedings of the 11th ACM Conference on Recommender Systems(RecSys)_. pages 130–137.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization Machines. In _Proceedings of the 2010 International Conference on Data Mining(ICDM)_. pages 995–1000.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback. In _Proceedings of the 25th Conference on Uncertainty in Artificial Intelligence(UAI)_. pages 452–461.
  * Ruocco et al. (2017) Massimiliano Ruocco, Ole Steinar Lillestøl Skrede, and Helge Langseth. 2017\.  Inter-Session Modeling for Session-Based Recommendation. In _Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems(DLRS)_. pages 24–31.
  * Sarwar et al. (2001) Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001.  Item-Based Collaborative Filtering Recommendation Algorithms. In _In Proceedings of the 10th international conference on World Wide Web(WWW)_. pages 285–295.
  * Smirnova and Vasile (2017) Elena Smirnova and Flavian Vasile. 2017.  Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks. In _Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems(DLRS)_. pages 2–9.