# Reinforcement Learning based Path Exploration for Sequential Explainable
Recommendation

Yicong Li1, Hongxu Chen1 2, Yile Li, Lin Li, Philip S. Yu, Guandong Xu2
Yicong Li, Hongxu Chen and Guandong Xu are with Data Science and Machine
Intelligence Lab, Faculty of Engineering and Information Technology,
University of Technology Sydney, Sydney, New South Wales, 2007, Australia.
E-mail<EMAIL_ADDRESS>{Hongxu.Chen<EMAIL_ADDRESS>
Yile Li is with School of Transportation Engineering, Tongji University,
Shanghai, 200092, China. E-mail<EMAIL_ADDRESS>
Lin Li is with College of Computer Science and Technology, Wuhan University of
Technology, Wuhan, 430070, China. E-mail<EMAIL_ADDRESS>
Philip S. Yu is with Department of Computer Science, University of Illinois at
Chicago, Chicago, Illinois, 60637, the United States. E-mail<EMAIL_ADDRESS>
* Both authors contributed equally to this research.†Corresponding author.

###### Abstract

Recent advances in path-based explainable recommendation systems have
attracted increasing attention thanks to the rich information provided by
knowledge graphs. Most existing explainable recommendations only utilize
static knowledge graphs and ignore the dynamic user-item evolutions, leading
to less convincing and inaccurate explanations. Although there are some works
that realize that modelling user’s temporal sequential behaviour could boost
the performance and explainability of the recommender systems, most of them
either only focus on modelling user’s sequential interactions within a path or
independently and separately of the recommendation mechanism. In this paper,
we propose a novel Temporal Meta-path Guided Explainable Recommendation
leveraging Reinforcement Learning (TMER-RL), which utilizes reinforcement
item-item path modelling between consecutive items with attention mechanisms
to sequentially model dynamic user-item evolutions on dynamic knowledge graph
for explainable recommendation. Compared with existing works that use heavy
recurrent neural networks to model temporal information, we propose simple but
effective neural networks to capture users’ historical item features and path-
based context to characterize the next purchased item. Extensive evaluations
of TMER on two real-world datasets show state-of-the-art performance compared
against recent strong baselines.

###### Index Terms:

Reinforcement learning, sequential recommendation, meta-path, explanation.

##  1 Introduction

Reasoning with paths over user-item associated Knowledge Graphs (KGs) has been
becoming a popular means for explainable recommendations [1, 2, 3]. The path-
based recommendation systems have successfully achieved promising
recommendation performance, as well as appealing explanations via searching
the connectivity information between users and items across KGs. One category
of existing works on path-based explainable recommendations seeks auxiliary
meta-paths (pre-defined higher-order relational compositions between various
types of entities in KGs) as similarity measures and evidence for possible
explanations between users and items.

However, most of existing path-based methods for explainable recommendation
simply treat the underlying KGs as static graphs, ignoring the dynamic and
evolving nature of user-item interactions in real-world recommendation
scenarios. The dynamics and evolutions of users’ interactions with items play
a pivotal role in both recommendation precision and explanations for a users’
real intent. Take the scenario in Figure 1 as an example, Alice purchased a
phone case and a phone film after a recent buy of a new phone. If we ignore
the sequential information between each purchase (in Fig 1(a)) and treat the
whole information as a static graph (in Fig 1(b)), the system probably gives
an explanation of buying the phone case is because a similar customer also
bought this phone case by exploring co-purchasing relationships. Whilst the
explanation, in this case, might be valid and the underlying reasoning
mechanism (collaborative filtering signal) can be used for recommendation, it
is still sub-optimal as a more appealing motivation of buying a phone case is
due to the recent purchase of a new phone. For this reason, in this example,
it is important for a system to be capable of considering temporal and
sequential user-item interactions and disentangling the importance of various
reasons when generating possible explanations.

(a) Intuitive sequential modelling

(b) Existing static knowledge-graph based modelling

Figure 1: Existing Sequential modelling and static knowledge-graph based
modelling.

Although some prior works have considered some extent of the sequential
information for knowledge-aware path-based explainable recommendation problem,
they still fail to explicitly model the dynamics of users’ activities. To
allow effective reasoning on paths to infer the underlying rationale of a
user-item interaction, the method proposed in [2] takes the advantages of path
connectivity and leverages the sequential dependencies of entities and
sophisticated relations of a path connecting a user-item pair. Nevertheless,
the methods only consider the sequential information within specific paths and
did not consider the importance of the user’ historical sequences on
reflecting the user’s dynamic interactions with items. To improve, the KARN
model [4] fuses the user’s clicked history sequence and path connectivity
between users and items in KGs for recommendation. However, the method models
the user’s sequential behaviours and user-item interactions separately and in
a coarse-grained manner (treating a user’s click history as a whole), which
may restrict the expressiveness of users’ temporal dynamics on recommendation
explainability.

In light of this, in this paper, we challenge the problem of exploring users’
temporal sequential dynamics in the context of path-based knowledge-aware
recommendation. Different from existing works that either only consider the
sequential information within a path or treat the user’s sequential
interactions as a whole and separately, we aim to 1) explicitly model and
integrate the dynamic user-item interactions over time into the path-based
knowledge-aware recommendation, and 2) leverage the captured dynamics of user-
item interactions to improve the performance and explainability of the
recommendation.

It is worth noting that modelling users’ temporal sequential behaviour with
path-based knowledge-aware explainable recommendation is a non-trial and
challenging task. First, path-based knowledge-aware recommender systems are
built upon the well-constructed KGs, and its expected accuracy and
explainability are highly related to the underlying KGs and distilled paths.
If temporal information between users and items is considered, the original
underlying static KGs will be cast into multiple snapshots w.r.t. the
timestamps. Comparing with the static KGs that consist of all users’ full
timelines, each snapshot at a certain timestamp only has partial observations
of the global knowledge, which will result in inferior recommendation
performance. To deal with the issue, we devise a general framework for
temporal knowledge aware reinforcement path-based explainable recommender
systems, namely Temporal Meta-path Guided Explainable Recommendation
leveraging Reinforcement Learning (TMER-RL). TMER-RL provides a solution that
can explicitly depict users’ sequential behaviour while being able to be aware
of global knowledge of the entire underlying KGs. Specifically, to model the
temporal dynamics between users and items, TMER-RL naturally models the task
as a sequential recommendation problem and takes as input users and their
corresponding sequential purchase history. To learn the sequential
dependencies between consecutive items purchased by a user, TMER-RL novelly
explores item-item paths between consecutive items and embed the paths as
context with elaborated designed attention mechanisms to model the dynamics
between user and items. Compared to existing path-based explainable
recommendation systems that only consider user-item paths as evidence and
support for the recommendation decision, TMER-RL contributes another creamy
layer on the top of existing works, which makes use of the powerful
expressiveness of temporal information between users and items.

In addition, when taking paths into consideration for the above purpose,
another challenge lies in the existence of a large number of possible paths.
It is time-consuming for a model to select several paths that are meaningful,
expressive, and have positive impacts on both recommendation performance and
explainability. To this end, inspired by prior work [5], our previous work
TMER [6] leverages the concept of meta-path [7] and explore diverse meta-path
schemas to characterize the context of dynamic interactions between users and
items. However, it has to pre-define meta-paths and randomly sample path
instances from all generated ones, which involves human and random factor in
the following recommendation module. To explore paths effectively, we utilize
reinforcement learning with our designed useful score function for paths to
mine diverse meta-path instances. With resort to the powerful transformer
model for sequential modelling, we also elaborate item-item path attention and
user-item path attention units to learn combinational features of multiple
paths to further characterize users’ temporal purchasing motivations and their
general shopping tastes, respectively. The rationale for developing such path-
based attention units is that a user’s motivation towards buying a certain
product is complex and consists of multiple factors. For example, when buying
a new phone, a customer may consider several factors including a phone’s
intrinsic features such as functionality, display, camera, etc., as well as
other external factors such as the choices of their close friends, and their
previous purchase of certain related electronics using the same operating
system. With help of the designed path-based attention units, the proposed
TMER-RL framework is able to learn different weights for various possible
paths which will be then used as explanations for recommendations (we show the
effectiveness of using the proposed path-based attention units for the
explainable recommendation in the experiments by running case studies).

Another shining point of our work is that the introduced item-item path
modelling also serves as the core of the simple yet effective sequential
modelling architecture of TMER-RL. The combinational meta-path enriched
features learned by the item-item path attention units are aggregated with
features of previous purchased item will serve as the prediction signals for
the next item. The intuition behind is that the item-item paths connect two
consecutive items purchased by a user, and represent various reasons and
factors that may lead to the next purchased item, which intrinsically provides
strong sequential dependencies between items. Compared to existing works on
the sequential recommendations that rely on Recurrent Neural Nets such as
(RNNs[8], GRUs[9], LSTM[2, 4]), the proposed methods in this paper is light,
simple but effective.

The main contributions of this paper are summarized as follows:

  * •

We point out that explicitly modelling dynamic user-item interactions over
time can significantly benefit the recommendation performance and
explainability. We introduce, to the best of our knowledge, the first study to
model users’ temporal sequential behaviour with path-based knowledge-aware
explainable recommendation.

  * •

We propose Temporal Meta-path Guided Explainable Recommendation leveraging
Reinforcement Learning (TMER-RL), which considers users’ dynamic behaviours on
top of the global knowledge graph for sequential-aware recommendation and
explores both user-item and item-item meta-path paths with well-designed
reinforcement framework and attention mechanisms for explainable
recommendation.

  * •

Extensive evaluations on three real-world benchmark datasets have been
conducted, and the experimental results demonstrate the superiority,
effectiveness and temporal explainability of the proposed TMER-RL model.

##  2 Problem Definition

In this section, we first give some essential definitions and define the
problem.

###### Definition 1

Information Networks. An information network is a simple graph
$G=(\mathcal{V},E)$. Each edge $e\in E$ represents a particular relation $r\in
R$ of two entities$(v_{1},v_{2})$ linked by edge $e$. Each entity
$v\in\mathcal{V}$ belongs to a particular type $T$. $(v_{1},r,v_{2})$ is a
triplet means head entity $v_{1}$ to tail entity $v_{2}$ is with relation $r$.
In general, relation $r$ and its reverse ${r}^{-1}$ is not the same unless the
relation is symmetrical.

###### Definition 2

Heterogeneous Information Network. A heterogeneous information network (HIN)
is a special type of information network. In heterogeneous information network
$H$, edges $E$ and entities $\mathcal{V}$ are in different types, that is,
$|R|>1,|T|>1$.

Figure 2: An example of a heterogeneous information network: product
recommendation network

###### Example 1

As shown in Figure 2, A product recommendation network $G_{P}$ is a HIN, in
which entities $\mathcal{V}$ could be many types, including items, users,
categories, colors, brands, etc. $E$ could represent different relations.
Edges between users and items denote the buy relation, edges between items and
brands denote the is brand of relation, etc.

###### Definition 3

Meta-path. A meta-path [7] $P$ in a network from entity $v_{0}$ to $v_{k}$, is
denoted as $v_{0}\stackrel{{\scriptstyle
r_{0}}}{{\longrightarrow}}v_{1}\stackrel{{\scriptstyle
r_{1}}}{{\longrightarrow}}v_{2}\stackrel{{\scriptstyle
r_{2}}}{{\longrightarrow}}...\stackrel{{\scriptstyle
r_{k-1}}}{{\longrightarrow}}v_{k}$, where composite relation from $v_{0}$ to
$v_{k}$ is $r=r_{0}\circ r_{1}\circ r_{2}\circ...\circ r_{k-1}$. $\circ$
represents the composition operator on relations.

###### Example 2

In the product recommendation network $G_{P}$ in Figure 2, there are many
meta-paths; one of the meta-paths is $UIBI$, which means $user\ {\rightarrow}\
item\ {\rightarrow}\ brand\ {\rightarrow}\ item$. In the paper, we use the
meta-path instances instead of meta-paths. A meta-path instance is a specific
path, like $u_{1}\ {\rightarrow}\ AirPods\ {\rightarrow}\ Apple\
{\rightarrow}\ Beats\,Solo$ is a meta-path instance of meta-path $UIBI$.

###### Problem 1

Sequential knowledge-aware explainable recommendation. For a user $u_{i}\in
U$, given the item set $I$, the user transaction sequence $\pi_{u}$ and the
item associated information network $G$, the target of knowledge-aware
explainable recommendation is to predict top $k$ items that $u_{i}$ will
interact with, as well as the possible reasoning of recommended items.

##  3 Methodology

Figure 3: It is the architecture of Reinforcement Learning based Temporal
Meta-path Instances Guided Explainable Recommendation (TMER-RL). Here shows an
example: User Alice bought iPhone 11 pro, Phone case and Phone film in a
sequential order and the training process for Alice. The number in each
network block is the step order of the model. Details are in Section 3.1.

In this section, we introduce the proposed model Reinforcement Learning based
Temporal Meta-path Instances Guided Explainable Recommendation (TMER-RL). In
the remaining of the paper, we use the notation summarized in Table I to refer
to the variables and parameters used throughout the paper.

Symbol | Description
---|---
$G$ | Information network
$H$ | Heterogeneous information network (HIN)
$e\in E$ | An edge
$v\in\mathcal{V}$ | An entity or node
$R$ | Type of edges (relations)
$T$ | Type of nodes (entities)
$\mathcal{U}$ | User entity set
$\mathcal{I}$ | Item entity set
$\mathcal{B}$ | Brand entity set
$\mathcal{C}$ | Category entity set
$\mathcal{S}$ | State set
$\mathcal{A}$ | Action set
$p\in P$ | A meta-path in an HIN
$u\in U$ | User $u$
$i\in I$ | Item $i$
$W$ | Weight matrix
$b$ | Bias vector
$\phi\in\Phi$ | Path instance in a HIN
$\mathcal{N}$ | Neighbour function
$h$ | Embedding
$\alpha$ | Attention mechanism parameter
$\hat{r}$ | User-item rating
TABLE I: Description of major notations used in this paper

###  3.1 Overview of TMER-RL Architecture

The overall architecture of the proposed TMER-RL model is shown in Figure 3.
It mainly consists of five components. First, to initialize users and items,
we use DeepWalk [10] to embed user and item entities. Secondly, instead of
pre-defined meta-paths and randomly generating meta-paths instances [6], we
utilize a reinforcement learning with a Markov Decision Process (MDP)
environment to explore useful and meaningful sequential (temporal) and non-
sequential paths to improve the recommendation performance and
personalization. In this step, we obtain item-item instance paths between
consecutive items using reinforcement learning framework. In the third step,
after embedding instances, we use the multi-head attention to learn the
weights of instances as the weights of reasoning paths for the specific user.
Next, for the item updating, we employ a two-layer attention to make items
contain reasoning information. This step also models the users’ sequential
purchased information, feeding the previous item’s feature to the next one.
Finally, we feed item embeddings, user embeddings, and instances to the
recommendation network to do recommendation. The specific steps are elaborated
in the following subsections.

###  3.2 Initialize User and Item Representations

We firstly learn latent representations for involved users and items by
treating random truncated random walks in a user-item bipartite network as an
equivalent of sentences in DeepWalk [10], which optimizes the co-occurrence
probability among the entities in a walk by using skipgram based on word2vec
[11]. Other recent advanced graph-based embedding initialization methods can
be also applied, like GraphSage [12], GAT [13], and so on. These recent works
usually can even outperform than DeepWalk. However, through extensive
comparison with these methods, DeepWalk is the best choice, the possible
reason behind this is because DeepWalk pays more attention to the embedding of
nodes in a path, while GCNs, such as GraphSage, GAT, etc., learn embedding of
each entity with the aggregated feature information from its local
neighborhood. That is, these approaches pay more attention to local
relationships. However, if only considering the recommendation task itself,
although the local relationships are significant, the global higher-order
relationships learned by walks on a graph also play a notable role. For
example, the transitivity of co-purchasing relationships between friends and
the substitute relationship of items can be discovered by modelling higher-
order relations in a bipartite user-item graph.

###  3.3 Incorporating Meta-path based Context

In recommendation tasks, external features related to users and items, such as
product attributes, user social network, and user demographic information are
usually considered as additional auxiliary information to complement
traditional collaborative filtering methods. However, how to utilize the
heterogenous additional information efficiently is an open problem. Some prior
works [14, 15, 16] attempted to consider social relations as the user-side
information to boost the recommendation performance. To seek help from
injecting more complex additional information, recent works [17, 18]
introduced meta-paths into recommendation methods to describe relational
compositions between various types of entities in heterogeneous information
networks. In [17], the authors proposed to diffuse user preferences along
different meta-paths in information networks to generate latent features of
users and items. Related work [18] firstly extracts different-aspect features
with meta-paths from a HIN, and then fuse aspect-level latent factors to the
recommendation systems. However, these methods largely rely on the latent
factors obtained from constructed meta-path based similarity matrix, which are
too general and only can reflect mutual interaction between different types of
entities in a graph but cannot capture the specific information along
particular path instances. Therefore, inspired by existing work [5], our prior
work TMER and the proposed model TMER-RL explore to improve both
recommendation performance and explainability by modelling more specific meta-
path instances.

Different from existing works, we differentiate meta-path instances into two
different categories based on the involved entities in a recommendation
scenario (i.e., user-item and item-item meta-path instances). Through
modelling these path instances, we learn a more detailed meta-path based
context to further characterize the motivations, reasons as well as leading
factors between each pair of user-item interaction. While previous works such
as [5] mainly focuses on modelling meta-path instances between a user and an
item, this paper highlights the item-item meta-path instances, which we think
is beneficial in multiple aspects to sequential explainable recommender
systems. Firstly, only considering user-item paths is restrictive for
recommendation explainability as user-item paths only represent a user’s
general shopping interests. In comparison, item-item paths are more expressive
and can reflect devise reasons by exploring higher-order relations among
items, such as complementary products (e.g., phone and a phone case),
substitutable items of known items (e.g., iPhone - phone film - Huawei Phone),
co-purchased products with other people, etc. In addition, item-item paths
sometimes also serve as sequential modelling signals that naturally capture
the temporal dependencies between each consecutive item purchased by users,
which will be of great impact for sequential explainable recommendations.

Despite the powerful expressiveness of meta-paths in exploring HIN based
knowledge-aware recommendation, it is still challenging mainly because the
number of meta-paths is too large to handle (i.e., the amount of edges is
cubic to the number of entities). Taking the electric product recommendation
situation for an example, for $IUIBI$ meta-path schema, if we fix the starting
node - iPhone11 pro, there are many instances: $iPhone11\ pro\rightarrow
Alice\rightarrow iPad\rightarrow Apple\rightarrow AirPods$, $iPhone11\
pro\rightarrow Amy\rightarrow Phone\ case\rightarrow Miracase\rightarrow
Phone\ film$, and so on. Therefore, it is necessary to explore useful path
instances while limiting the total amount to simplify later calculations. The
path instances exploration is introduced in the next subsection.

###  3.4 Reinforcement Learning for Paths Exploration

Our previous work [6] pre-defines use-item and item-item meta-paths on the
recommendation knowledge graph, and randomly selects meta-path instances from
all existing ones. The hand-crafted meta-paths not only need human efforts,
but also are difficult to be determined when dealing with large recommendation
knowledge graphs. Therefore, we propose a reinforcement learning module to
explore potential useful meta-path instances on the recommendation knowledge
graph. The definition of usefulness in our work is to have contribution to the
recommendation module learning. The process of meta-path instances exploration
can be defined as a Markov Decision Process (MDP)[19], and we use
reinforcement learning to train the exploration policy.

  * •

State. It is the status of each step and the state set is defined as
$\mathcal{S}$. At step $t$, the state $\mathcal{S}_{t}$ is $(e_{t},his_{t})$,
where $e_{t}\in(\mathcal{U}\cup\mathcal{I}\cup\mathcal{B}\cup\mathcal{C})$ is
the current visiting entity and $his_{t}=\\{e_{0},e_{1},...,e_{t}\\}$ is the
history visited entities including the current one. Inspired by [1, 20, 21,
22], we add self-loop and inverse edges on knowledge graph $G$ to facilitate
graph traversal.

  * •

Action. The action set $\mathcal{A}_{t}$ at step $t$ means all candidate
entities to go. We define $\mathcal{A}_{t}=\mathcal{N}(e_{t})\cup e_{t}$,
which means the the candidate entities are the neighbours of the current
visiting entity $e_{t}$ and itself. Moreover, we allocate each action a weight
to present the probability of choosing each action. The score function of each
action at step $t$ is as follows.

$s_{e_{i}}=softmax(\frac{h_{e_{t}}\cdot
h_{e_{i}}^{T}}{||h_{e_{t}}||||h_{e_{i}}||}),\ where\ e_{i}\in\mathcal{A}_{t}$
(1)

where $h_{e_{t}}$ and $h_{e_{i}}$ mean the presentation of current visiting
entity and one of the actions, respectively. We believe cosine function could
indicate the similarity of two entities and it is more likely to choose
similar entities to form a path on the knowledge graph. However, on the
knowledge graph, some entities have dense structure, leading to large action
set. Thus, pruning unimportant candidate entities is necessary to improve
efficient. We rank each actions score descendingly and obtain top $k$ as new
action set.

  * •

Transition. Given a current state $\mathcal{S}_{t}=(e_{t},his_{t})$ and an
action $e_{i}\in\mathcal{A}_{t}$, the transition probability to the next state
$\mathcal{S}_{t+1}=(e_{t+1},his_{t+1})$ is

$p(\mathcal{S}_{t+1}|\mathcal{S}_{t},e_{i})=1$ (2)

  * •

Reward. The target of the reinforcement learning is to find user-item meta-
path instances and item-item meta-path instances, so the target entity is
known for each user. Therefore, if get to the target entity before pre-defined
maximum step $T$, the reward is 1, otherwise 0, mathematically,

$\mathcal{R}=\left\\{\begin{array}[]{lr}1,\ if\ get\ to\ the\ target\ entity\
before\ step\ $T$,&\\\ 0,\ otherwise&\end{array}\right.$ (3)

  * •

Optimization. The whole reinforcement learning module maximizes the final
reward to learn policy $\pi$ to find useful user-item and item-item meta-path
instances, mathematically,

$J(\theta)=\mathbb{E}_{e_{0}\in(\mathcal{U}\cup\mathcal{I})}(\mathbb{E}_{\pi}(\mathcal{R}))$
(4)

where $e_{0}$ is the initial entity. We use REINFORCE [23] algorithm to train
the objective function.

According to the above MDP framework, we can get candidate user-item and item-
item meta-path instances. Taking user-item meta-path instances from $u_{n}$ to
$i_{m}$ for an example, there should be $p$ candidate paths. We use a score
function to calculate each path’s score, mathematically,

$c_{u_{n}\rightarrow i_{m}}=\frac{\sum_{t=1}^{T}s_{e_{t}}}{T}$ (5)

where $s_{e_{t}}$ is each step’s action score. The rationale to choose average
step’s score as the path’s score is that we do not want the length of path to
influence the path’s score. We rank the score of paths descendingly and get
top $q$ candidate paths.

###  3.5 Parameterizing Combinational Features of Meta-paths as Recommendation
Context

####  3.5.1 Learning combinational path-based features with Self-Attention

After obtaining candidate user-item meta-path instances and item-item meta-
path instances, we first regard paths as sentences, nodes as tokens in
sentences, using Word2Vec[11] method and $Mean(\cdot)$ operations to learn
path embedding. Then, we employ multi-head self-attention units to learn the
meta-path based context (the User-Item and Item-Item Path Attention modules
shown in Figure 3). The rationale of deploying such self-attention units here
is that because, after sampling, there are still multiple paths between each
pair of item-item (or user-item) representing particular distinct reasons
(reasoning paths); and we observe that the reasons for buying consecutive two
items are not simply unique; rather, the reasons are more complex and likely a
mixture of multiple different reasons. For example, the reasons for a customer
to buy a phone case right after his/her previous purchase of a new phone are
probably a mixture of 1) his/her friends who own a similar phone and bought
this particular phone case, 2) the phone case is the most popular match for
the purchased new phone, 3) the color of the phone case matches the customer’s
preference. The potential reasons can be more than the listed, and again they
can be represented by using various meta-paths.

Based on this observation, self-attentive properties of the Transformer model
[24], we aim to learn the combinational features from multiple path instances
to better characterize the complex reasons between each connected pair of
entities in the KG.

$\begin{split}&Attention(Q_{\phi},K_{\phi},V_{\phi})=f(\frac{Q_{\phi}K_{\phi}^{T}}{\sqrt{d_{k}}})V_{\phi},\\\
&MultiHead(Q_{\phi},K_{\phi},V_{\phi})=Concat(head_{1},...,head_{m})W^{O},\\\
\end{split}$ (6)

where $head_{i}$ is
$Attention(W_{i}^{Q}Q_{\phi},W_{i}^{K}K_{\phi},W_{i}^{V}V_{\phi}).$ $f(.)$ is
a softmax function. Query $Q$, key $K$ and value $V$ are self-attention
variables associated with path $\phi$, and $W$ is the weight. $d_{k}$ is the
dimensionality (here $d_{k}=100$). $Concat(.)$ is the concatenation operation.

####  3.5.2 Modelling temporal dependencies with item-item meta-path instances

To learn the temporal dynamics of each user, the proposed TMER-RL framework
artfully resorts on the above-mentioned item-item meta-path instances together
with the well-designed architecture to capture the sequential dependencies
between two consecutive items. Compared with most existing works on sequential
recommendation [2, 4, 25] that utilize recurrent neural networks to encode the
temporal effects between items in a user’s interacted sequence, the proposed
TMER-RL bypasses the de-facto default deployment of RNNs or LSTMs that
sometimes make the model even heavier. Specifically, the proposed framework
novelly model the temporal dependencies between two items by capturing 1) the
information passed from the previous item though an item-attention unit, 2)
item-item connectivity trough a specific candidate path instance. Notably, the
information passed from the previous item is an attentive aggregation of
previous item-item connectivity information. For example, in Figure 3, whether
Alice will buy the phone film is modelled by considering 1) the information
passed from the phone case (which includes the connectivity between iphone11
pro and the phone case), and 2) the paths between the phone case to the phone
film. As a result, the long-range and short-term ”memory” in a sequence can be
captured, and the extent of the goodness of the long and short term memory can
be influenced by the length of the modelled sequence in different scenarios
with different datasets.

####  3.5.3 Updating item representations

After updating representations of user-item meta-path instances and item-item
meta-path instances according to a multi-head self-attention mechanism, we
employ a different kind of attention mechanism to update item representations.
It is obvious that the current item is mostly related to the last one, which
means it is better to add the last item’s information to the current one to
contain temporal information. Besides, the current item is also related to the
instances that arrived at this item. Therefore, we perform a two-layer
attention mechanism to update item representations. Mathematically,

$h_{i}^{(1)}=g(W_{i-1}h_{i-1}+W_{\phi_{i-1\rightarrow
i}}h_{\phi_{i-1\rightarrow i}}+b_{i}^{(1)})\odot h_{i-1}$ (7)
$h_{i}^{(2)}=g(W_{i}h_{i}+W_{\phi_{i-1\rightarrow
i}}^{(2)}h_{\phi_{i-1\rightarrow i}}+b_{i}^{(2)})\odot h_{i}$ (8)

where $h_{i}^{(1)}$ and $h_{i}^{(2)}$ mean the first and second layer output
of the item attention module, respectively. $h_{i-1}$ is the last item’s
latent representation. $\phi_{i-1\rightarrow i}$ is the instance from the
$(i-1)^{th}$ item to the $i^{th}$ item. $W$ and $b$ with different
superscripts denote different variables’ weights and bias. $g(.)$ is ReLU
function. However, until now, here is a problem with the calculation of the
first item, because there is no item before it. To solve this problem, we
involve the user-item instance from user $u$ to the first item into the update
of the first item, as Eq. 9 shows. Actually, the instance from $u$ to the
first item is really important in the recommendation, because it is the first
item user has bought and most of the other bought items have a sequential
relation with the first item to some extent. That is why we embed it into the
first item.

$h_{i=1}=g(W_{i}h_{i}+W_{\phi_{u\rightarrow i}}h_{\phi_{u\rightarrow
i}}+b_{i})\odot h_{i}$ (9)

where $\phi_{u\rightarrow i}$ represents the path from user $u$ to the first
item.

###  3.6 The Complete Recommendation Model

Finally, we concatenate user, item and instances information (calculated in
3.5.2) to a vector according to Eq. 10, and get user-item prediction scores
through Multilayer Perceptron (MLP) with explainability instances.

$h_{u,i}=[h_{u};h_{i}^{(1)};h_{i}^{(2)}]$ (10)

where $[;]$ means vector concatenation. Here $h_{u,i}$ denotes the explicit
mutual vector of the user, item, and implicit mutual of user-item meta-path
instances and item-item meta-path instances. For the first item, the
concatenation operation is different because of the dimension problem, and
therefore, for the first item related to each user, the vector fed in MLP
involves user-item instance, mathematically,

$h_{u,i=1}=[h_{u};h_{\phi_{u\rightarrow 1}};h_{i=1}]$ (11)

After that, final user-item rating calculates as follows.

$r_{u,i}=MLP(h_{u,i})$ (12)

where the MLP contains a two-hidden-layer neural network with ReLU as the
activation function and sigmoid function as the output layer. According to [5,
26], neural network models can learn more abstractive features of data via
using a small number of hidden units for higher layers, we employ a tower
structure for the MLP component, halving the layer size for each successive
higher layer.

We use implicit feedback loss with negative sampling [27, 28] as the whole
loss function:

$loss_{u,i}=-E_{j\sim P_{neg}}\left[\log\left(1-r_{u,j}\right)\right]$ (13)

where models the negative feedback drawn from the noise distribution
$P_{neg}$, which is a uniform distribution following [5].

###  3.7 Discussion of Explanation for Recommendation with Attention Mechanism

Attention Model [29, 30] was first introduced in machine translation task and
the attention weights were later widely used in natural language processing
tasks as explanations in neural networks [31, 32]. Other than Natural Language
Processing (NLP) tasks, the attention mechanism is also a near-ubiquitous
method in recommendation tasks used as explanations in some works. [33, 34,
35, 36] However, there are different opinions on whether attention mechanism
could be used as a way to explain data [37, 38, 39].

In our proposed method TMER-RL, the item-item meta-path instances with
attention weights learned by Item-Item Path Attention module in Figure. 3 are
used as explanations. To be specific, for all existing item-item meta-path
instances, we use reinforcement learning to explore some useful paths, because
it is difficult to process all paths whose number is large. After obtaining
the candidate item-item meta-path instances, we consider these paths as the
explanations for purchasing the target item. For example, in Figure. 3, user
Alice has bought iPhone 11 pro, Phone case and Phone film in a sequential
order. For paths from Phone case to Phone film, the paths includes $Phone\
case\rightarrow Phone\ accessories\rightarrow Phone\ film$, $Phone\
case\rightarrow Miracase\rightarrow Phone\ film$ and $Phone\ case\rightarrow
Abby\rightarrow Phone\ film$. In this situation, the explanations for buying
Phone film are that Alice has bought other Phone accessories, Alice has bought
other Miracase product, and that Abby has bought the Phone case and the Phone
film together. Based on them, we use a self-attention module as Item-Item Path
Attention module to learn a distribution of these paths (explanations) to
further explain the recommendation. The learned weights are considered as the
explainable weighted scores for the candidate item-item instances. Detailed
case study is in subsection 4.8.

##  4 Experiment

In this section, we present our experimental settings and give analysis on the
evaluation results.

###  4.1 Experiment Settings

####  4.1.1 Datasets

We use two public data collections to conduct experiments, Amazon datasets
111http://jmcauley.ucsd.edu/data/amazon/ [40] and Goodreads dataset
222http://goodreads.com [41, 42]. The Amazon dataset contains 29 categories,
from which we choose musical instruments dataset, automotive dataset, and toys
and games dataset. Each dataset includes brand and category as the metadata.
The Goodreads dataset is a public book online rating and review collection. We
select mystery thriller crime genre books and choose authors and publishers as
the metadata of books. More details are shown in Table II.

We select the latest twelve items for each user and order these items by
timestamps, and then we choose the first two items as bridge items, the next
four items as training items and the rest as test items. We create the
Heterogeneous Information Networks using user, item, category and brand in
Amazon datasets, and using user, book, author, publisher in Goodreads dataset,
respectively. Last, user-item meta-path instances and item-item meta-path
instances are explored according to section 3.4.

Datasets | User | Item | Category | Brand
---|---|---|---|---
Musical Instruments | 1450 | 11457 | 429 | 1185
Automotive | 4600 | 36663 | 1592 | 3790
Toys and Games | 9300 | 58743 | 820 | 5404
| User | Book | Author | Publisher
Goodreads | 11800 | 12142 | 3633 | 1442
TABLE II: Dataset Information.

####  4.1.2 Evaluations

We leverage Top $K$ Hit Ratio (HR@$K$) and Top $K$ Normalized Discounted
Cumulative Gain (NDCG@$K$) as our metrics. For each positive test instance
($u,i^{+}$), we mix the ground truth item $i^{+}$ with $\mathcal{N}$ random
negative items, then rank all these $\mathcal{N}+1$ items and compute the
HR@$K$ and NDCG@$K$ scores. We set $\mathcal{N}=500$ and $K=1,5,10,20$ for
evaluations. To alleviate the high-bias and low-variance of sampled metrics,
we use corrected evaluation metrics introduced in [43]. For evaluation of the
explainability of the recommendation, we use case studies to show the
explanations in detail.

####  4.1.3 Baselines

  * •

GRU4Rec [44, 9]: GRU4Rec is a session-based recommendation method using GRU.
For each user, we treat the training items as a session.

  * •

NARRE [45]: NARRE utilizes neural attention mechanism to build an
explainability recommendation system.

  * •

MCRec [5]: MCRec develops a deep neural network with the co-attention
mechanism to learn rich meta-path based context information for
recommendations.

  * •

NFM [46]: NFM effectively combines the linear Factorization Machines (FM) and
nonlinear neural networks for prediction under sparse settings.

  * •

Chorus [47]: Chorus considers both relations among items and corresponding
temporal dynamics to model the recommendation data in a knowledge-aware and
time-aware way. The enhanced item representations improve the performance.

  * •

$\mathbf{S^{3}Rec}$ [48]: $\mathrm{S^{3}Rec}$ utilizes the intrinsic data
correlation to employ self-supervised learning tasks to learn data
representations for sequential recommendation enhancement.

###  4.2 Implementation Details

For GRU4Rec, we use ReChorus package [47] to implement the algorithm. For
others, we directly run the provided code by each paper. To fairly compare the
evaluation results, we modify the each baseline’s evaluation code as the same
as TMER-RL. Especially, NARRE is a rating prediction model, we turn it into a
link prediction model by rating 1 positive item and 500 negative items and
ranking them.

###  4.3 Parameter Settings

We choose the first and second items for each user as the bridge items, the
training or testing process, the last 6 items for each user as the testing
positive items, and the remaining 4 are training items. The hyperparameters
are carefully tuned to achieve optimal results in all experiments. We
implement GRU4Rec, NARRE and NFM based on the paper by pytorch package. The
meta-paths and settings in MCRec are the same as the original paper. The meta-
paths in FMG are User-Item(UI), User-Item-Brand-Item(UIBI) and User-Item-
Category-Item(UICI).

In terms of the TMER compared in the following experiments, which removes the
path generation via reinforcement learning module, we use UIBI, UICI, UIBICI
and UICIBI as user-item meta-paths and ICIBI, IBICI, ICICI, IBIBI, IUIUI,
ICIUI and IBIUI as item-item meta-paths, where U, I, B and C denote user,
item, brand and category, respectively. For our proposed TMER-RL, the learning
rate for Amazon Musical Instruments dataset is $1e-5$, for the Amazon
Automotive dataset is $5e-5$ and for Amazon Toys and Games is $1e-4$. The
parameters for other baselines are searched for their best results. We set the
maximum length of path instance exploration as 6 because we set 6 as the max
length of meta-path in our previous work [6] and it is convenient to compare
the performance.

###  4.4 Effectiveness Analysis on Recommendation Results

We compare TMER-RL with 6 other popular and recent baselines, including four
sequential recommendations (GRU4Rec, Chorus, $\mathrm{S^{3}Rec}$ and our
former proposed TMER), an explainable recommendation (NARRE),a path-based
recommendation (MCRec) and a factorization machines-based recommendation (NFM)
on 4 datasets with 500 negative sampling.

As shown in Table III, our proposed TMER-RL achieves the best results and in
the most situations, TMER-RL could always give the correct items among 500
negative items, especially in Amazon Automotive dataset. The advantages also
hold on other datasets. These results demonstrate that our framework can
achieve obvious advantages through explicitly modelling each user’s sequential
purchased information with the temporal paths, while NARRE and NFM ignore the
sequential information for each user. MCRec ignores the item-item intrinsic
relations and just utilizes user-item interactions to train the
recommendation. GRU4Rec utilizes RNNs to model session-based recommendation.
Chorus focuses on the way to better learning items’ representations by
combining the complement and substitute relations among items on the temporal
items. $\mathrm{S^{3}Rec}$ uses different ways to pre-train embeddings to
improve recommendation. However, the above three sequential recommendations
all overlook the relation along paths and only focus on the temporal
historical sequence, leading to worse results.

Datasets | Metrics | GRU4Rec | NARRE | MCRec | NFM | Chorus | S3Rec | TMER | TMER-RL
---|---|---|---|---|---|---|---|---|---
Amazon Music Instruments | HR@1 | 0.3379 | 0.6171 | 0.5838 | 0.4457 | 0.1020 | 0.3448 | 0.8620 | 0.8762
HR@5 | 0.6833 | 0.8215 | 0.6178 | 0.5091 | 0.5386 | 0.5483 | 0.9473 | 0.9559
HR@10 | 0.8244 | 0.8828 | 0.6444 | 0.5505 | 0.8190 | 0.6448 | 0.9642 | 0.9761
HR@20 | 0.9272 | 0.9495 | 0.6819 | 0.6169 | 0.9651 | 0.7559 | 0.9736 | 0.9880
NDCG@1 | 0.3379 | 0.6171 | 0.5838 | 0.4457 | 0.1020 | 0.3448 | 0.8620 | 0.8762
NDCG@5 | 0.5187 | 0.7272 | 0.6009 | 0.4898 | 0.3185 | 0.4505 | 0.9306 | 0.9368
NDCG@10 | 0.5647 | 0.7472 | 0.6095 | 0.5037 | 0.4097 | 0.4817 | 0.9364 | 0.9438
NDCG@20 | 0.5910 | 0.7580 | 0.6189 | 0.5208 | 0.4474 | 0.5095 | 0.9389 | 0.9470
Amazon Automotive | HR@1 | 0.4376 | 0.6521 | 0.6899 | 0.6033 | 0.1389 | 0.4457 | 0.9070 | 0.9258
HR@5 | 0.7708 | 0.8229 | 0.7295 | 0.6542 | 0.5746 | 0.6843 | 0.9632 | 0.9832
HR@10 | 0.8813 | 0.8703 | 0.7546 | 0.6937 | 0.8205 | 0.7722 | 0.9701 | 0.9910
HR@20 | 0.9526 | 0.9080 | 0.7867 | 0.7430 | 0.9604 | 0.8511 | 0.9745 | 0.9954
NDCG@1 | 0.4376 | 0.6521 | 0.6899 | 0.6033 | 0.1389 | 0.4457 | 0.9070 | 0.9258
NDCG@5 | 0.6146 | 0.7450 | 0.7100 | 0.6383 | 0.3587 | 0.5737 | 0.9550 | 0.9734
NDCG@10 | 0.6506 | 0.7604 | 0.7180 | 0.6517 | 0.4386 | 0.6019 | 0.9574 | 0.9761
NDCG@20 | 0.6688 | 0.7699 | 0.7261 | 0.6644 | 0.4746 | 0.6221 | 0.9585 | 0.9773
Amazon Toys and Games | HR@1 | 0.3914 | 0.6698 | 0.6802 | 0.5365 | 0.1753 | 0.4741 | 0.8327 | 0.8632
HR@5 | 0.7385 | 0.8745 | 0.7181 | 0.5963 | 0.5734 | 0.7324 | 0.9478 | 0.9647
HR@10 | 0.8618 | 0.9234 | 0.7412 | 0.6412 | 0.7886 | 0.8172 | 0.9605 | 0.9811
HR@20 | 0.9431 | 0.9570 | 0.7661 | 0.6951 | 0.9368 | 0.8870 | 0.9680 | 0.9908
NDCG@1 | 0.3914 | 0.6698 | 0.6802 | 0.5365 | 0.1753 | 0.4741 | 0.8327 | 0.8632
NDCG@5 | 0.5745 | 0.7813 | 0.6996 | 0.5775 | 0.3773 | 0.6117 | 0.9292 | 0.9456
NDCG@10 | 0.6147 | 0.7973 | 0.7070 | 0.5929 | 0.4472 | 0.6400 | 0.9337 | 0.9512
NDCG@20 | 0.6355 | 0.8059 | 0.7133 | 0.6069 | 0.4852 | 0.6578 | 0.9356 | 0.9534
Goodreads | HR@1 | 0.2168 | 0.5007 | 0.4679 | 0.0789 | 0.1490 | 0.1967 | 0.5991 | 0.6276
HR@5 | 0.5477 | 0.7766 | 0.5469 | 0.1751 | 0.5224 | 0.4381 | 0.8499 | 0.8640
HR@10 | 0.7014 | 0.8570 | 0.5978 | 0.2414 | 0.7419 | 0.5527 | 0.9055 | 0.9168
HR@20 | 0.8502 | 0.9136 | 0.6600 | 0.3350 | 0.9075 | 0.6703 | 0.9395 | 0.9569
NDCG@1 | 0.2168 | 0.5007 | 0.4679 | 0.0789 | 0.1490 | 0.1967 | 0.5991 | 0.6276
NDCG@5 | 0.3910 | 0.6494 | 0.5083 | 0.1451 | 0.3379 | 0.3232 | 0.7968 | 0.8116
NDCG@10 | 0.4441 | 0.6755 | 0.5247 | 0.1677 | 0.4091 | 0.3603 | 0.8162 | 0.8305
NDCG@20 | 0.4818 | 0.6899 | 0.5404 | 0.1919 | 0.4514 | 0.3901 | 0.8251 | 0.8395
TABLE III: Performance Comparison with Baselines.

###  4.5 Effectiveness Analysis on Modules of TMER-RL

To study the impact of the modules of TMER-RL for the recommendation
performance, including the reinforcement learning path generation module,
user-item and item-item instances modules, we future compare our model (TMER-
RL) with three variants, namely TMER, $\neg$ UI and $\neg$ II. TMER removes
the reinforcement learning module and generates user-item and item-item
instances according to pre-defined meta-paths as mentioned in 4.3. $\neg$ UI
means we do not consider user-item instance and remove the corresponding self-
attention module. $\neg$ II means we remove the self-attention module for
item-item instances. We report the compared results with 100 negative sampling
in Table IV.

TMER-RL achieves the best performance on all evaluation methods, especially on
Amazon Automotive dataset. We can see that the user-item and item-item
instances self-attention modules significantly boost the recommendation
method, adaptively adjust the importance of different instance paths. The
self-learned user-item and item-item instance paths contribute to improving
the performance of recommendation finally. Especially, using the item-item
meta-path instances self-attention module helps TMER-RL improve 5.87% and
6.85% on HR@1 on Musical Instruments dataset and Automotive dataset,
respectively.

Dataset | Metrics | TMER-RL | TMER |  $\neg$ UI |  $\neg$ II
---|---|---|---|---|---
Amazon Musical Instruments | $H@1$ | 0.9252 | 0.9216 | 0.8797 | 0.8739
$H@5$ | 0.9862 | 0.9855 | 0.9301 | 0.9300
$H@10$ | 0.9942 | 0.9934 | 0.9366 | 0.9372
$H@20$ | 0.9979 | 0.9965 | 0.9479 | 0.9471
$N@1$ | 0.9252 | 0.9216 | 0.8797 | 0.8739
$N@5$ | 0.9741 | 0.9736 | 0.9259 | 0.9247
$N@10$ | 0.9765 | 0.9761 | 0.9281 | 0.9272
$N@20$ | 0.9775 | 0.9770 | 0.9310 | 0.9297
Amazon Automotive | $H@1$ | 0.9594 | 0.9482 | 0.8933 | 0.8979
$H@5$ | 0.9946 | 0.9903 | 0.9458 | 0.9464
$H@10$ | 0.9976 | 0.9939 | 0.9515 | 0.9531
$H@20$ | 0.9989 | 0.9963 | 0.9600 | 0.9618
$N@1$ | 0.9594 | 0.9482 | 0.8933 | 0.8979
$N@5$ | 0.9897 | 0.9844 | 0.9419 | 0.9422
$N@10$ | 0.9908 | 0.9857 | 0.9439 | 0.9445
$N@20$ | 0.9911 | 0.9863 | 0.9460 | 0.9468
TABLE IV: Impact of User-Item and Item-Item Instances Self-Attention
Mechanisms. $H@K$ and $N@K$ mean $HR@K$ and $NDCG@K$, respectively.

###  4.6 Variable Sensitivity

To test how the recommendation perform when variable changes, we conduct
variable sensitivity test on Amazon Musical Instruments dataset and Amazon
Automotive dataset. Figure 4 shows the influence of learning rate of
recommendation for HR@1. The x-axis represents the learning rate of
recommendation, which ranges among {1e-3, 5e-4, 1e-4, 5e-5, 1e-5}, and the
y-axis means the HR@1 value.

From Figure 4, we could see that the HR@1 fluctuation of TMER-RL is smaller
than TMER so we could get the conclusion that the performance of TMER-RL is
more stable than TMER. Moreover, our proposed TMER-RL could always outperform
TMER with different learning rates on these two datasets. This further proves
the effectiveness of TMER-RL. Especially, on the larger dataset (Amazon
Automotive), TMER-RL is more effective than TMER, which means our
reinforcement learning path generation module could show superiority on a
larger dataset, which is also required in the real scenario.

(a) HR@1 w.r.t. Learning rate on Amazon Musical Instruments

(b) HR@1 w.r.t. Learning rate on Amazon Automotive

Figure 4: HR@1 w.r.t Learning rate

###  4.7 Impact of the Amounts of Training Data

(a) HR@1 w.r.t. Training Proportion on Amazon Automotive

(b) HR@20 w.r.t. Training Proportion on Amazon Automotive.

Figure 5: HR@1 and HR@20 w.r.t Training Proportion on Amazon Automotive. The
x-axis represents the training proportion of the full Amazon Automotive
dataset.

To test whether our proposed method TMER-RL could show superiority on sparse
dataset, we conduct the TMER-RL and TMER on different training proportion of
the full Amazon Automotive dataset. We select {0.2, 0.4, 0.6, 0.8} as the
training proportion (x-axis in Figure 5) and compare the HR@1 (Figure 5(a))
and HR@20 (Figure 5(b)) of TMER-RL and TMER.

As we can see, the more training proportion of the full dataset, the better
performance of TMER and TMER-RL. This is in line with the common sense because
lower training proportion means sparser data, which leads to less paths
generation and captures less information among users and items. This will
result in lower performance of recommendation. However, in another
perspective, when dealing with the least training proportion data (0.2) in
Figure 5(a) and Figure 5(b), the gap of TMER-RL and TMER is the most and the
proposed TMER-RL always keep the better performance. This result could get the
result that the TMER-RL is more suitable for sparse dataset, which means it
could explore paths in a sparse data environment and do better recommendation.
Besides, the TMER-RL could always outperform TMER on HR@1 and HR@20 when the
training proportion varies, which further proves the effectiveness of TMER-RL.

###  4.8 Case Study: Generating Explanation for Recommendation

One of this work’s contributions is to give explanations on instances paths
while recommending preferable items. This is because our method generates
multiple item-item instance paths according to user purchase history, and then
it utilizes attention mechanism to learn the weight of each item-item instance
path and aggregate multiple item-item instance paths for each item-item pair.
To demonstrate this, we show a random example drawn from TMER on Amazon
Musical Instruments dataset.

We randomly select a user, whose id is $u273$, and track his item-item
instance paths’ parameters. In the training dataset, $u273$’s purchase history
is $i2954$, $i2280$, $i4514$ and $i11158$. We shows several sampled item-item
instance paths with high attention parameters in Figure 6 and demonstrates our
explanations.

Figure 6: It shows $u273$ and related historical purchased items in the upper
part of the figure. In the lower part, it displays the reasoning item-item
paths along this historical purchased items related to $u273$. Green blocks
represent the category attributes. Blue blocks represent the brand attributes.
Black blocks without physical pictures do not have meta information in the
dataset.

  * •

According to the Figure 6, there are three reasons for purchasing $i2280$. The
most probable reason with the highest attention weight is that $u273$ has
bought the last musical instrument category $i2954$ and $i2237$ with the
attention weight $0.18$. For the next item $i4514$, the reason for purchasing
it is that the user $u273$ has bought $i2280$ who has the same brand and
category with item $i4514$. There is also only one item-item instance path
between some items because the item-brand and item-category data are sparse.
Therefore, our method can model the reason through item-item instance paths
with different weights.

  * •

Besides, our model can capture sequential information according to user
purchase history thanks to item-item instance paths. These item-item instance
paths learn the reason path from the current item to the next item. In the
whole model, these reason paths will feed to the item attention module.
Therefore, our model can recommend with learned sequential information.

###  4.9 Case Study: Generating Paths via Reinforcement Learning

Figure 7: Summary of 10 generated paths via reinforcement learning from i2954
to i2280 for u273.

Compared with randomly generating paths according to pre-defined meta-paths in
our previous work [6], we design and implement a reinforcement learning
framework to learn and mine instances in this work. Also, instead of randomly
choosing paths among large amount of instances, we design a rank method to
sort the scores of generated paths and get top $k$ paths to feed into the
following modules. We provide a case study for paths generation by
reinforcement learning framework on Amazon Musical Instruments dataset.
Details are shown in Figure 7.

To be consistent with 4.8, we also choose $u273$ as the example. $i2954$
(Microphone Pop filter) and $i2280$ (Right Angle Instrument Cable) are the
first and the second items for $u273$. We summarize the generated 10 paths to
schemes in Figure 7. The generated paths schemes contain, IIBICI, IICICI,
ICICI and IICI.

Although the number of summarized schemes is less than that of pre-defined
meta-paths, the generated paths via reinforcement learning contributes to
recommendation more than randomly generation, which could be proved in the
above experiments. To be specific, when randomly generating, the learned paths
will be exactly consistent with the pre-defined meta-paths, but it is unsure
that the generated paths are useful for the recommendation. However, utilizing
reinforcement learning to mine paths could at least guarantee the correlation
of nodes on the paths because of the action calculation during exploring. The
proposed ranking equation could obtain the $k$ most relevant paths.

Moreover, the pre-defined meta-paths may not be able to define some more
useful schemes. For example, in Figure 7, there are 4 schemes, but only ICICI
is pre-defined meta-path in TMER. Furthermore, if the meta-path defines all
existing conditions and considers all situations, the calculating complexity
will be very large. Therefore, using reinforcement learning to explore schemes
is a better choice.

##  5 Related work

Early recommendation systems mostly rely on Collaborative Filtering (CF) [49,
50], which are based on the idea that users with similar history will be more
likely to purchase similar items. However, CF-based recommendations always
have sparsity issues and cold-start problems. Therefore, some works utilize
side information, like user and item attributes [51], item contents [52] to
solve this issue. Among them, methods based on knowledge graph [53, 1, 4, 2]
show great advantages in the recommendation performance and explainability.

Knowledge graph-based recommendations are roughly be categorized into
embedding-based approaches and path-based approaches. Prior efforts on
embedding-based knowledge graph recommendations [54] always use embeddings of
the knowledge graph to model the user-item interactions for recommendations.
For example, exiting works [55] utilizes TranE [56] to embed user-item
interactions to integrate knowledge into the recommendation system. Similarly,
[57] embeds user and item vectors into the same embedding space for
recommendation. The above approaches model the relations of users and items
using knowledge graph embedding methods, which achieves great improvement in
model expressiveness. However, these methods are sensitive to the quality of
related knowledge graphs.

To this end, meta-paths [7], and the connectivity of different types of nodes
are introduced to recommendations, which have the ability to learn the
explicit expression along each meta-path schema. In [58], the authors
introduce Matrix Factorization (MF) and Factorization Machine (FM) to learn
similarities generated by each meta-path for recommendation. [18] models rich
objects and relations in knowledge graph and extracts different aspect-level
similarity matrices thanks to meta-paths for the top-N recommendation.
Although they achieved appealing performance, the limitation is still obvious
that structured meta-path based similarity latent factors can only reflect
mutual infectivity along meta-path schemas in a graph but cannot capture the
specific information along particular path instances, which limits the
explainability of recommendation.

More recently, injecting meta-paths as recommendation context (aggregation of
meta-path instances) [5] was introduced for top-N recommendation. It provides
fine-grained explanations based on specific instances. However, it ignores the
important sequential dynamics of user-item interactions, which limits the
performance of the recommendation performance and interpretability. To
consider the sequential information, [2] utilizes LSTM to model the sequential
information, but it only considers path-based sequential information between
users and items and ignores the importance of the user’s clicked history
sequences, which are highly informative to infer user’s preferences. To tackle
this issue, [4] attempts to model the sequences of user’s behaviours and path
connectivity between users and items for recommendation. Nevertheless, it only
considers user-item paths, which ignores the item-item intrinsic relation
information and cannot learn some complex semantic information between items.
Moreover, it has to pre-define some meta-paths and randomly sample meta-path
instances, which involves too much human efforts and random factor. Based on
the above research, we propose a reinforcement learning based path exploration
for recommendation with differentiated user-item and sequential item-item
instances to enhance the learning ability for explainability recommendation.

##  6 Conclusion

We propose TMER-RL, which explicitly models dynamic user-item interactions
over time with path-based knowledge-aware explainable capabilities. We explore
item-item paths between consecutive items with attention mechanisms for users’
sequential behaviour modelling via a reinforcement learning framework. For
evaluation, we conduct 6 sets of experiment to prove the effectiveness of
TMER-RL, including comparing with 8 state-of-the-art baselines on 4 datasets
to show the high performance, ablation test on modules of TMER-RL to analysis
the importance of each module, variable sensitivity test to show the influence
of the learning rate varies, impact of the amounts of training data to prove
the ability to process sparse data, case study to explain the way to explain
recommendation via generated paths and another case study to show the
generated paths via reinforcement learning framework. All the above
experiments have proved the explainability, effectiveness, high-performance of
the TMER-RL. Future works may include the following directions: 1) generate
human-readable explanations for recommendation with NLP techniques, and 2)
explore causality learning [59] to discover more appealing paths for
explainablity.

## References

  * [1] Y. Xian, Z. Fu, S. Muthukrishnan, G. De Melo, and Y. Zhang, “Reinforcement knowledge graph reasoning for explainable recommendation,” in Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 285–294, 2019.
  * [2] X. Wang, D. Wang, C. Xu, X. He, Y. Cao, and T.-S. Chua, “Explainable reasoning over knowledge graphs for recommendation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5329–5336, 2019.
  * [3] K. Zhao, X. Wang, Y. Zhang, L. Zhao, Z. Liu, C. Xing, and X. Xie, “Leveraging demonstrations for reinforcement recommendation reasoning over knowledge graphs,” in Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 239–248, 2020.
  * [4] Q. Zhu, X. Zhou, J. Wu, J. Tan, and L. Guo, “A knowledge-aware attentional reasoning network for recommendation.,” in AAAI, pp. 6999–7006, 2020.
  * [5] B. Hu, C. Shi, W. X. Zhao, and P. S. Yu, “Leveraging meta-path based context for top-n recommendation with a neural co-attention model,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1531–1540, 2018.
  * [6] H. Chen, Y. Li, X. Sun, G. Xu, and H. Yin, “Temporal meta-path guided explainable recommendation,” in Proceedings of the 14th ACM International Conference on Web Search and Data Mining, p. 1056–1064, 2021\.
  * [7] Y. Sun, J. Han, X. Yan, P. S. Yu, and T. Wu, “Pathsim: Meta path-based top-k similarity search in heterogeneous information networks,” Proceedings of the VLDB Endowment, vol. 4, no. 11, pp. 992–1003, 2011.
  * [8] Q. Liu, S. Wu, D. Wang, Z. Li, and L. Wang, “Context-aware sequential recommendation,” in 2016 IEEE 16th International Conference on Data Mining (ICDM), pp. 1053–1058, IEEE, 2016.
  * [9] B. Hidasi and A. Karatzoglou, “Recurrent neural networks with top-k gains for session-based recommendations,” in Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 843–852, 2018.
  * [10] B. Perozzi, R. Al-Rfou, and S. Skiena, “Deepwalk: Online learning of social representations,” in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701–710, 2014.
  * [11] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013\.
  * [12] W. Hamilton, Z. Ying, and J. Leskovec, “Inductive representation learning on large graphs,” in NIPS, 2017.
  * [13] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, “Graph attention networks,” arXiv, 2017.
  * [14] M. Jamali and M. Ester, “A matrix factorization technique with trust propagation for recommendation in social networks,” in Proceedings of the fourth ACM conference on Recommender systems, pp. 135–142, 2010.
  * [15] H. Ma, H. Yang, M. R. Lyu, and I. King, “Sorec: social recommendation using probabilistic matrix factorization,” in Proceedings of the 17th ACM conference on Information and knowledge management, pp. 931–940, 2008.
  * [16] Q. Yuan, L. Chen, and S. Zhao, “Factorization vs. regularization: fusing heterogeneous social relationships in top-n recommendation,” in Proceedings of the fifth ACM conference on Recommender systems, pp. 245–252, 2011.
  * [17] X. Yu, X. Ren, Y. Sun, Q. Gu, B. Sturt, U. Khandelwal, B. Norick, and J. Han, “Personalized entity recommendation: A heterogeneous information network approach,” in Proceedings of the 7th ACM international conference on Web search and data mining, pp. 283–292, 2014.
  * [18] X. Han, C. Shi, S. Wang, S. Y. Philip, and L. Song, “Aspect-level deep collaborative filtering via heterogeneous information networks.,” in IJCAI, pp. 3393–3399, 2018.
  * [19] G. Shani, D. Heckerman, R. I. Brafman, and C. Boutilier, “An mdp-based recommender system.,” Journal of Machine Learning Research, vol. 6, no. 9, 2005.
  * [20] M. Saebi, S. Krieg, C. Zhang, M. Jiang, and N. Chawla, “Heterogeneous relational reasoning in knowledge graphs with reinforcement learning,” arXiv preprint arXiv:2003.06050, 2020.
  * [21] W. Xiong, T. Hoang, and W. Y. Wang, “Deeppath: A reinforcement learning method for knowledge graph reasoning,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), (Copenhagen, Denmark), ACL, September 2017.
  * [22] R. Das, S. Dhuliawala, M. Zaheer, L. Vilnis, I. Durugkar, A. Krishnamurthy, A. Smola, and A. McCallum, “Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning,” in ICLR, 2018.
  * [23] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3-4, pp. 229–256, 1992.
  * [24] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in neural information processing systems, pp. 5998–6008, 2017.
  * [25] C. Xu, P. Zhao, Y. Liu, J. Xu, V. S. S. S. Sheng, Z. Cui, X. Zhou, and H. Xiong, “Recurrent convolutional neural network for sequential recommendation,” in The World Wide Web Conference, pp. 3398–3404, 2019\.
  * [26] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [27] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” in Proceedings of the 26th international conference on world wide web, pp. 173–182, 2017.
  * [28] J. Tang, M. Qu, M. Wang, M. Zhang, J. Yan, and Q. Mei, “Line: Large-scale information network embedding,” in Proceedings of the 24th international conference on world wide web, pp. 1067–1077, 2015.
  * [29] D. Bahdanau, K. H. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in 3rd International Conference on Learning Representations, ICLR 2015, 2015.
  * [30] S. Chaudhari, V. Mithal, G. Polatkan, and R. Ramanath, “An attentive survey of attention models,” arXiv preprint arXiv:1904.02874, 2019.
  * [31] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention,” in International conference on machine learning, pp. 2048–2057, PMLR, 2015.
  * [32] J. Mullenbach, S. Wiegreffe, J. Duke, J. Sun, and J. Eisenstein, “Explainable prediction of medical codes from clinical text,” arXiv preprint arXiv:1802.05695, 2018.
  * [33] X. Wang, Y. Wang, and Y. Ling, “Attention-guide walk model in heterogeneous information network for multi-style recommendation explanation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 6275–6282, 2020.
  * [34] J. Chen, F. Zhuang, X. Hong, X. Ao, X. Xie, and Q. He, “Attention-driven factor model for explainable personalized recommendation,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 909–912, 2018.
  * [35] O. Tal, Y. Liu, J. Huang, X. Yu, and B. Aljbawi, “Neural attention frameworks for explainable recommendation,” IEEE Transactions on Knowledge and Data Engineering, 2019.
  * [36] D. Cong, Y. Zhao, B. Qin, Y. Han, M. Zhang, A. Liu, and N. Chen, “Hierarchical attention based neural network for explainable recommendation,” in Proceedings of the 2019 on International Conference on Multimedia Retrieval, pp. 373–381, 2019.
  * [37] S. Jain and B. C. Wallace, “Attention is not explanation,” arXiv preprint arXiv:1902.10186, 2019.
  * [38] S. Wiegreffe and Y. Pinter, “Attention is not not explanation,” arXiv preprint arXiv:1908.04626, 2019.
  * [39] S. Serrano and N. A. Smith, “Is attention interpretable?,” arXiv preprint arXiv:1906.03731, 2019.
  * [40] J. Ni, J. Li, and J. McAuley, “Justifying recommendations using distantly-labeled reviews and fine-grained aspects,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 188–197, 2019.
  * [41] M. Wan and J. J. McAuley, “Item recommendation on monotonic behavior chains,” in Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018, pp. 86–94, ACM, 2018\.
  * [42] M. Wan, R. Misra, N. Nakashole, and J. J. McAuley, “Fine-grained spoiler detection from large-scale review corpora,” in Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 2605–2610, Association for Computational Linguistics, 2019.
  * [43] W. Krichene and S. Rendle, “On sampled metrics for item recommendation,” in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1748–1757, 2020.
  * [44] B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk, “Session-based recommendations with recurrent neural networks,” arXiv preprint arXiv:1511.06939, 2015.
  * [45] C. Chen, M. Zhang, Y. Liu, and S. Ma, “Neural attentional rating regression with review-level explanations,” in Proceedings of the 2018 World Wide Web Conference, pp. 1583–1592, 2018.
  * [46] X. He and T.-S. Chua, “Neural factorization machines for sparse predictive analytics,” in Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval, pp. 355–364, 2017.
  * [47] C. Wang, M. Zhang, W. Ma, Y. Liu, and S. Ma, “Make it a chorus: knowledge-and time-aware item modeling for sequential recommendation,” in Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 109–118, 2020.
  * [48] K. Zhou, H. Wang, W. X. Zhao, Y. Zhu, S. Wang, F. Zhang, Z. Wang, and J.-R. Wen, “S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization,” in Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1893–1902, 2020.
  * [49] J. B. Schafer, D. Frankowski, J. Herlocker, and S. Sen, “Collaborative filtering recommender systems,” in The adaptive web, pp. 291–324, Springer, 2007.
  * [50] B. Sarwar, G. Karypis, J. Konstan, and J. Riedl, “Item-based collaborative filtering recommendation algorithms,” in Proceedings of the 10th international conference on World Wide Web, pp. 285–295, 2001.
  * [51] S. Gong, “Employing user attribute and item attribute to enhance the collaborative filtering recommendation.,” JSW, vol. 4, no. 8, pp. 883–890, 2009.
  * [52] P. Melville, R. J. Mooney, and R. Nagarajan, “Content-boosted collaborative filtering for improved recommendations,” Aaai/iaai, vol. 23, pp. 187–192, 2002.
  * [53] X. Wang, Y. Xu, X. He, Y. Cao, M. Wang, and T.-S. Chua, “Reinforced negative sampling over knowledge graph for recommendation,” arXiv preprint arXiv:2003.05753, 2020.
  * [54] F. Zhang, N. J. Yuan, D. Lian, X. Xie, and W.-Y. Ma, “Collaborative knowledge base embedding for recommender systems,” in Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 353–362, 2016.
  * [55] J. Huang, W. X. Zhao, H. Dou, J.-R. Wen, and E. Y. Chang, “Improving sequential recommendation with knowledge-enhanced memory networks,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 505–514, 2018.
  * [56] A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko, “Translating embeddings for modeling multi-relational data,” in Advances in neural information processing systems, pp. 2787–2795, 2013.
  * [57] L. Grad-Gyenge, A. Kiss, and P. Filzmoser, “Graph embedding based recommendation techniques on the knowledge graph,” in Adjunct Publication of the 25th Conference on User Modeling, Adaptation and Personalization, pp. 354–359, 2017.
  * [58] H. Zhao, Q. Yao, J. Li, Y. Song, and D. L. Lee, “Meta-graph based recommendation fusion over heterogeneous information networks,” in Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 635–644, 2017.
  * [59] G. Xu, T. D. Duong, Q. Li, S. Liu, and X. Wang, “Causality learning: A new perspective for interpretable machine learning,” arXiv preprint arXiv:2006.16789, 2020.

|  Yicong Li is currently a PhD student of Data Science and Machine
Intelligence (DSMI) Lab of Advanced Analytics Institute, University of
Technology Sydney. She obtained her Master’s degree from National University
of Defense Technology in 2019. Her research interests mainly focus on data
science, graph neural networks, recommender systems, natural language
processing and so on. In particular, her current research is focusing on the
explainable machine learning, especially the application in recommendation
area. She has published papers in international conferences and journals, such
as WSDM, KSEM and IEEE Access. She have also reviewed submitted papers in many
top-tier conferences and journals, like AAAI, KDD, WWW, IJCAI, WSDM, ICONIP
and so on. In addition, she has been invited to review manuscripts in IEEE
Transactions on Neural Networks and Learning Systems (TNNLS), which is a top-
tier journal in artificial intelligence.
---|---
|  Hongxu Chen is a Data Scientist, now working as a Postdoctoral Research
Fellow in School of Computer Science at University of Technology Sydney,
Australia. He obtained his Ph.D. in Computer Science at The University of
Queensland in 2020. His research interests mainly focus on data science in
general and expend across multiple practical application scenarios, such as
network science, data mining, recommendation systems and social network
analytics. In particular, his research is focusing on learning representations
for information networks and applying the learned network representations to
solve real-world problems in complex networks such as biology, e-commerce and
social networks, financial market and recommendations systems with
heterogeneous information sources. He has published many peer-reviewed papers
in top-tier high-quality international conferences and journals, such as
SIGKDD, ICDE, ICDM, AAAI, IJCAI, TKDE. He also serves as program committee
member and reviewers in multiple international conference, such as CIKM, ICDM,
KDD, SIGIR, AAAI, PAKDD, WISE, and he also acts as invited reviewer for
multiple journals in his research fields, including Transactions on Knowledge
and Data Engineering (TKDE), WWW Journal, VLDB Journal, IEEE Transactions on
Systems, Man and Cybernetics: Systems, Journal of Complexity, ACM Transactions
on Data Science, Journal of Computer Science and Technology.
---|---
|  Yile Li has received her Master degree from Tongji University, China, in
2021, in transportation engineering, and has received Bachelor’s degree from
Central South University, China, in 2018. Her research interests include
recommendation-system, graph neural network, reinforcement learning and multi-
agent system.
---|---
|  Lin Li is a professor in the School of Computer Science and Technology,
Wuhan University of Technology, the Distinguished Overseas Researcher of Iwate
University, Japan, and the Senior Entrepreneurship Instructor Certified by the
Ministry of Human Resources and Social Security. She has received a Ph.D. in
Information Science and Technology from the University of Tokyo in 2009. She
has long been engaged in research in the fields of information retrieval,
recommendation systems and text mining. She has served as a journal editor for
Web Intelligence and Human-Centric Intelligent Systems, and a PC or SPC for
international conferences such as IJCAI, AAAI, ACM MM, ICME, COLING, EMNLP.
She is a reviewer of ACM TOMM, IPM, ACM TOIS, WWWJ, KBS, etc, and is also a
guest editor of WWWJ. She organized a special session for ICME 2021. She has
published more than 80 academic papers, and edited or participated in the
editing of 2 English monographs (17,094 chapter downloads), 2 Chinese
monographs, 4 national invention patents, and more than 10 software
copyrights.
---|---
|  Philip S. Yu is a Distinguished Professor in Computer Science at the
University of Illinois at Chicago and also holds the Wexler Chair in
Information and Technology. His research interests include data mining,
privacy preserving publishing and mining, data streams, database systems,
Internet applications and technologies, multimedia systems, parallel and
distributed processing, and performance modeling. Dr. Yu has published more
than 970 papers in refereed journals and conferences with more than 143,604
citations and an H-index of 1215. He holds or has applied for more than 300 US
patents. He is a Fellow of the ACM and of the IEEE. He is associate editors of
ACM Transactions on the Internet Technology and ACM Transactions on Knowledge
Discovery from Data. He is on the steering committee of IEEE Conference on
Data Mining and was a member of the IEEE Data Engineering steering committee.
He was the Editor-in-Chief of IEEE Transactions on Knowledge and Data
Engineering (2001-2004), an editor, advisory board member and also a guest co-
editor of the special issue on mining of databases. He had also served as an
associate editor of Knowledge and Information Systems. He also received an
IEEE Region 1 Award for ”promoting and perpetuating numerous new electrical
engineering concepts” in 1999. He had received several UIC honors, including
Research of the Year at 2013 and UI Faculty Scholar at 2014. He also received
many IBM honors including 2 IBM Outstanding Innovation Awards, an Outstanding
Technical Achievement Award, 2 Research Division Awards and the 94th plateau
of Invention Achievement Awards. He was an IBM Master Inventor. Dr. Yu
received the B.S. Degree in E.E. from National Taiwan University, the M.S. and
Ph.D. degrees in E.E. from Stanford University, and the M.B.A. degree from New
York University.
---|---
|  Guandong Xu is a professor in the School of Computer Science and Data
Science Institute at UTS and an award-winning researcher working in the fields
of data mining, machine learning, social computing and other associated
fields. He is Director of the UTS-Providence Smart Future Research Centre,
which targets research and innovation in disruptive technology to drive
sustainability. He also heads the Data Science and Machine Intelligence Lab,
which is dedicated to research excellence and industry innovation across
academia and industry, aligning with the UTS research priority areas in data
science and artificial intelligence. Guandong has had more than 220 papers
published in the fields of Data Science and Data Analytics, Recommender
Systems, Text Mining, Predictive Analytics, User Behaviour Modelling, and
Social Computing in international journals and conference proceedings in
recent years, with increasing citations from academia. He has shown strong
academic leadership in various professional activities. He is the founding
Editor-in-Chief of Human-centric Intelligent System Journal, the Assistant
Editor-in-Chief of World Wide Web Journal, as well as the founding Steering
Committee Chair of the International Conference of Behavioural and Social
Computing Conference.
---|---