430074

# Reinforced MOOCs Concept Recommendation in Heterogeneous Information
Networks

Jibing Gong<EMAIL_ADDRESS>School of Information Science and
Engineering, the Key Lab for Computer Virtual Technology and System
Integration, and the Key Laboratory for Software Engineering of Hebei
ProvinceYanshan UniversityNo. 438 West Hebei AvenueQinhuangdaoHebeiChina066004
,  Yao Wan<EMAIL_ADDRESS>School of Computer Science and
TechnologyHuazhong University of Science and TechnologyLuoyu Road
1037WuhanHubeiChina430074 ,  Ye Liu<EMAIL_ADDRESS>Salesforce
Research556 Xixi RoadChicagoUSA ,  Xuewen Li<EMAIL_ADDRESS>School of Information Science and Engineering, the Key Lab for Computer
Virtual Technology and System Integration, and the Key Laboratory for Software
Engineering of Hebei ProvinceYanshan UniversityNo. 438 West Hebei
AvenueQinhuangdaoHebeiChina066004 ,  Yi Zhao<EMAIL_ADDRESS>School of
Information Science and Engineering, the Key Lab for Computer Virtual
Technology and System Integration, and the Key Laboratory for Software
Engineering of Hebei ProvinceYanshan UniversityNo. 438 West Hebei
AvenueQinhuangdaoHebeiChina066004 ,  Cheng Wang<EMAIL_ADDRESS>School of
Information Science and Engineering, the Key Lab for Computer Virtual
Technology and System Integration, and the Key Laboratory for Software
Engineering of Hebei ProvinceYanshan UniversityNo. 438 West Hebei
AvenueQinhuangdaoHubeiChina066004 ,  Yuting Lin<EMAIL_ADDRESS>School
of Information Science and Engineering, the Key Lab for Computer Virtual
Technology and System Integration, and the Key Laboratory for Software
Engineering of Hebei ProvinceYanshan UniversityNo. 438 West Hebei
AvenueQinhuangdaoHebeiChina066004 ,  Xiaohan Fang<EMAIL_ADDRESS>School of Information Science and Engineering, the Key Lab for Computer
Virtual Technology and System Integration, and the Key Laboratory for Software
Engineering of Hebei ProvinceYanshan UniversityNo. 438 West Hebei
AvenueQinhuangdaoHebeiChina066004 ,  Wenzheng Feng
<EMAIL_ADDRESS>Department of Computer ScienceTsinghua
UniversityBeijingChina100084 ,  Jingyi Zhang<EMAIL_ADDRESS>School of
Cyber Science and TechnologyBeihang University37 Xueyuan
RoadBeijingChina100191 and  Jie Tang<EMAIL_ADDRESS>Department of
Computer ScienceTsinghua UniversityBeijingChina100084

(2022)

###### Abstract.

Massive open online courses (MOOCs), which offer open access and widespread
interactive participation through the internet, are quickly becoming the
preferred method for online and remote learning. Several MOOC platforms offer
the service of course recommendation to users, to improve the learning
experience of users. Despite the usefulness of this service, we consider that
recommending courses to users directly may neglect their varying degrees of
expertise. To mitigate this gap, we examine an interesting problem of concept
recommendation in this paper, which can be viewed as recommending knowledge to
users in a fine-grained way. We put forward a novel approach, termed HinCRec-
RL, for Concept Recommendation in MOOCs, which is based on Heterogeneous
Information Networks and Reinforcement Learning. In particular, we propose to
shape the problem of concept recommendation within a reinforcement learning
framework to characterize the dynamic interaction between users and knowledge
concepts in MOOCs. Furthermore, we propose to form the interactions among
users, courses, videos, and concepts into a heterogeneous information network
(HIN) to learn the semantic user representations better. We then employ an
attentional graph neural network to represent the users in the HIN, based on
meta-paths. Extensive experiments are conducted on a real-world dataset
collected from a Chinese MOOC platform, XuetangX, to validate the efficacy of
our proposed HinCRec-RL. Experimental results and analysis demonstrate that
our proposed HinCRec-RL performs well when compared with several state-of-the-
art models.

HIN, reinforcement learning, MOOCs, concept recommendation

††copyright: acmcopyright††journalyear: 2022††doi:
10.1145/1122445.1122456††journal: JACM††journalvolume: 37††journalnumber:
4††article: 1††publicationmonth: 1

##  1\. Introduction

(a) The heterogeneous network in MOOCs

(b) Studying progress

(c) Studying speed

Figure 1. The motivation of concept recommendation. (a) The heterogeneous
network in MOOCs. (b) and (c) denote the progress of studying, and the speed
of studying, respectively.

Massive Open Online Courses (MOOCs), which seek to offer widespread
interactive participation and open access over the internet, are quickly
becoming a reliable platform for online and remote education.
Coursera111https://www.coursera.org, Udacity222https://www.udacity.com, and
edX333https://www.edx.org are the three most well-known MOOC platforms, which
have provided online access to millions of users to study from anywhere in the
world. As one of China’s leading MOOC platforms,
XuetangX444https://www.xuetangx.com has more than 1,000 courses available and
more than 6,000,000 users worldwide. In MOOCs, course concepts are used to
describe the knowledge concepts and related subjects covered in the course
videos. This can make it easier for users to find the information in the
course videos more quickly.

Several approaches of data mining have been developed to discover user
preferences and the relationships among users, course topics, and course
videos, to enable users to have a better learning experience in MOOCs. These
approaches have been used in various related tasks, e.g., behavior prediction
(Qiu et al., 2016), course recommendation (Jing and Tang, 2017; Zhang et al.,
2019), and user intention understanding (Feng et al., 2019). The task most
similar to ours is course recommendation, which seeks to recommend relevant
courses to users based on their past behavior. From our investigation, we
observe that a course is always composed of a series of video lectures, each
of which often conveys multiple specific knowledge concepts. Consequently, we
argue that recommending courses to users directly ignores the various
knowledge levels of users. Figure 1(a) shows the heterogeneous information
network existing in MOOCs. For example, considering the Machine Learning
course taught by different instructors, the knowledge covered in each course
will differ largely (e.g., some instructors will put more focus on the theory
of machine learning, whereas others will put more focus on the applications of
machine learning in the area of computer science). If a student has a
background in statistics theory, he/she may be familiar with several
fundamental knowledge concepts, such as VC dimension and gradient descent, so
that he/she will be interested in further applications of machine learning.
Additionally, a student who is a beginner in machine learning, he/she may be
interested in several fundamental knowledge concepts of machine learning.

Based on this motivation, this paper examines an interesting problem of
concept recommendation, with the aim of recommending related topics to users
to help them with their online studies. To better illustrate the necessity of
this research, we study the effects of clicked course concepts on the progress
and speed of studying, as shown in Figure 1(b) and Figure 1(c). The progress
of studying is defined as the portion of videos the user has watched in a
course, whereas the speed of studying is defined as the number of videos the
user has watched in one week. From these two figures, we can see that the
number of clicked concepts has a positive correlation with the progress and
speed of study. It confirms the fact that clicking relevant knowledge concepts
can indeed help the users with their online studies.

(a) Concept length distribution

(b) Concept click count

Figure 2. A statistical analysis of XuetangX dataset.

For the task of concept recommendation, one primary line of work is based on
collaborative filtering. The collaborative-filtering-based approaches provide
recommendations according to the historical interactions among users and
items. However, this kind of approaches has two major limitations. (1) Dynamic
environment adaptation. The collaborative-filtering-based approaches consider
the procedure of recommendation as a static process, and are intended to
maximize the immediate reward of the recommendation, resulting in a fixed
greedy recommendation strategy. Therefore, this type of recommendation
strategy is sub-optimal and cannot adapt to a dynamic environment where the
preferences of users are always dynamic. On the other hand, when representing
the historical dynamic preferences of users, it is a challenge to handle the
long dependency issue existing in the past click history, as shown in Figure
2(a). From Figure 2(a), we can observe that the length of concepts clicked by
users is mainly between 10 and 30, which is a relatively long sequential
dependency when compared with traditional recommendation tasks, such as
e-commerce recommendation. (2) Data sparsity issue. The interactions among
users and concepts are always sparse, particularly as the number of users and
concepts is increasingly large. Figure 2(b) shows the distribution of user
click count for each concept. This distribution follows the long-tail
distribution. That is, the majority of concepts are clicked by a few users,
which confirms the necessity of concept recommendation. To overcome this
issue, several approaches are attempting to include additional supplementary
information, such as the social network (Wu et al., 2019; Zhou et al., 2020),
user/item attributes (Wu et al., 2020; Sun et al., 2020), and contextual
information (Intayoad et al., 2020; Liu et al., 2022a), as to enrich the
interactions.

These aforementioned restrictions encourage us to develop a model to
understand the intents of users, with the ability to recommend personalized
concepts in a dynamic interactive environment. We propose in this paper to
shape the task of concept recommendation as a sequential recommendation
problem. To preserve the dynamic interactions between users and concepts, a
reinforcement-learning-based method is proposed by combining exploitation and
exploration into a unified framework. Our reinforcement-learning-based
recommendation has two advantages: (1) The recommendation agent can adaptively
update the trial-and-error strategy during interaction until convergence; (2)
The reinforced strategy can produce more diverse recommended results,
recommending those items with substantial long-term benefits rather than
short-term benefits (i.e., small intermediate rewards). To overcome the data
sparsity issue, we observe that there exist heterogeneous relationships among
users and concepts, which can be exploited to enrich the side information and
boost the performance of recommendation in the following two perspectives: (1)
Semantic relationships between knowledge concepts may be provided, which can
aid in identifying latent interactions; (2) Users’ interest can be understood
by tracking the previous records of users alongside these relationships.

This paper puts forward a novel approach (termed HinCRec-RL) for Concept
Recommendation in MOOCs, based on Heterogeneous Information Networks (HINs)
and Reinforcement Learning. We begin by shaping the concept recommendation as
a Markov Decision Process (MDP). We then form the interactions among users,
courses, videos, and concepts, into an HIN. We carry out extensive experiments
on a real-world dataset obtained from a Chinese MOOC platform, XuetangX, to
validate the efficacy of the proposed model. Note that this paper is related
to our previous work ACKRec (Gong et al., 2020) for knowledge concept
recommendation, which models the entities in MOOCs as an HIN, and proposes an
attentional graph convolutional neural network to represent the entities in
HIN. To enable the situations of online dynamic interaction, we extend ACKRec
by modeling the concept recommendation problem within a reinforcement learning
framework.

The primary contributions of this paper include the following three aspects.

  * •

Firstly, we investigate an interesting problem of concept recommendation,
which can be considered as a fine-grained recommendation task of course
recommendation in MOOCs.

  * •

Secondly, an end-to-end model based on reinforcement learning is proposed to
cope with the dynamic interaction among users and concepts, providing a more
diverse recommendation. To represent users with sparse data accurately, we
propose a meta-path-based user embedding approach with hierarchical graph
attention networks.

  * •

Thirdly, we validate the proposed model HinCRec-RL on a dataset collected from
a popular Chinese MOOC platform, XuetangX. Experimental results and analysis
demonstrate the effectiveness of HinCRec-RL when comparing it with several
state-of-the-art models.

We structure the rest paper as follows. We first formulate the concept
recommendation problem in Sec. 2. We present the details of the proposed
approach in Sec. 3. We analyze the XuetangX dataset in Sec. 4, and show the
experimental results with analyses in Sec. 5. We highlight some related work
with a comparison in Sec. 6. We have a discussion of the mechanism of our
proposed model in Sec. 7. Finally, Sec. 8 concludes this paper with some
potential directions suggested.

##  2\. Problem Formulation

In MOOCs, suppose that we have $U$ users, $C$ courses, $V$ videos and $K$
concepts. Let $\mathcal{U}=\\{u_{1},u_{2},\ldots,u_{U}\\}$ be the set of
users, $\mathcal{C}=\\{c_{1},c_{2},\ldots,c_{C}\\}$ be the set of courses,
$\mathcal{V}=\\{v_{1},v_{2},\ldots,v_{V}\\}$ be the set of videos and
$\mathcal{K}=\\{k_{1},k_{2},\ldots,k_{K}\\}$ be the set of concepts. In
reality, a course $c_{i}$ is always composed of multiple videos, each of which
consists of multiple knowledge concepts. From the perspective of users, a user
$u_{j}$ will take a course, watch the videos of the course, and click several
related concepts to learn the knowledge presented in the course videos.
Consequently, the aim of concept recommendation is to enhance the learning
experiences of users by automatically recommending relevant concepts to users
based on their preferences acquired from historical experience.

Figure 3. Network schema of HIN in XuetangX.

###  2.1. Recommendation as an MDP

As mentioned before, to adapt to the dynamic environment, we aim to learn a
recommendation agent that interacts with users by sequentially recommending
knowledge concepts at each time step. We formulate the problem of sequential
recommendation as a Markov Decision Process (MDP) consisting of a tuple of
five items $(\mathcal{S},\mathcal{A},P,R,\gamma)$:

  * •

State space $\mathcal{S}$ denotes the space of user states. Specifically, the
state $S_{t}$ denotes the past preference of user $u$, observed until time
step $t$. In our scenario, the past preference of user $u$ are the historical
learned courses and clicked concepts. More specifically, the state in our
scenario represents the user embedding in an HIN.

  * •

Action space $\mathcal{A}$ denotes the set of concepts to be selected, i.e.,
$\mathcal{A}=\mathcal{K}$. Without loss of generality, we sequentially
recommend one concept to users at each time step. In each state $s_{t}$, an
action $a_{t}$ can be drawn from the available action set
$\mathcal{A}(s_{t})$, recursively defined as
$\mathcal{A}(s_{t})=\mathcal{A}(s_{t-1})\backslash\\{a_{t-1}\\}$, for $t\neq
0$. This indicates that the agent is not permitted to select those concepts
that have been recommended in prior time steps.

  * •

Reward $R$, formulated as
$R^{a}_{ss^{\prime}}=\mathbb{E}[r_{t+1}|s_{t}=s,a_{t}=a,s_{t+1}=s^{\prime}]$,
represents the expected immediate reward produced by the environment when
state $s$ changes to state $s^{\prime}$ as a result of action $a$. In our
scenario, the immediate reward of performing an action $a$ is determined by
the operation of clicking or not, given by user $u$. Therefore, we define
$R^{a}_{ss^{\prime}}=R_{ua}$.

  * •

Transition function $P$, formulated as
$P^{a}_{ss^{\prime}}=P[s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a]$, denotes the
likelihood that the environment would transit from state $s$ to state
$s^{\prime}$ after performing an action $a$.

  * •

Discount factor $\gamma$ is used to measure the proportion of reward the
reinforced agent considers in the distant future versus those in the immediate
future, with a range of $[0,1]$.

The concept recommendation task can be formally characterized as follows,
using the aforementioned notations and terminologies. Given an MDP
$(\mathcal{S},\mathcal{A},P,R,\gamma)$, the aim of concept recommendation is
to find a policy $\pi:\mathcal{\pi}\rightarrow\mathcal{A}$ that maximizes the
cumulated reward from users, i.e., increasing the clickthrough rate of
concepts.

###  2.2. Construction of Heterogeneous Information Network

Figure 4. Defined meta-paths in XuetangX.

In order to model different types of entities (e.g., course, video, concept,
and user) and their complex relationships in MOOCs, we resort to considering
the MOOC networks as typical heterogeneous information networks (HINs). First,
we introduce some preliminary concepts about HINs.

##### Heterogeneous information network (HIN)

Let $\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$ denote a graph, which is
composed of a collection of nodes $\mathcal{V}$ and edges $\mathcal{E}$. The
edges in a graph can describe the relationships/connections among nodes. We
refer to the graph as an HIN when the types of nodes and edges are not unique
(Sun and Han, 2012). In our scenario, there are four entities of the HIN in
XuetangX (i.e., course ($\mathcal{C}$), video ($\mathcal{V}$), user
($\mathcal{U}$), and knowledge concept ($\mathcal{K}$)). The relationships
among these entities include learn, click, watch, and has. Figure 3 describes
the schema of network existing in XuetangX MOOC. We can extract the meta-path
from the network schema, which has been frequently utilized to represent the
semantic paths between a pair of entities.

##### Meta-path

A meta-path is a specific path defined based on the network schema. That is a
sequence of relations between two node types, in the form of
$A_{1}\overset{R_{1}}{\rightarrow}A_{2}\overset{R_{2}}{\rightarrow}\cdots\overset{R_{l}}{\rightarrow}A_{l+1}$,
where $A_{i}$ denotes the node type (Sun et al., 2011). In our scenario, we
manually identify four meta-paths as follows to connect users in XuetangX
MOOC, which will also enrich the semantic representation of each user.

  * •

MP1 ($\mathbf{U}_{1}$-$\mathbf{K}$-$\mathbf{U}_{2}$) indicates that if two
users click the same concept, they are considered to be connected.

  * •

MP2
($\mathbf{U}_{1}$-$\mathbf{K}_{1}$-$\mathbf{U}_{2}$-$\mathbf{K}_{2}$-$\mathbf{U}_{3}$)
indicates that the two concepts clicked by users 1 and 2, respectively, are
both clicked by user 3.

  * •

MP3
($\mathbf{U}_{1}$-$\mathbf{C}_{1}$-$\mathbf{K}$-$\mathbf{C}_{2}$-$\mathbf{U}_{2}$)
denotes that users who take the courses that contain the same concepts will be
connected.

  * •

MP4
($\mathbf{U}_{1}$-$\mathbf{C}_{1}$-$\mathbf{V}$-$\mathbf{C}_{2}$-$\mathbf{U}_{2}$)
shows that users are related through paths of courses sharing some common
videos.

We also show these four meta-paths in Figure 4. In Sec. 5.2, we will
investigate the effect of each meta-path and their combinations.

The notations used throughout this paper are listed in Table 1.

Table 1. A list of notations used in this paper. Notation | Description
---|---
$\mathcal{G}$ | the heterogeneous information network
$\mathcal{V}$ | the entity set
$\mathcal{E}$ | the relationship set
$\mathcal{SM}$ | the network schema
$\mathcal{MP}$ | the meta-path set
$\mathcal{P}$ | the set of paths sampled based on meta-paths
$\mathcal{S}$ | the state of MDP
$R_{ss^{\prime}}^{a}$ | the reward obtained when transferring state $s$ to $s^{\prime}$ by taking action $a$
$\gamma$ | the discount factor
$\pi$ | the recommendation policy
$\lambda$ | the weight to regularize the entropy
$\Phi$ | meta-path
${\mathcal{N}}_{i}^{\phi}$ | meta-path-based neighbors of node ${i}$
$\mathbf{h}$ | the initial node feature
$\mathbf{u}$ | the user embedding
$\mathbf{c}$ | the concept embedding

##  3\. Proposed Approach

We describe our proposed methodology in depth in this section, beginning with
an overview of the entire framework. To learn the user embedding, we present
our graph neural network to represent the introduced HIN. Subsequently, we
present our reinforcement learning strategy for recommending concepts to
users.

###  3.1. Overview

Figure 5 depicts the structure of our proposed neural network, which is
composed of three sub-modules. (a) Meta-path sampling (Sec. 3.2.1). In this
module, we first build an HIN among users, courses, and concepts.
Subsequently, we sample meta-paths for each given user using a random walk in
the network. (b) Meta-path-based user embedding with a hierarchical attention
network (Sec. 3.2.2 and Sec. 3.2.3). In this module, a hierarchical attention
network is introduced to embed each math-path into a hidden space. On the
node-level, we employ a self-attention layer to represent the user with its
neighbors. On the path-level, we apply another attention layer to aggregate
semantic representations along each path to the user. (c) Reinforced concept
recommendation (Sec. 3.3). In this module, we propose to recommend concepts to
users within a reinforcement learning framework. Comparing with other HIN-
based representation learning approaches for node classification and
recommendation, this is the first time that the HIN and reinforcement learning
have been combined to recommend concepts to users in MOOCs.

###  3.2. Meta-Path Based User Embedding

We represent the user based on meta-paths. Our intuition is that the
representation of users can be enriched by the information aggregated from
multi-hop neighbors along the meta-paths. For a given user $u_{i}$, we first
collect multi-hop neighbors by meta-path sampling, starting from this node. We
then design a heterogeneous attention network to aggregate the information
along each sampled path.

Figure 5. Overview of the proposed method. (a) Meta-path sampling in the HIN
of MOOCs. (b) Meta-path based user embedding with hierarchical attention
networks. (c) Reinforced concept recommendation.

####  3.2.1. Meta-Path Sampling

To index the network structure of $\mathcal{G}$, we propose to sample paths on
$\mathcal{G}$ based on random walk. Inspired by DeepWalk (Perozzi et al.,
2014), we start random walk from each user in our training dataset
$\mathcal{D}$ for whom a concept will be recommended. Specifically, we first
employ a depth-first search, starting from the user node, to sample instances
under the restrictive scheme of each meta-path, and set a limited walk length
$l$ determining at most how many neighbors would be searched along the sampled
path and controlling the size of its neighborhood. Then we utilize a breadth
search to repeat the aforementioned procedure $N$ times in parallel for each
meta-path, as each path potentially focuses on different input neighbors,
which makes it possible to capture rich semantics. Thus, for each user
$u_{i}\in\mathcal{U}$, we sample $|\mathcal{MP}|\times N$ paths. In total, we
have $|\mathcal{U}|\times|\mathcal{MP}|\times N$ paths over the HIN. We denote
the meta-path as $\mathcal{MP}=\\{\Phi_{1},\Phi_{2},\ldots,\Phi_{P}\\}$ and
denote the path corpus as
$\mathcal{P}=\\{\rho_{\Phi_{1}},\rho_{\Phi_{2}},\ldots,\rho_{\Phi_{P}}\\}$,
where $\rho_{\Phi_{k}}$ denotes a path set sampled based on the meta-path
$\Phi_{k}$.

####  3.2.2. Node-level Attention

In order to obtain the semantic-specific user embedding, we design a module of
node-level attention which first determines the weights of neighbors along a
meta-path, and then combines them. As the node types are heterogeneous and
various node types have different feature spaces, a simple semantic feature
space alignment should be designed. We first project the node features into a
common space. For node $i$, let $\mathbf{h}_{i}$ denote its corresponding
feature. The projected feature $\mathbf{h^{\prime}}_{i}$ can be calculated as
follows:

(1) $\mathbf{h^{\prime}}_{i}=\mathbf{M}_{\phi_{i}}\cdot\mathbf{h}_{i}\,,$

where $\mathbf{M}_{\phi_{i}}$ is the projection matrix of type $\phi_{i}$. We
argue that different nodes for a given meta-path should contribute different
weights to the final user embedding. Thus, an attention mechanism is designed
to calculate the weight of node $i$ to its meta-path-based neighbors
$\mathcal{N}_{i}^{\Phi}$. Formally, for a given meta-path $\Phi$, the node-
level attention weight $\mathbf{a}_{\Phi}$ between nodes $i$ and $j$ can be
calculated as follows:

(2)
$\alpha_{ij}^{\Phi}=\frac{\exp\left(\sigma\left(\mathbf{a}_{\Phi}^{\mathrm{T}}\cdot\left[\mathbf{h}_{i}^{\prime}\|\mathbf{h}_{j}^{\prime}\right]\right)\right)}{\sum_{k\in\mathcal{N}_{i}^{\Phi}}\exp\left(\sigma\left(\mathbf{a}_{\Phi}^{\mathrm{T}}\cdot\left[\mathbf{h}_{i}^{\prime}\|\mathbf{h}_{k}^{\prime}\right]\right)\right)}\,,$

where $||$ denotes the concatenation operator, and $\sigma$ denotes the
activation function. Note that, for the sake of simplicity, we randomly select
one sampled path of meta-path-based neighbors to calculate the node-level
attention. Following (Wang et al., 2019), we then aggregate the node embedding
$\mathbf{u}_{i}$ for node $i$ along a meta-path $\Phi$. Furthermore, the node-
level attention is repeated $L$ times and the attention embeddings are
concatenated through a layer of linear projection. Consequently, the user
embedding $\mathbf{u}_{i}^{\Phi}$ along a meta-path $\Phi$ is formulated as
follows:

(3)
$\mathbf{u}_{i}^{\Phi}=\|_{l=1}^{L}\sigma\left(\sum_{j\in\mathcal{N}_{i}^{\Phi}}\alpha_{ij}^{\Phi}\cdot\mathbf{h}_{j}^{\prime}\right)\,,$

where $||_{l=1}^{L}$ represents the concatenation operation. For a set of
meta-paths $\\{\Phi_{1},\Phi_{2},\ldots,\Phi_{P}\\}$, we denote the user
embedding along the meta-path set as
$\mathbf{u}_{i}^{\Phi_{1}},\mathbf{u}_{i}^{\Phi_{2}},\ldots,\mathbf{u}_{i}^{\Phi_{P}}$.

####  3.2.3. Path-level Attention

Similar to node-level attention, we use attention at the path level. We assign
different weights to meta-paths via path-level attention, to obtain the final
user representation. The importance of meta-path $w_{\Phi_{k}}$ can be
calculated as follows:

(4)
$w_{\Phi_{k}}=\frac{1}{|\mathcal{P}|}\sum_{p=1}^{|\mathcal{P}|}\mathbf{q}^{\mathrm{T}}\cdot\tanh\left(\mathbf{W}\cdot\mathbf{u}_{i}^{\Phi_{k}}+\mathbf{b}\right)\,,$

where $\mathbf{q}$ represents the path-level attention vector and
$|\mathcal{P}|$ denotes the number of meta-path-based sampled paths. That is,
we average all the sampled paths for one meta-path to calculate the path-level
attention, which expresses sophisticated functions beyond the simple meta-
path-based neighbors average. Consequently, the importance of meta-path
$w_{\Phi_{k}}$ can be normalized using a softmax layer, i.e.,
$\beta_{\Phi_{k}}=\frac{\exp\left(w_{\Phi_{k}}\right)}{\sum_{k=1}^{|\mathcal{MP}|}\exp\left(w_{\Phi_{k}}\right)}$.
The final embedding of user $i$ can be reformulated as the following formula,
according to the normalized importance of the meta-path.

(5)
$\mathbf{u}_{i}=\sum_{k=1}^{|\mathcal{MP}|}\beta_{\Phi_{k}}\cdot\mathbf{u}_{i}^{\Phi_{k}}\,.$

The two hierarchical attention modules (i.e., node-level attention and path-
level attention) to assign different weights to the node neighbors and
different paths have been verified in the node classification task in (Wang et
al., 2019, 2021). We have adopted them in this study with a slight
modification. An important difference is that the HIN in our scenario is
dynamically updated according to the feedback from the users. That is, a link
will be assigned if a user clicks a recommended concept.

###  3.3. Reinforced Concept Recommendation

As previously mentioned, to simulate the dynamic interactions between users
and concepts, we frame the concept recommendation within a reinforcement
learning framework. Based on a predetermined loss function, we aim to develop
a model to reduce the discrepancy between the model predictions and immediate
user responses.

We begin by introducing the simplified setting of concept recommendation
within a typical reinforcement learning framework, the goal of which is to
find a recommendation policy $\pi_{\theta}$ that maximizes the expected
reward:

(6)
$\mathcal{L}_{rl}(\theta)=\mathbb{E}_{c_{t}\sim\pi_{\theta}(\cdot|\mathbf{u}_{t})}\sum_{t=1}^{T}r(c_{t}|\mathbf{u}_{t})\,,$

where $r(c_{t}|\mathbf{u}_{t})$ is the immediate reward. Note that, in
contrast to the previous section, the dynamic embedding of user $u$ at time
step $t$ is indicated by the subscript $t$ in $\mathbf{u}_{t}$ in this
section. Given a particular user $u$, if the predicted concept $c_{t}$ is
true, then the reward will be set to 1, otherwise, it will be -1. In
particular, $R(c_{t}|\mathbf{u}_{t})$ is the cumulated reward, which can be
defined as follows:

(7)
$R(c_{t}|\mathbf{u}_{t})=\gamma^{0}R(c_{t}|\mathbf{u}_{t})+\cdots+\gamma^{T-t}R(c_{T}|\mathbf{u}_{T})\,,$

where $\gamma$ is the discount factor, which is set in the range from 0 to 1.
If the recommended concept $c_{t}$ is correct, an edge will be established in
the current HIN to connect the user and the correct recommended concept.
Therefore, we will obtain the new user embedding through the previous HIN
embedding step. If the recommended concept is correct, HinCRec-RL will
continually recommend a new concept, until it incorrectly recommends a concept
or arrives at the max time step $T$.

However, if the recommended concept is incorrect, no new relation is added,
and the structure of HIN $\mathcal{G}$ remains unchanged. Consequently, the
user embedding will remain unchanged. As the Q-learning network (Van Hasselt
et al., 2016) requires the next step transition $s_{t+1}$ of current step
$s_{t}$ to update, if the recommendation is incorrect, then the predicted
Q-value $Q_{t+1}$ will be the same as the target $Q_{t}$, and consequently,
Q-learning cannot be used in our setting. Therefore, we seek to use the policy
gradient to update the model.

We can then learn the optimal policy by using the policy gradient method
called REINFORCE (Williams, 1992; Sutton et al., 2000), a popular learning
approach in reinforcement learning. The gradient of the expected cumulative
reward can be determined using the following formula, according to the policy
gradient method:

(8)
$\nabla_{\theta}\mathcal{L}_{rl}(\theta)=\sum_{t=1}^{T}\left[\nabla_{\theta}\log\pi_{\theta}(c_{t}|\mathbf{u}_{t})R(c_{t}|\mathbf{u}_{t})\right]\,.$

In practice, we often observed a collapse onto a suboptimal deterministic
policy. To prevent the model from not being able to explore new concepts that
could lead to a better recommendation, we add an entropy regularization term
to the objective function, as follows:

(9)
$\mathbf{H}[\pi_{\theta}(c|\mathbf{u})]=\sum_{t=1}^{T}\sum_{c_{t}\in\mathcal{C}}\log\pi_{\theta}(c_{t}|\mathbf{u}_{t})\pi_{\theta}(c_{t}|\mathbf{u}_{t})\,.$

Therefore, the ultimate goal of our model is reformulated as follows:

(10)
$\mathcal{J}=\mathbb{E}_{c\sim\pi_{\theta}(\cdot|\mathbf{u})}\mathcal{L}_{rl}(\theta)+\lambda\mathbf{H}[\pi_{\theta}(c|\mathbf{u})]\,.$

Here, $\lambda$ is the regularization weight. The policy gradient directly
updates policy $\pi_{\theta}$ to increase the probability of $c_{t}$ given
state $s_{t}$ when the reward is positive, and vice versa.

Algorithm 1 depicts the HinCRec-RL algorithm with reinforcement learning.
Given the training set $\mathcal{U}_{train}$, clicking data $R$, the number of
time steps $T$, the number of episodes $E$, discount factor $\gamma$, and
$\epsilon$-greedy parameter $\epsilon$, our goal is to learn the
recommendation policy $\pi_{\theta}$. We first conduct meta-path sampling and
initialize the recommendation policy $\pi_{\theta}$, which will be used later.
At the reinforcement learning phase, for each episode, we first pick a user
and learn the corresponding embedding based on Eq. 5 (see line 4-6). We then
dynamically update the policy through reinforcement learning (line 8-15).

###  3.4. Time Complexity of HinCRec-RL

Here, we analyze the time complexity of HinCRec-RL, as shown in Algorithm 1.
The total complexity for a training episode is
$O({L}\sum_{{v}\in{V}}{F}_{v}{F}_{1}+{L}{T}\sum_{i=1}^{|\mathcal{MP}|}(|{{E}_{{\Phi}_{i}}}|{F}_{1}+|{V}_{{\Phi}_{i}}|{F}_{1}{F}_{2}))$,
where $L$ is the number of attention heads, $T$ is the number of time steps of
an episode, ${|\mathcal{MP}|}$ is the number of hand-crafted meta-paths, and
${F}_{v}$ is the input dimension of the specific type node. ${F}_{1}$ and
${F}_{2}$ are the output dimensions of node type projection and path-specific
transformation, respectively. $|{{E}_{{\Phi}}}|$ and $|{{V}_{{\Phi}}}|$ are
the numbers of sampled-path-based node pairs and user nodes, respectively.
More specifically, the time complexity of a single node-level attention head
can be expressed as
$O(|{V}|+|{E}|)=O(\sum_{{v}\in{V}}{F}_{v}{F}_{1}+|{{E}_{\Phi}}|{F}_{1})$,
including input feature transformation, the computation of sampled path
attention coefficients, and node-level aggregation. The time complexity of the
path-level attention can be expressed as
$O(|{V}|)=O({|{V}_{\Phi}|}{F}_{1}{F}_{2})$, including path-specific
transformation and meta-path based sampled path aggregation. Note that we only
need to pick one user for each episode uniformly and recompute its user
embedding iteratively during sequential recommendation training with
reinforcement learning. Within an episode, the action space of
$\epsilon$-greedy influences the number of time steps, whereas the requirement
to converge is correlated to the action space. Thus, HinCRec-RL is highly
computationally efficient, and the total complexity is linearly proportional
to the number of nodes and meta-path-based node pairs.

1

Input : Training set $\mathcal{U}_{train}$, clicking data $R$, number of time
steps $T$, number of episodes $E$, discount factor $\gamma$, and
$\epsilon$-greedy parameter $\epsilon$.

Output : The learned recommendation policy $\pi_{\theta}$.

2 Meta-path sampling

3Initialize recommendation policy $\pi_{\theta}$ with random weights

4for  _episode $\leftarrow 1$ to $E$_ do

5 Initialize the parameters of HIN $\mathcal{G}$ and user embeddings
$\mathcal{U}_{train}$

6 Uniformly pick a user $u_{0}\in\mathcal{U}_{train}$

7 Learn the user embedding $\mathbf{u}_{1}$ by Eq. 5

8 ${t}\leftarrow{1}$

9 while  _$c_{t}$ is correct and $t\leq T$_ do

10 Select the concept $c_{t}$ using $\epsilon$-greedy policy w.r.t
$\pi_{\theta}$, observe reward $R(c_{t}|\mathbf{u}_{t})$

11 if  _$c_{t}$ is correct_ then

12 Add a new link between the user $u_{t}$ and concept $c_{t}$ on the HIN
$\mathcal{G}$

13 Recompute the user embedding $\mathbf{u}_{t+1}$ by Eq. 5

14 end if

15 ${t}\leftarrow{t+1}$

16 end while

17 Update $\pi_{\theta}$’s weights $\theta$ according to Eq. 10

18 end for

Algorithm 1 HinCRec-RL

##  4\. Experimental Setup

We carry out experiments by addressing the following research issues to assess
the performance of our proposed model HinCRec-RL.

  * •

RQ1: How effective is the proposed model HinCRec-RL for concept recommendation
when compared with baseline methods?

  * •

RQ2: How do various involved meta-paths and their combinations affect user
embedding?

  * •

RQ3: How dose the proposed model HinCRec-RL perform when the hyper-parameters
are changed?

###  4.1. Experimental Dataset

To conduct our experiments, we construct training and test datasets using the
data from XuetangX MOOC platform, where the training set spans from 1st
October, 2016 to 30th December, 2017, whereas the test set spans from 1st
January, 2018 to 31st March, 2018. We pair a target course with all historical
enrolled courses to form a sequence which is further considered as an instance
of our datasets. For each instance when conducting the training phase, we
consider the final course in the sequence to be the target course and the
remaining to be the historical courses. When conducting the test process, for
a user, a target concept in the test set indicates a concept that must be
involved in the enrolled courses, and historical concepts indicate the
corresponding concepts to the same user in the training set. On the other
hand, the same concept label is usually clicked by a user, and thus, multiple
clicked concept records are generated. In our experiments, we serialize these
records into a single one. We then paired each positive instance with a random
sample of 99 negative ones in the test set.

Table 2. The description of XuetangX datasets. Nodes |  Count |  Links |  Count
---|---|---|---
concept |  2,527 |  concept-course |  21,507
concept-video |  11,732
user |  3,111,637 |  user-course |  15,045,219
user-video |  53,481,869
course |  7,327 |  course-concept |  69,012
course-video |  811,841
course-user |  16,724,852
video |  62,191 |  video-course |  247,433
video-concept |  11,732
video-user |  53,971,707
Total |  3,183,682 |  Total |  140,096,904

Overall, the obtained dataset contains 62,191 videos, 7,327 courses, 3,111,637
users, 2,527 concepts and 140,096,904 relations between these entities. The
description of this dataset is presented in Table 2.

###  4.2. Evaluation Metrics

We pick several metrics commonly used in recommender systems to evaluate the
proposed model. They are Mean Reciprocal Rank ($MRR$), Normalized Discounted
Cumulative Gain of top-K items ($NDCG@K$) and Hit Ratio of top-K items
($HR@K$). We choose 5, 10 and 20 for $K$ in our experiments.

$HR@K$ is used to measure the proportion of ground-truth instances which are
correctly recommended among the top-$K$ items. This is a recall-based metric
and can be formulated as

(11) $HR@K=\frac{{\\#Hits}@K}{|GT|}\,,$

where $|GT|$ is the size of the test dataset. $HR@K$ measures the percentage
of successes in prediction to the number of attempts. $NDCG$ is usually
applied to assess a retrieval syetem’s performance when considering the graded
relevance of retrieved entities. Thus, it is also used as another ranking
metric in our experiments and can be defined as

(12)
$NDCG@K=\frac{1}{\left|Q\right|}\sum_{q=1}^{\left|Q\right|}Z_{kq}\sum_{j=1}^{k}\frac{2^{r(j)}-1}{\log(1+j)}\,,$

where $r(j)$ denotes a relevance score calculated for a retrieved entity for
query $q$. $Z_{kq}$ indicates a normalization factor and is usually used to
ensure that the $NDCG$ for a perfect ranking at $k$ for query $q$ is $1$.
$MRR$ is used to measure the ranking of our searched results and can be
defined as

(13) $MRR=\frac{1}{|Q|}\sum_{i=1}^{|Q|}\frac{1}{rank_{i}}\,,$

where $|Q|$ indicates the size of a query set.

Moreover, we leverage Area Under the ROC curve ($AUC$) to measure the quality
of recommendation ranking.

Table 3. Quantitative results (%) of experiments on XuetangX MOOCs datasets.
(Best scores are in boldface.)

Model | HR@5 | HR@10 | HR@20 | NDCG@5 | NDCG@10 | NDCG@20 | MRR | AUC
---|---|---|---|---|---|---|---|---
BPR | 41.50 | 56.28 | 72.12 | 28.95 | 33.74 | 37.76 | 28.68 | 84.34
MLP | 32.75 | 52.66 | 69.88 | 17.67 | 25.32 | 30.71 | 18.34 | 82.73
FM | 36.03 | 54.40 | 72.39 | 23.99 | 29.89 | 34.44 | 24.46 | 83.20
FISM | 36.41 | 54.31 | 70.79 | 20.15 | 27.73 | 33.83 | 21.65 | 84.24
NAIS | 36.47 | 57.92 | 77.99 | 21.93 | 28.10 | 34.12 | 22.15 | 86.76
NASR | 38.67 | 53.21 | 67.13 | 21.19 | 27.84 | 32.93 | 19.69 | 79.90
HinCRec-SL | 60.73 | 78.41 | 90.04 | 40.89 | 46.62 | 49.60 | 37.85 | 92.10
HinCRec-RL | 66.21 | 81.53 | 88.74 | 46.21 | 51.35 | 53.22 | 42.82 | 89.64

Table 4. Different results (%) from different combinations of meta-paths.
(Best scores are in boldface.)

Meta-Path | HR@5 | HR@10 | HR@20 | NDCG@5 | NDCG@10 | NDCG@20 | MRR | AUC
---|---|---|---|---|---|---|---|---
MP1 | 63.51 | 75.23 | 90.99 | 43.08 | 46.89 | 50.90 | 39.24 | 91.89
MP2 | 45.33 | 61.68 | 79.44 | 29.59 | 34.90 | 39.43 | 28.44 | 86.92
MP3 | 46.15 | 67.42 | 84.62 | 30.37 | 37.27 | 41.66 | 29.76 | 88.69
MP4 | 50.68 | 69.86 | 86.76 | 34.32 | 40.53 | 44.78 | 33.01 | 91.32
MP1 & MP2 | 63.47 | 77.17 | 89.04 | 44.33 | 48.83 | 51.81 | 41.04 | 93.15
MP1 & MP3 | 60.55 | 76.15 | 87.61 | 43.37 | 48.42 | 51.39 | 40.96 | 93.12
MP1 & MP4 | 65.02 | 78.48 | 88.34 | 45.69 | 50.02 | 52.48 | 42.15 | 89.69
MP2 & MP3 | 46.98 | 68.37 | 81.86 | 29.91 | 36.78 | 40.23 | 28.64 | 89.30
MP2 & MP4 | 53.02 | 73.02 | 89.30 | 35.68 | 42.09 | 46.20 | 33.94 | 92.09
MP3 & MP4 | 47.44 | 65.58 | 84.65 | 32.30 | 38.16 | 42.97 | 31.56 | 86.51
MP1 & MP2 & MP3 | 57.73 | 75.45 | 89.55 | 41.06 | 46.86 | 50.47 | 39.26 | 91.82
MP1 & MP2 & MP4 | 59.35 | 77.23 | 89.76 | 43.24 | 49.12 | 51.38 | 41.36 | 89.12
MP1 & MP3 & MP4 | 61.26 | 80.82 | 88.58 | 44.73 | 50.90 | 52.84 | 42.37 | 89.04
MP2 & MP3 & MP4 | 38.01 | 60.18 | 80.09 | 23.87 | 30.93 | 35.85 | 24.00 | 90.04
MP1 & MP2 & MP3 & MP4 | 66.21 | 81.53 | 88.74 | 46.21 | 51.35 | 53.22 | 42.82 | 89.64

###  4.3. Implementation Details

The proposed model is trained on an Nvidia GeForce GTX1080Ti GPU with 11 GB of
RAM. The dataset is split into two portions for our evaluation tasks. The
first portion contains 80% instances for training and the second portion
contains 20% instances for testing. We choose 64 as the dimension size of the
final user embedding. First, a cross-entropy loss function with 10,000
episodes is leveraged to pre-train the model. The learning rate is set to
0.001, and the mini-batch size is set to 8. Subsequently, the learning rate is
reduced to 0.0001 and the loss function is adjusted to the policy gradient.
Furthermore, the weight of the regularization term $\lambda$ is set to 0.08 to
prevent the model from overfitting. The sequence length of recommended
concepts is automatically maintained consistent with the mini-batch size.
Through the entire training of the HinCRec-RL model, we update the parameter
using the Adam optimizer (Kingma and Ba, 2015).

##  5\. Experimental Results

In this section, we respond to the aforementioned research issues through
conducting experiments and analyzing the results.

###  5.1. Comparison with Baselines (RQ1)

We compare the proposed model with the following baselines in order to
evaluate its performance.

  * •

BPR (Rendle et al., 2009) obeys a Bayesian method to minimize the pairwise
ranking loss for recommendation tasks.

  * •

MLP (He et al., 2017) utilizes a multi-layer perceptron to compute the
recommendation probability by leveraging embedding pairs of user and concept.

  * •

FM (Rendle, 2012) is a typical matrix decomposition method that can easily
address the problem of feature combination. For fairness, we use only the
embeddings of users and concepts in our experiments.

  * •

FISM (Kabbur et al., 2013) is a type of CF algorithm. It can utilize the
average historical concept embedding and the target one to perform
recommendation tasks.

  * •

NAIS (He et al., 2018) is another type of CF algorithm. It can differentiate
among the importance of all historical concepts by utilizing the attention
mechanism.

  * •

NASR (Li et al., 2017) is considered an improved GRU model, and can calculate
an attention coefficient for each historical concept by using the
corresponding hidden vectors produced by GRU.

  * •

HinCRec-SL represents the proposed model that only applies an HIN for user
embedding, without reinforcement learning.

  * •

HinCRec-RL represents the proposed model that combines the HIN for user
embedding and the reinforcement learning for better recommendation.

We conduct comparative experiments between our proposed HinCRec-RL model and
several baseline ones using offline datasets. According to the comparative
experiments results reported in Table 3, HinCRec-RL largely outperforms all
other baselines in terms of $HR@K$, $NDCG@K$, $MRR$ and $AUC$. We also compare
the HinCRec-SL/RL model with these baselines and observe that the proposed HIN
contains more useful semantic information of users. When comparing HinCRec-SL
with HinCRec-RL, we can observe that the proposed reinforcement learning also
has a positive effect on the performance based on supervised learning. For
example, in terms of $HR@5$ and $NDCG@5$, HinCRec-RL is more than 10% better
than HinCRec-SL. This can be illustrated by the fact that the exploration
mechanism in reinforcement learning can yield a better optimized result with
our model. More specifically, HinCRecRL achieves an $HR@5$ of 66.21%, an
$HR@10$ of 81.53%, an $HR@20$ of 88.74%, an $NDCG@5$ of 46.21%, an $NDCG@10$
of 51.35%, an $NDCG@20$ of 53.22, and an $MRR$ of 42.82% on the the dataset of
XuetangX MOOCs. Although HinCRec-RL performs slightly worse than HinCRec-SL in
terms of $HR@20$, it shows significant improvements when compared with the
baselines (i.e., 9.74%-27.54% increment in terms of $HR@5$, $NDCG@5$, $MRR$
and $AUC$). We attribute such performance enhancements to the fantastic
HinCRec-RL modeling: (1) The technique based on the hierarchical graph
attention aggregation guided by the user’s meta-paths is capable of encoding
the interrelationship between concepts and users, hence providing a richer and
more comprehensive representation learning of concepts and users. (2)
Benefiting from the representation learning, HinCRec-RL can preserve the
user’s state information and collect more informative rewards from the
reinforcement learning process. (3) HinCRec-RL first combines the HIN and
reinforcement learning to recommend knowledge concepts for users’ course
learning in MOOC platforms. In contrast, all the baselines ignore this
strategy.

###  5.2. Effect of Different Meta-Paths (RQ2)

We examine various meta-paths and their combinations to demonstrate the
effectiveness of these heterogeneous graph patterns in improving the
performance of HinCRec-RL. More specifically, to describe the relatedness
among pairs of users, four kinds of meta-paths are chosen for our study. These
meta-paths are $MP_{1}\text{: }U_{1}\text{-}K_{1}\text{-}U_{1}$,
$MP_{2}\text{: }U_{1}\text{-}K\text{-}U_{2}\text{-}K\text{-}U_{3}$,
$MP_{3}\text{: }U_{1}\text{-}C_{1}\text{-}K\text{-}C_{2}\text{-}U_{2}$, and
$MP_{4}\text{: }U_{1}\text{-}C_{1}\text{-}V\text{-}C_{2}\text{-}U_{2}$. The
combination of some single or simple meta-paths (e.g., $MP_{1}$) may affect
our model’s performance. To illustrate this, we conduct related experiments
and show the results in Table 4. Each single meta-path (i.e., $MP_{1}$,
$MP_{2}$, $MP_{3}$, $MP_{4}$) and the combinations thereof contribute
different performances to the proposed model. Thus, we can conclude that
different single meta-paths can result in nonnegligible effects on HinCRec-
RL’s performance. It is apparent that the adjacent combination of single or
simple meta-paths has a positive correlation property. Among the four meta-
paths, $MP1$ can achieve a high score in several metrics; thus, it can be
considered the best designed meta-path for user representation. This is
appropriate since the interests of users are usually affected by their
neighbors. Moreover, the more meta-paths the combinations include, the better
is the model performance. The most convincing evidence to illustrate this
aspect is that the combination of all the four meta-paths causes the best
performance of the proposed model.

###  5.3. Parameter Analysis (RQ3)

(a) $L$ on $HR@K$

(b) $L$ on $NDCG@K$

(c) $L$ on $MRR$

(d) $L$ on $AUC$

Figure 6. Parameter sensitivity of HinCRec-RL over attention head $L$ in HIN
embedding.

(a) $lr$ on $HR@K$

(b) $lr$ on $NDCG@K$

(c) $lr$ on $MRR$

(d) $lr$ on $AUC$

Figure 7. Parameter sensitivity of HinCRec-RL over learning rate $lr$.

(a) $d$ on $HR@K$

(b) $d$ on $NDCG@K$

(c) $d$ on $MRR$

(d) $d$ on $AUC$

Figure 8. Parameter sensitivity of HinCRec-RL over embedding dimension $d$ in
HIN embedding.

(a) $\lambda$ on $HR@K$

(b) $\lambda$ on $NDCG@K$

(c) $\lambda$ on $MRR$

(d) $\lambda$ on $AUC$

Figure 9. Parameter sensitivity of HinCRec-RL over $\lambda$.

(a) $\epsilon$ on $HR@K$

(b) $\epsilon$ on $NDCG@K$

(c) $\epsilon$ on $MRR$

(d) $\epsilon$ on $AUC$

Figure 10. Parameter sensitivity of HinCRec-RL over the probability of
exploration $\epsilon$.

We study parameter sensitivity and present the results of HinCRec-RL for
diverse parameters in Figures 6-9.

####  5.3.1. Impact of Attention Head $L$ in HIN Embedding

We analyze the performance of HinCRec-RL with varying numbers of attention
heads, proving the validity of multi-head attention involved in HIN
embeddings. In sequence, there are 2, 4, 6, 8, 10 and 12 attention heads in
our analytical design. The experimental results are given in Figure 6. We can
see that the increase of the number of heads slightly elevates the model’s
performance. Although the attention mechanism included in our model requires
slightly longer training time, the results prove that it can make our model
more reliable and robust.

(a) Case 1

(b) Case 2

Figure 11. A case of the diversity of concept recommendation, which contains a
sequence of clicked concepts within XuetangX’s dynamic learning environment.

####  5.3.2. Impact of Learning Rate $lr$

We tune $lr$ to learn a recommendation policy better. The impact of $lr$ on
HinCRec-RL is described in Figure 7. We can see that, when increasing the
learning rate from $10e\text{-}6$ to $10e\text{-}5$, the performance is
improved in terms of the $HR@K$, $NDCG@K$, and $MRR$, whereas the performance
is degraded when increasing $lr$ further. Therefore, we can conclude that the
learning rate affects HinCRec-RL. The best $lr$ is approximately
$10e\text{-}5$.

####  5.3.3. Impact of Embedding Dimension $d$

Aiming to investigate the embedding dimension settings of entities, we conduct
experiments with various numbers of dimensions (i.e., 32, 64, 96 and 128). The
experiment results are shown in Figure 8. When increasing $d$ from 32 to 64,
the performance is improved in terms of $HR@K$, $NDCG@K$, and $MRR$, whereas
the performance is degraded when increasing $d$ further. Our final settings
have the user and knowledge concept vector dimension set to 64. In light of
the experiment findings, we can conclude that a reasonable embedding dimension
is crucial to enhance recommendation performance in our task.

####  5.3.4. Impact of Regularization Rate $\lambda$

We conduct experiments on entropy regularization to investigate the efficacy
of the parameter $\lambda$. We set $\lambda$ to 0.01, 0.1, 1 and 10, and show
the corresponding experimental results in Figure 9. In addition, for more
fine-grained analysis, we set $\lambda$ to 0.02, 0.04, 0.06 and 0.08, and
present the results in a zoomed view. Figure 9 shows that, when increasing
$\lambda$ from $10e\text{-}2$ to $10e\text{-}1$, the performance is improved
in terms of the $HR@K$, $NDCG@K$, $MRR$ and $AUC$, whereas the performance is
degraded when increasing $\lambda$ further. Therefore, we can conclude that
$\lambda$ affects the performance of HinCRec-RL, and the best $\lambda$ is
approximately $10e\text{-}1$.

####  5.3.5. Impact of Exploration Probability $\epsilon$

We investigate the random action probability in an $\epsilon$-greedy policy.
We set the range of $\epsilon$ range from 0 to 1 in steps of 0.1, and present
the corresponding results in Figure 10. When increasing $\epsilon$ from 0 to
0.2, the performance is improved in terms of the $HR@K$, $NDCG@K$, $MRR$ and
$AUC$, whereas the performance is degraded when increasing $\epsilon$ from 0.2
to 1 further. The thumbnail of Figure 10 shows that HinCRec-RL performs
optimally when $\epsilon$ is approximately 0.18. Therefore, the exploration
probability affects the performance of HinCRec-RL.

###  5.4. Case Study

Figure 11 shows a case study to examine the availability of HinCRec-RL,
particularly its ability to adapt to dynamic environments when providing
personalized concept recommendations. User A (ID: 6806276) and User B (ID:
8136117) are randomly chosen from the testing dataset. They have different
educational backgrounds and consequently experience different learning paths
in the XuetangX MOOC platform. Figure 11(a) first shows the clicked history of
the two users and then displays the concept lists recommended to them
separately. From this figure, we can observe that User A is learning a course
named “Introduction to Algorithms”, whereas User B is learning a course named
“Natural Language Processing”. Both users selected distinct knowledge concepts
pertaining to the same target course “Operating System”. In this case, our
model can still adaptively consider the currently selected concepts, and
creatively recommend the next one depending on the past. We attribute it to
the reinforcement learning mechanism we have adopted, i.e., the proposed
approach can dynamically interact with the environment.

On the other hand, we also show a case of recommending different concept lists
for a given user according to different meta-paths, as shown in Figure 11(b).
For User C with ID 8982675, given his click history, we show the recommended
results by meta-paths $MP1\&MP2\&MP3\&MP4$ and $MP1$, respectively. The result
recommended by $MP1\&MP2\&MP3\&MP4$ is more precise than that recommended by
$MP1$. We attribute this to the fact that the meta-paths can aggregate the
information related to users from neighbors, as data augmentation. This also
demonstrates that introducing heterogeneous graph information into user
embedding is effective.

##  6\. Related Work

We will present the relevant studies from three aspects, namely, mining in
MOOCs, HIN learning, and recommender systems.

###  6.1. Mining in MOOCs

MOOCs can be viewed as interactive systems containing large-scale multimedia
objects, user interaction records, and HINs. Several data mining techniques
have been employed to mine these plentiful resources, to improve the user
experience on online learning platforms. These data can also be exploited to
reveal, in detail, how students behave and how learning takes place. Existing
studies on MOOCs fall under two major categories. One line of work focused on
the course content, such as course concept extraction (Pan et al., 2017b; Lu
et al., 2019), course concept expansion (Yu et al., 2019b), and prerequisite
relation discovery (Pan et al., 2017a; Yu et al., 2020b). It is common to
learn how the representations of entities leverage both context information
and content. In addition to the title and description of each course, other
types of entities, such as teachers, videos, and concepts, can be constructed
into heterogeneous networks to explore more corresponding beneficial semantic
relationships. Pan et al. (2017a) explored the prerequisite relations among
concepts by leveraging the concept semantics of course , the video context of
course and the structure of course. The other line of work pays attention to
the learning behavior patterns of students (Qiu et al., 2016). For example,
some studies (Jing and Tang, 2017; Zhang et al., 2019) recommended courses to
students. Additionally, some studies (Feng et al., 2019; Liao et al., 2019)
predicted dropouts, whereas other studies (Karimi et al., 2020) predicted the
performances of students. Diverse students’ behaviors are essential for MOOC
research because they are meaningful to mine the motivations (Feng et al.,
2019), cognitive abilities (Chen et al., 2018a), and social activities (Ke et
al., 2021) of students static course research and contents. Yu et al. (2021)
collected student behaviors data from MOOCs, which indicates the relationships
between courses and concepts. These records include student profiles, video
watching behaviors, exercise behaviors, comments and replies.

Among the related studies, course recommendation is the task most relevant to
ours. Our study not only deploys concept recommendation in the dynamic
interactive environment, but also puts more focus on modeling the dynamic
interaction among users and concepts, and enriching the representation of
users based on heterogeneous information networks. Another related work to
ours is (Gong et al., 2020), which proposed an attentional graph convolutional
network for concept recommendation. Different to (Gong et al., 2020), we
resort to modeling the dynamic interactions among users and concepts by
reinforcement learning.

###  6.2. Learning with Heterogeneous Information Networks

Many applications, e.g recommender systems (Hu et al., 2018a), social event
detection systems (Peng et al., 2021a; Cao et al., 2021), etc, capture the
meaningful semantic and structural information by constructing an HIN in
auxiliary data. Specially, lots of entity types (e.g., users, courses, and
knowledge concepts) and rich edges among them which naturally constitute an
HIN, can be easily found in E-commerce applications or online platforms (e.g.,
Amazon, eBay and XuetangX MOOC).

HIN-based methods have outperformed the other previous methods in mining more
meaningful information owing to their excellent ability. Sun et al. (2011)
suggested using meta-paths to explore the information network’s meta-structure
which can systematically explore fruitful semantic relationships across
several kinds of objects. Metapath2vec (Dong et al., 2017) generated the meta-
path by utilizing skip-gram and random walk to represent the HIN. Cai et al.
(2018) proposed GAN-HBNR which is based on generative adversarial
bibliographic network module for citation recommendation. Cen et al. (2019)
introduced an embedding method for the attributed multiplex heterogeneous
network to boost the behavior of the recommender system. Chen et al. (2018b)
proposed a novel metric learning based heterogeneous information network
embedding model named PME which uses a unified technique to capture 1st-order
and 2nd-order proximities. HAN (Wang et al., 2019) introduced the attention
mechanism into an HIN and achieved a comparable performance in node
classification and clustering. Sun et al. (2021) presented a heterogeneous
hypergraph learning model based on graph neural networks to characterize
multiple non-pairwise relations. Furthermore, the attention mechanism has been
adopted in several approaches, such as HERec (Shi et al., 2019) and MCRec (Hu
et al., 2018b).

MOOCs are more typical for utilizing the emerging HINs because they contain
richer links and semantics in objects, and are thus a fresh direction for data
mining. We construct a framework by combining extended deep reinforcement
learning with the representations of user status to provide recommendations of
relevant knowledge concepts in this study.

###  6.3. Recommender System

Recent recommendation works can be grouped approximately into four categories
(Park et al., 2012; Aggarwal et al., 2016; Liu et al., 2022b): collaborative-
filtering (CF) based recommendation, content-based (CB) recommendation,
conventional hybrid recommendation, and deep-learning-based recommendation.

####  6.3.1. CF-based Recommendation.

CF has become one of the most efficient recommendation algorithms because of
its simplicity and efficiency. It aims to recommend items on the basis of
historical data of user–item interactions (Aggarwal et al., 2016). Recommender
systems rely on either explicit or implicit feedback, which contains abundant
information about the interests of users (Yu et al., 2020a). There are two
kinds of algorithms deriving from CF, namely user-based CF (UserCF) and item-
based CF (ItemCF) (Ghazarian and Nematbakhsh, 2015). The major challenges of
CF approaches are data sparsity issues (Yin et al., 2020) and the cold-start
problem (Zhu et al., 2019). In order to alleviate these problems in MOOC
course recommendation, Pang et al. (2017) proposed MLBR, which uses learner-
course vectors rather than a huge and sparse learner-course matrix. Rabahallah
et al. (2018) combined ontology and CF to recommend personalized courses to
online students, where ontology can provide a semantic description of students
to mitigate the cold-start problem.

####  6.3.2. Content-based Recommendation.

Compared with CF, CB prioritizes the ratings of the target users and the
attributes of items liked by them in the past. Currently, several CB
approaches have been proposed for course recommendations on MOOCs. Huang and
Lu (2018) utilized feature modeling for items and user profiles constructing
and matching the item features with the user’s preferences. In comparison to
CF, the CB suggestion does not require a substantial rating matrix and offers
the benefits of user independence and the capacity to suggest new goods.
Although CB approaches are significantly better at proposing new things, they
cannot yet deliver services to new system users.

####  6.3.3. Hybrid Recommendation.

Hybrid models seek to get the best of both worlds by combining both the basic
and the enhanced methods (Aggarwal et al., 2016). They can incorporate the
advantages of different types of recommendation methods to create a unified
method that is more robust in a wide range of contexts. Typically, basic
models contain CF and CB methods, whereas enhanced ones include factorization
machine (FM) (Loni et al., 2019; Meng et al., 2022), GBDT (Ke et al., 2017),
and other combined variants. Classical methods include GBDT+LR (He et al.,
2014) and LS-PLM (Gai et al., 2017). The GBDT+LR model has the ability to
perform higher-order feature combination. It promotes the trend of feature
engineering modeling. The LS-PLM model is a three-layer neural network with an
attention mechanism, which has a stronger expression. Several related studies
leverage hybrid models for online course recommendation (Khalid et al., 2022;
Chen et al., 2014). Wan and Niu (2019) proposed an approach called SI-IFL for
recommending courses to learners, which integrates sequential pattern mining,
the learner influence model, and self-organization-based recommendation
strategy. Although deep learning has demonstrated considerable potential for
learning effective representations, traditional recommendation models still
retain irreplaceable advantages owing to their strong interpretability and
fast training speed. In general, there are complicated relationships between
traditional recommendation models and deep learning models.

####  6.3.4. Deep-learning-based Recommendation.

Compared with traditional recommendation methods, deep learning recommendation
methods have deeper neural network layers, and hence, they can explore more
complex connections between users and items.

Graph neural networks (GNNs), particularly dynamic GNNs (Peng et al., 2020),
graph convolutional networks (GCNs) (Welling and Kipf, 2016), graph inductive
representation learning (GraphSAGE) (Hamilton et al., 2017), graph attention
networks (Veličković et al., 2018), and graph isomorphism networks (Xu et al.,
2019), have gained considerable attention. The main idea behind GNN is to
aggregate the neighbor features of nodes by introducing convolution. For
example, MixGCF (Huang et al., 2021a) introduces negative samples by using
both positive mixing and hop mixing methods, which leverages the information
of several local graphs to train GNN-based CF recommendation models. SHCF (Li
et al., 2021) is a novel sequence-aware graph-based CF method for
recommendation that combines both high-order collaborative messages and
sequential patterns. Deng et al. (2020) predicted paper review rating via a
hierarchical network (HabNet) that leverages a bidirectional self-attention
network. Fan et al. (2019) proposed a heterogeneous graph network through a
meta-path-guided method to model complex entities and rich edges for intent
recommendation. Jin et al. (2020) proposed an interaction model considering
the information of neighborhoods for recommendation in heterogeneous views.
Another study (Zhang et al., 2021) presented a comprehensive analysis on
network embedding methods for recommendation. Further, the interpretability
(Chen et al., 2021) and privacy of recommender systems (Bosri et al., 2021;
Omar et al., 2019) have been investigated.

Recently, reinforcement learning has been widely utilized in several fields;
for example, OpenAI’s AlphaStar (Vinyals et al., 2019) defeated professional
players in StarCraft II. Silver et al. (2013) proposed a reinforcement
learning method to build a system for customer interaction. Peng et al. (2022)
employed a multi-agent RL algorithm to select optimal aggregation strategies
in a multi-relational GNN for social event detection tasks. Especially in
recommender systems, as a method that focuses on how the agents act in the
environment, reinforcement learning enables the recommender to capture the
real-time feedback of users effectively for learning optimal recommendation
strategies. In practice, several studies have begun to regard recommendation
as a deep reinforcement learning problem, and have achieved comparable
performances in movie recommendation (Zhao et al., 2019; Liu et al., 2018),
news recommendation (Zheng et al., 2018; Shen et al., 2018), music
recommendation (Pan et al., 2019), e-commerce (Zhao et al., 2018), and health
care (Zhang et al., 2017).

More specifically, Zhou et al. (2020) proposed a knowledge-graph-enhanced RL
method on an interactive recommender system to deal with the sample efficiency
issues. Chen et al. (2020) introduced the attention method to aggregate the
state vector of knowledge graphs and then calculated the Q value in the
user–item interest graph through graph convolution. Zou et al. (2019)
introduced a novel Q network to explore user’s long-term interest. On this
basis, they also proposed a RL framework (Zou et al., 2020) to guarantee
stable convergence and low time complexity. Huang et al. (2021b) proposed a
method using RNN and RL to improve the accuracy of long-term recommendations.
Yu et al. (2019a) designed VL-Rec, a vision-language recommendation framework
with an attribute-augmented RL, which provides natural language feedback to
perform effective interactions with the user. Zhao et al. (2020b) proposed
MaHRL based on hierarchical RL for recommendation. The low-level agents learn
the short-term preferences of users while the high-level agents focus on
learning the uses’ long-term preferences and guide the low-level agents. Xie
et al. (2021) addressed the integrated recommendation task which also employed
a hierarchical model. Zhang et al. (2019) removed noisy courses from
historical course sequences and then performed hierarchical reinforcement
learning. Feng et al. (2018) and Zhao et al. (2020a) combined two
recommendation scenarios which contain the homepage and product detail page
into one recommendation target on an e-commerce platform. Peng et al. (2021b)
proposed to address the complexity of networks by designing a novel
neighborhood-selection-guided multi-relational GNN, which maintains relation-
dependent representations simultaneously.

Different from existing recommendation methods based on reinforcement learning
(Pan et al., 2019; Chen et al., 2019), our model leverages rich semantic meta-
path-based context, which can learn better representations by specific
interactions for students, courses, concepts, items, etc.

##  7\. Discussion

In this section, we discuss our study from three aspects: comparison with our
previous model, verifying the effectiveness our model, and presenting the
threats to validity.

###  7.1. Comparison with Our Previous Work

We revisit our previous model ACKRec (Gong et al., 2020) for knowledge concept
recommendation, and compare it with HinCRec-RL.

##### Attention-based GCNs versus Hierarchical Attention Networks.

By combining information from neighbors based on meta-paths, our models can
learn the node representation in heterogeneous graphs. In node-level
aggregation, we use information propagation in GCNs within meta-path-based
neighbors in ACKRec, which sets the symmetric normalization term between two
nodes as ${1/\sqrt{|\mathcal{N}_{i}||\mathcal{N}_{j}|}}$. This encodes the
prior that the intermediate representation of node
$j\in{\mathcal{N}_{i}^{{\Phi}_{k}}}$ to node $i$ under a single meta-path
${\Phi}_{k}$ exploits the proximity structure of the graph. HinCRec-RL, in
contrast to GCN, uses self-attention to learn the importance of
${\alpha}_{ij}^{{\Phi}_{k}}$ among meta-path-based neighbors. By assigning
weights to the meta-path-based neighbors, HinCRec-RL may be applied to graph
nodes with varied degrees. Both of our models use the attention mechanism
during path-level aggregation to combine the representations of entities
learnt while following various meta-paths and produce the final attentional
joint representation. Experimental results show that, in the representation
learning stage, both ACKRec and HinCRec-RL can both fully utilize the
structure of HINs to learn excellent node embeddings. This is useful for the
further downstream recommendation task.

##### Extended Matrix Factorization versus REINFORCE

We feed both content and context information into the basic matrix
factorization model to form our previous, robust and explainable model,
ACKRec. It adopts gradient descent to obtain a scalar product of user latent
vectors ${p}_{k}\in{\mathbb{R}^{d}}$ and knowledge concept ones
${q}_{u}\in{\mathbb{R}^{d}}$ plus users representations ${e}_{u}$ and
knowledge concepts ${e}_{k}$ approximate original ratings ${r}_{uk}$. Although
ACKRec only slightly alleviates the classic data sparsity problem, it is no
longer useful in incremental recommendation settings. The fundamental problem
with such a system is that it continuously welcomes new users and concepts.
The recommender system for top-K concepts used at XuetangX is implemented in
practice by the HinCRec-RL model presented in this paper. When the system is
running, it may concurrently learn the interests of the users and utilize
those interests. This is distinct from the offline setting of ACKRec. In this
study, we apply REINFORCE to enable HinCRec-RL to adapt to the dynamic
interaction environment between users and concepts, where simultaneous
exploration and exploitation of the search space takes place. In the simplest
case, the recommendation agent may have to decide the concept lists to
recommend. The agent would either earn a reward for the success of the
recommendation or receive a penalty when a user clicks the particular notion
on the list. Thus, the online recommendation policy network changes can be
performed in real time. Experimental results show that HinCRec-RL can perform
well in dynamic environment. We will run HinCRec-RL stably in a live
production system in the future.

###  7.2. Why does HinCRec-RL Work?

Here we summarize three main features of HinCRec-RL that may explain its
effectiveness in concept recommendation in MOOCs.

##### Mining Latent Information in Heterogeneous Graphs.

Traditional approaches aimed at solving the data sparsity issue by
supplementing information about the context and its own attributes are not
work effective in our scenario owing to sparse interactions between users and
knowledge concepts coupled with a lack of attributes. To enhance the
relatedness between users and concepts, we introduce course and video entities
and build a heterogeneous graph as a bridge for message passing from concepts
to users. In addition to direct interactions between users and concepts, the
heterogeneous graph contains rich latent semantic information which is not
easy to mine but is crucial.

##### Attentional Multiple Meta-path-based Node Aggregation Representations.

HinCRec-RL learns user embedding representation by integrating the
representations of other adjacent nodes, which are those nodes on a path
selected based on a specific meta-path schema in the heterogeneous graph. In
the model, node weights and meta-paths are included in addition to node
properties. Therefore, it can capture the fine-grained connection between
users and concepts better and interpret users’ preferences by tracking and
tracing the users’ historical behaviors with the node relationships in these
meta-paths.

##### Dynamic Interaction through Reinforcement Learning.

In several real-world scenarios, a user’s interest always changes rapidly with
time, which requires methods that can react quickly to such changes.
Traditional deep-learning-based methods, owing to their training and learning
in a known data context and static environment, have intuitive limitations in
capturing dynamic changes. We formulate the concept recommendation task as a
sequential recommendation problem and model it as an MDP. Every action taken
by a reinforcement learning agent relies on a constantly changing user
embedding vector. Thus, HinCRec-RL can better adapt to the dynamic environment
in MOOCs.

###  7.3. Threats to Validity

The validity of our assessment is mostly at risk from two factors.

  * •

Generalizability of our framework. The main objective of our study is to
improve the performance of personalized recommendations in MOOCs through a
fine-grained concept recommendation approach. One threat is the differences
between education scenes and other scenes (e.g. e-commerce) in the real world.
In our experiments, we used a large dataset, containing 3,111,637 users and
2,527 concepts, to account for as many user behaviors as possible. Moreover,
we made the training and testing datasets as significantly different as
possible. In our future work, we will utilize even larger-scale datasets and
datasets of other scenarios to evaluate the effectiveness of HinCRec-RL.

  * •

The returned recommendations may have low relevance scores. We consider only
the top 20 returned results based on the ranking score. Increasing the scale
and diversity of datasets can avoid such low relevance threat. Moreover, we
believe that it is reasonable to consider the top 20 results. Users typically
pay attention to the top 20 or even fewer items in the recommended list in
real-world recommender systems and disregard the other items. This indicates
that items starting from $k+1$ ($k$=20) are not significant from the
perspective of users.

##  8\. Conclusion

This paper presented HinCRec-RL, a novel reinforcement learning architecture
that incorporates HINs, for the task of concept recommendation in MOOCs.
HinCRec-RL has a better ability to learn user representations by meta-path
sampling over an HIN, and can also accommodate the interactions between users
and knowledge concepts in the dynamic context of MOOCs. From experimental
results, we can have a conclusion that the proposed HinCRec-RL model can
considerably improve both the precision and diversity of the recommended list.
Our model can be applied in not only MOOC platforms for knowledge concept
recommendation, but also other practical scenarios, such as user
recommendation and commodity recommendation. In the future, we will deploy the
concept recommendation algorithm on the real-world MOOC platform XuetangX and
provide a commercial concepts recommendation service with personalized and
dynamic characteristics. In addition, we will explore how to apply inverse
reinforcement learning into the recommender system and further unleash the
power of HinCRec-RL.

###### Acknowledgements.

The work is supported by National Key R&D Program of China through grant
2022YFB3104700 and Hebei Natural Science Foundation of China through grant
F2022203072.

## References

  * (1)
  * Aggarwal et al. (2016) Charu C Aggarwal et al. 2016\.  _Recommender systems_. Vol. 1.  Springer.
  * Bosri et al. (2021) Rabeya Bosri, Mohammad Shahriar Rahman, Md Zakirul Alam Bhuiyan, and Abdullah Al Omar. 2021\.  Integrating blockchain with artificial intelligence for privacy-preserving recommender systems.  _IEEE Transactions on Network Science and Engineering_ 8, 2 (2021), 1009–1018.
  * Cai et al. (2018) Xiaoyan Cai, Junwei Han, and Libin Yang. 2018.  Generative adversarial network based heterogeneous bibliographic network representation for personalized citation recommendation. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Cao et al. (2021) Yuwei Cao, Hao Peng, Jia Wu, Yingtong Dou, Jianxin Li, and Philip S. Yu. 2021\.  Knowledge-Preserving Incremental Social Event Detection via Heterogeneous GNNs. In _Proceedings of the Web Conference 2021_ (Ljubljana, Slovenia) _(WWW ’21)_. Association for Computing Machinery, New York, NY, USA, 3383–3395.
  * Cen et al. (2019) Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang. 2019\.  Representation Learning for Attributed Multiplex Heterogeneous Network. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019_, Ankur Teredesai, Vipin Kumar, Ying Li, Rómer Rosales, Evimaria Terzi, and George Karypis (Eds.). ACM, 1358–1368.
  * Chen et al. (2021) Hongxu Chen, Yicong Li, Xiangguo Sun, Guandong Xu, and Hongzhi Yin. 2021.  Temporal meta-path guided explainable recommendation. In _Proceedings of the 14th ACM international conference on web search and data mining_. 1056–1064.
  * Chen et al. (2018b) Hongxu Chen, Hongzhi Yin, Weiqing Wang, Hao Wang, Quoc Viet Hung Nguyen, and Xue Li. 2018b.  PME: projected metric embedding on heterogeneous networks for link prediction. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1177–1186.
  * Chen et al. (2018a) Penghe Chen, Yu Lu, Vincent W Zheng, and Yang Pian. 2018a.  Prerequisite-driven deep knowledge tracing. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 39–48.
  * Chen et al. (2014) Wei Chen, Zhendong Niu, Xiangyu Zhao, and Yi Li. 2014\.  A hybrid recommendation algorithm adapted in e-learning environments.  _World Wide Web_ 17, 2 (2014), 271–284.
  * Chen et al. (2020) Xiaocong Chen, Chaoran Huang, Lina Yao, Xianzhi Wang, Wenjie Zhang, et al. 2020\.  Knowledge-guided deep reinforcement learning for interactive recommendation. In _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Chen et al. (2019) Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, and Le Song. 2019.  Neural Model-Based Reinforcement Learning for Recommendation. In _International Conference on Learning Representations_.
  * Deng et al. (2020) Zhongfen Deng, Hao Peng, Congying Xia, Jianxin Li, Lifang He, and Philip S. Yu. 2020\.  Hierarchical Bi-Directional Self-Attention Networks for Paper Review Rating Recommendation. In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , Donia Scott, Núria Bel, and Chengqing Zong (Eds.). International Committee on Computational Linguistics, 6302–6314.
  * Dong et al. (2017) Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017\.  metapath2vec: Scalable representation learning for heterogeneous networks. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 135–144.
  * Fan et al. (2019) Shaohua Fan, Junxiong Zhu, Xiaotian Han, Chuan Shi, Linmei Hu, Biyu Ma, and Yongliang Li. 2019.  Metapath-guided heterogeneous graph neural network for intent recommendation. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 2478–2486.
  * Feng et al. (2018) Jun Feng, Heng Li, Minlie Huang, Shichen Liu, Wenwu Ou, Zhirong Wang, and Xiaoyan Zhu. 2018.  Learning to collaborate: Multi-scenario ranking via multi-agent reinforcement learning. In _Proceedings of the 2018 World Wide Web Conference_. 1939–1948.
  * Feng et al. (2019) Wenzheng Feng, Jie Tang, and Tracy Xiao Liu. 2019.  Understanding dropouts in MOOCs. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 517–524.
  * Gai et al. (2017) Kun Gai, Xiaoqiang Zhu, Han Li, Kai Liu, and Zhe Wang. 2017.  Learning piece-wise linear models from large scale data for ad click prediction.  _arXiv preprint arXiv:1704.05194_ (2017).
  * Ghazarian and Nematbakhsh (2015) Sarik Ghazarian and Mohammad Ali Nematbakhsh. 2015.  Enhancing memory-based collaborative filtering for group recommender systems.  _Expert systems with applications_ 42, 7 (2015), 3801–3812.
  * Gong et al. (2020) Jibing Gong, Shen Wang, Jinlong Wang, Wenzheng Feng, Hao Peng, Jie Tang, and Philip S Yu. 2020.  Attentional graph convolutional networks for knowledge concept recommendation in moocs in a heterogeneous view. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 79–88.
  * Hamilton et al. (2017) William L Hamilton, Rex Ying, and Jure Leskovec. 2017\.  Inductive representation learning on large graphs. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_. 1025–1035.
  * He et al. (2018) Xiangnan He, Zhankui He, Jingkuan Song, Zhenguang Liu, Yu-Gang Jiang, and Tat-Seng Chua. 2018\.  NAIS: Neural Attentive Item Similarity Model for Recommendation.  _IEEE Trans. Knowl. Data Eng._ 30, 12 (2018), 2354–2366.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proceedings of the 26th International Conference on World Wide Web_. International World Wide Web Conferences Steering Committee, 173–182.
  * He et al. (2014) Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al. 2014\.  Practical lessons from predicting clicks on ads at facebook. In _Proceedings of the Eighth International Workshop on Data Mining for Online Advertising_. 1–9.
  * Hu et al. (2018a) Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Tianchi Yang. 2018a.  Local and Global Information Fusion for Top-N Recommendation in Heterogeneous Information Network. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_. ACM, 1683–1686.
  * Hu et al. (2018b) Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S. Yu. 2018b.  Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, 1531–1540.
  * Huang et al. (2021b) Liwei Huang, Mingsheng Fu, Fan Li, Hong Qu, Yangjun Liu, and Wenyu Chen. 2021b.  A deep reinforcement learning based long-term recommender system.  _Knowledge-Based Systems_ 213 (2021), 106706.
  * Huang and Lu (2018) Ran Huang and Ran Lu. 2018\.  Research on Content-based MOOC Recommender Model. In _2018 5th International Conference on Systems and Informatics (ICSAI)_. IEEE, 676–681.
  * Huang et al. (2021a) Tinglin Huang, Yuxiao Dong, Ming Ding, Zhen Yang, Wenzheng Feng, Xinyu Wang, and Jie Tang. 2021a.  Mixgcf: An improved training method for graph neural network-based recommender systems. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 665–674.
  * Intayoad et al. (2020) Wacharawan Intayoad, Chayapol Kamyod, and Punnarumol Temdee. 2020.  Reinforcement Learning Based on Contextual Bandits for Personalized Online Learning Recommendation Systems.  _Wireless Personal Communications_ (2020), 1–16.
  * Jin et al. (2020) Jiarui Jin, Jiarui Qin, Yuchen Fang, Kounianhua Du, Weinan Zhang, Yong Yu, Zheng Zhang, and Alexander J Smola. 2020\.  An efficient neighborhood-based interaction model for recommendation on heterogeneous graph. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 75–84.
  * Jing and Tang (2017) Xia Jing and Jie Tang. 2017\.  Guess you like: course recommendation in MOOCs. In _Proceedings of the International Conference on Web Intelligence_. ACM, 783–789.
  * Kabbur et al. (2013) Santosh Kabbur, Xia Ning, and George Karypis. 2013.  FISM: factored item similarity models for top-N recommender systems. In _The 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013_. 659–667.
  * Karimi et al. (2020) Hamid Karimi, Tyler Derr, Jiangtao Huang, and Jiliang Tang. 2020\.  Online Academic Course Performance Prediction Using Relational Graph Convolutional Neural Network.  _International Educational Data Mining Society_ (2020).
  * Ke et al. (2021) Gang Ke, Hong-Le Du, and Yeh-Cheng Chen. 2021.  Cross-platform dynamic goods recommendation system based on reinforcement learning and social networks.  _Applied Soft Computing_ 104 (2021), 107213.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ 30 (2017), 3146–3154.
  * Khalid et al. (2022) Asra Khalid, Karsten Lundqvist, and Anne Yates. 2022\.  A literature review of implemented recommendation techniques used in Massive Open online Courses.  _Expert Systems with Applications_ 187 (2022), 115926.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A Method for Stochastic Optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , Yoshua Bengio and Yann LeCun (Eds.).
  * Li et al. (2021) Chen Li, Linmei Hu, Chuan Shi, Guojie Song, and Yuanfu Lu. 2021.  Sequence-aware Heterogeneous Graph Neural Collaborative Filtering. In _Proceedings of the 2021 SIAM International Conference on Data Mining (SDM)_. SIAM, 64–72.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural Attentive Session-based Recommendation. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 - 10, 2017_. 1419–1428.
  * Liao et al. (2019) Jinzhi Liao, Jiuyang Tang, and Xiang Zhao. 2019.  Course drop-out prediction on MOOC platform via clustering and tensor completion.  _Tsinghua Science and Technology_ 24, 4 (2019), 412–422.
  * Liu et al. (2018) Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, and Yuzhou Zhang. 2018\.  Deep reinforcement learning based recommendation with explicit user-item interactions modeling.  _arXiv preprint arXiv:1810.12027_ (2018).
  * Liu et al. (2022a) Xiangyong Liu, Guojun Wang, and Md Zakirul Alam Bhuiyan. 2022a.  Personalised context-aware re-ranking in recommender system.  _Connection Science_ 34, 1 (2022), 319–338.
  * Liu et al. (2022b) Xiangyong Liu, Guojun Wang, and Md Zakirul Alam Bhuiyan. 2022b.  Re-ranking with multiple objective optimization in recommender system.  _Transactions on Emerging Telecommunications Technologies_ 33, 1 (2022), e4398.
  * Loni et al. (2019) Babak Loni, Roberto Pagano, Martha Larson, and Alan Hanjalic. 2019.  Top-N recommendation with multi-channel positive feedback using factorization machines.  _ACM Transactions on Information Systems (TOIS)_ 37, 2 (2019), 1–23.
  * Lu et al. (2019) Weiming Lu, Yangfan Zhou, Jiale Yu, and Chenhao Jia. 2019\.  Concept Extraction and Prerequisite Relation Learning from Educational Data. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 9678–9685.
  * Meng et al. (2022) Shunmei Meng, Shaoyu Fan, Qianmu Li, Xinna Wang, Jing Zhang, Xiaolong Xu, Lianyong Qi, and Md Zakirul Alam Bhuiyan. 2022.  Privacy-Aware Factorization-Based Hybrid Recommendation Method for Healthcare Services.  _IEEE Transactions on Industrial Informatics_ 18, 8 (2022), 5637–5647.
  * Omar et al. (2019) Abdullah Al Omar, Rabeya Bosri, Mohammad Shahriar Rahman, Nasima Begum, and Md Zakirul Alam Bhuiyan. 2019\.  Towards privacy-preserving recommender system with blockchains. In _International Conference on Dependability in Sensor, Cloud, and Big Data Systems and Applications_. Springer, 106–118.
  * Pan et al. (2019) Feiyang Pan, Qingpeng Cai, Pingzhong Tang, Fuzhen Zhuang, and Qing He. 2019.  Policy gradients for contextual recommendations. In _The World Wide Web Conference_. 1421–1431.
  * Pan et al. (2017a) Liangming Pan, Chengjiang Li, Juanzi Li, and Jie Tang. 2017a.  Prerequisite relation learning for concepts in moocs. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 1447–1456.
  * Pan et al. (2017b) Liangming Pan, Xiaochen Wang, Chengjiang Li, Juanzi Li, and Jie Tang. 2017b.  Course concept extraction in MOOCs via embedding-based graph propagation. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 875–884.
  * Pang et al. (2017) Yanxia Pang, Yuanyuan Jin, Ying Zhang, and Tao Zhu. 2017\.  Collaborative filtering recommendation for MOOC application.  _Computer Applications in Engineering Education_ 25, 1 (2017), 120–128.
  * Park et al. (2012) Deuk Hee Park, Hyea Kyeong Kim, Il Young Choi, and Jae Kyeong Kim. 2012.  A literature review and classification of recommender systems research.  _Expert systems with applications_ 39, 11 (2012), 10059–10072.
  * Peng et al. (2021a) Hao Peng, Jianxin Li, Yangqiu Song, Renyu Yang, Rajiv Ranjan, Philip S. Yu, and Lifang He. 2021a.  Streaming Social Event Detection and Evolution Discovery in Heterogeneous Information Networks.  _ACM Trans. Knowl. Discov. Data_ 15, 5, Article 89 (may 2021), 33 pages.
  * Peng et al. (2020) Hao Peng, Hongfei Wang, Bowen Du, Md Zakirul Alam Bhuiyan, Hongyuan Ma, Jianwei Liu, Lihong Wang, Zeyu Yang, Linfeng Du, Senzhang Wang, and Philip S. Yu. 2020.  Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting.  _Information Sciences_ 521 (2020), 277–290.
  * Peng et al. (2021b) Hao Peng, Ruitong Zhang, Yingtong Dou, Renyu Yang, Jingyi Zhang, and Philip S Yu. 2021b.  Reinforced neighborhood selection guided multi-relational graph neural networks.  _ACM Transactions on Information Systems (TOIS)_ 40, 4 (2021), 1–46.
  * Peng et al. (2022) Hao Peng, Ruitong Zhang, Shaoning Li, Yuwei Cao, Shirui Pan, and Philip Yu. 2022\.  Reinforced, incremental and cross-lingual event detection from social messages.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2022).
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  Deepwalk: Online learning of social representations. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 701–710.
  * Qiu et al. (2016) Jiezhong Qiu, Jie Tang, Tracy Xiao Liu, Jie Gong, Chenhui Zhang, Qian Zhang, and Yufei Xue. 2016.  Modeling and predicting learning behavior in MOOCs. In _Proceedings of the ninth ACM international conference on web search and data mining_. ACM, 93–102.
  * Rabahallah et al. (2018) Kahina Rabahallah, Latifa Mahdaoui, and Faiçal Azouaou. 2018\.  MOOCs Recommender System using Ontology and Memory-based Collaborative Filtering. In _ICEIS (1)_. 635–641.
  * Rendle (2012) Steffen Rendle. 2012\.  Factorization Machines with libFM.  _ACM TIST_ 3, 3 (2012), 57:1–57:22.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback _(UAI ’09)_. AUAI Press, Arlington, Virginia, USA, 452–461.
  * Shen et al. (2018) Yilin Shen, Yue Deng, Avik Ray, and Hongxia Jin. 2018\.  Interactive recommendation via deep neural memory augmented contextual bandits. In _Proceedings of the 12th ACM Conference on Recommender Systems_. 122–130.
  * Shi et al. (2019) Chuan Shi, Binbin Hu, Wayne Xin Zhao, and Philip S. Yu. 2019\.  Heterogeneous information network embedding for recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ 31, 2 (2019), 357–370.
  * Silver et al. (2013) David Silver, Leonard Newnham, David Barker, Suzanne Weller, and Jason McFall. 2013.  Concurrent reinforcement learning from customer interactions. In _International conference on machine learning_. PMLR, 924–932.
  * Sun et al. (2020) Changfeng Sun, Han Liu, Meng Liu, Zhaochun Ren, Tian Gan, and Liqiang Nie. 2020\.  LARA: Attribute-to-feature Adversarial Learning for New-item Recommendation. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 582–590.
  * Sun et al. (2021) Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Jiuxin Cao, Yingxia Shao, and Nguyen Quoc Viet Hung. 2021.  Heterogeneous hypergraph embedding for graph classification. In _Proceedings of the 14th acm international conference on web search and data mining_. 725–733.
  * Sun and Han (2012) Yizhou Sun and Jiawei Han. 2012.  Mining heterogeneous information networks: principles and methodologies.  _Synthesis Lectures on Data Mining and Knowledge Discovery_ 3, 2 (2012), 1–159.
  * Sun et al. (2011) Yizhou Sun, Jiawei Han, Xifeng Yan, Philip S. Yu, and Tianyi Wu. 2011.  Pathsim: Meta path-based top-k similarity search in heterogeneous information networks.  _Proceedings of the VLDB Endowment_ 4, 11 (2011), 992–1003.
  * Sutton et al. (2000) Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000.  Policy gradient methods for reinforcement learning with function approximation. In _Advances in neural information processing systems_. 1057–1063.
  * Van Hasselt et al. (2016) Hado Van Hasselt, Arthur Guez, and David Silver. 2016\.  Deep reinforcement learning with double q-learning. In _Thirtieth AAAI Conference on Artificial Intelligence_.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph Attention Networks. In _International Conference on Learning Representations_.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019\.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575, 7782 (2019), 350–354.
  * Wan and Niu (2019) Shanshan Wan and Zhendong Niu. 2019.  A hybrid e-learning recommendation approach based on learners’ influence propagation.  _IEEE Transactions on Knowledge and Data Engineering_ 32, 5 (2019), 827–840.
  * Wang et al. (2019) Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu. 2019.  Heterogeneous Graph Attention Network. In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_ , Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza-Yates, and Leila Zia (Eds.). ACM, 2022–2032.
  * Wang et al. (2021) Xiao Wang, Nian Liu, Hui Han, and Chuan Shi. 2021\.  Self-supervised heterogeneous graph neural network with co-contrastive learning. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 1726–1736.
  * Welling and Kipf (2016) Max Welling and Thomas N Kipf. 2016.  Semi-supervised classification with graph convolutional networks. In _J. International Conference on Learning Representations (ICLR 2017)_.
  * Williams (1992) Ronald J Williams. 1992\.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine learning_ 8, 3-4 (1992), 229–256.
  * Wu et al. (2019) Le Wu, Peijie Sun, Yanjie Fu, Richang Hong, Xiting Wang, and Meng Wang. 2019\.  A neural influence diffusion model for social recommendation. In _Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval_. 235–244.
  * Wu et al. (2020) Le Wu, Yonghui Yang, Kun Zhang, Richang Hong, Yanjie Fu, and Meng Wang. 2020\.  Joint item recommendation and attribute inference: An adaptive graph convolutional network approach. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 679–688.
  * Xie et al. (2021) Ruobing Xie, Shaoliang Zhang, Rui Wang, Feng Xia, and Leyu Lin. 2021.  Hierarchical Reinforcement Learning for Integrated Recommendation. In _Proceedings of AAAI_.
  * Xu et al. (2019) Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. 2019\.  How Powerful are Graph Neural Networks?. In _International Conference on Learning Representations_.
  * Yin et al. (2020) Hongzhi Yin, Qinyong Wang, Kai Zheng, Zhixu Li, and Xiaofang Zhou. 2020.  Overcoming data sparsity in group recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).
  * Yu et al. (2020b) Jifan Yu, Gan Luo, Tong Xiao, Qingyang Zhong, Yuquan Wang, Wenzheng Feng, Junyi Luo, Chenyu Wang, Lei Hou, Juanzi Li, et al. 2020b.  MOOCCube: a large-scale data repository for NLP applications in MOOCs. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. 3135–3142.
  * Yu et al. (2019b) Jifan Yu, Chenyu Wang, Gan Luo, Lei Hou, Juanzi Li, Jie Tang, and Zhiyuan Liu. 2019b.  Course Concept Expansion in MOOCs with External Knowledge and Interactive Game. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. ACL, 4292–4302.
  * Yu et al. (2021) Jifan Yu, Yuquan Wang, Qingyang Zhong, Gan Luo, Yiming Mao, Kai Sun, Wenzheng Feng, Wei Xu, Shulin Cao, Kaisheng Zeng, et al. 2021\.  MOOCCubeX: A Large Knowledge-centered Repository for Adaptive Learning in MOOCs. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 4643–4652.
  * Yu et al. (2020a) Runlong Yu, Qi Liu, Yuyang Ye, Mingyue Cheng, Enhong Chen, and Jianhui Ma. 2020a.  Collaborative list-and-pairwise filtering from implicit feedback.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).
  * Yu et al. (2019a) Tong Yu, Yilin Shen, Ruiyi Zhang, Xiangyu Zeng, and Hongxia Jin. 2019a.  Vision-language recommendation via attribute augmented multimodal reinforcement learning. In _Proceedings of the 27th ACM International Conference on Multimedia_. 39–47.
  * Zhang et al. (2019) Jing Zhang, Bowen Hao, Bo Chen, Cuiping Li, Hong Chen, and Jimeng Sund. 2019\.  Hierarchical Reinforcement Learning for Course Recommendation in MOOCs.  _Psychology_ 5, 4.64 (2019), 5–65.
  * Zhang et al. (2021) Sixiao Zhang, Hongxu Chen, Xiao Ming, Lizhen Cui, Hongzhi Yin, and Guandong Xu. 2021\.  Where Are We in Embedding Spaces? _(KDD ’21)_. Association for Computing Machinery, New York, NY, USA, 2223–2231.
  * Zhang et al. (2017) Yutao Zhang, Robert Chen, Jie Tang, Walter F Stewart, and Jimeng Sun. 2017.  LEAP: learning to prescribe effective and safe treatment combinations for multimorbidity. In _proceedings of the 23rd ACM SIGKDD international conference on knowledge Discovery and data Mining_. 1315–1324.
  * Zhao et al. (2020b) Dongyang Zhao, Liang Zhang, Bo Zhang, Lizhou Zheng, Yongjun Bao, and Weipeng Yan. 2020b.  MaHRL: Multi-goals Abstraction Based Deep Hierarchical Reinforcement Learning for Recommendations. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 871–880.
  * Zhao et al. (2019) Wei Zhao, Benyou Wang, Min Yang, Jianbo Ye, Zhou Zhao, Xiaojun Chen, and Ying Shen. 2019.  Leveraging long and short-term information in content-aware movie recommendation via adversarial training.  _IEEE transactions on cybernetics_ 50, 11 (2019), 4680–4693.
  * Zhao et al. (2018) Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, and Jiliang Tang. 2018\.  Deep reinforcement learning for page-wise recommendations. In _Proceedings of the 12th ACM Conference on Recommender Systems_. ACM, 95–103.
  * Zhao et al. (2020a) Xiangyu Zhao, Long Xia, Lixin Zou, Hui Liu, Dawei Yin, and Jiliang Tang. 2020a.  Whole-Chain Recommendations. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 1883–1891.
  * Zheng et al. (2018) Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. 2018.  DRN: A deep reinforcement learning framework for news recommendation. In _Proceedings of the 2018 World Wide Web Conference on World Wide Web_. International World Wide Web Conferences Steering Committee, 167–176.
  * Zhou et al. (2020) Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang, Xiuqiang He, and Yong Yu. 2020\.  Interactive recommender system via knowledge graph-enhanced reinforcement learning. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 179–188.
  * Zhu et al. (2019) Yu Zhu, Jinghao Lin, Shibi He, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2019.  Addressing the item cold-start problem by attribute-driven active learning.  _IEEE Transactions on Knowledge and Data Engineering_ 32, 4 (2019), 631–644.
  * Zou et al. (2019) Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, and Dawei Yin. 2019\.  Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2810–2818.
  * Zou et al. (2020) Lixin Zou, Long Xia, Pan Du, Zhuo Zhang, Ting Bai, Weidong Liu, Jian-Yun Nie, and Dawei Yin. 2020.  Pseudo Dyna-Q: A reinforcement learning framework for interactive recommendation. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 816–824.