# Phase reconstruction from amplitude spectrograms
based on von-Mises-distribution deep neural network

###### Abstract

This paper presents a deep neural network (DNN)-based phase reconstruction
from amplitude spectrograms. In audio signal and speech processing, the
amplitude spectrogram is often used for processing, and the corresponding
phase spectrogram is reconstructed from the amplitude spectrogram on the basis
of the Griffin-Lim method. However, the Griffin-Lim method causes unnatural
artifacts in synthetic speech. Addressing this problem, we introduce the von-
Mises-distribution DNN for phase reconstruction. The DNN is a generative model
having the von Mises distribution that can model distributions of a periodic
variable such as a phase, and the model parameters of the DNN are estimated on
the basis of the maximum likelihood criterion. Furthermore, we propose a
group-delay loss for DNN training to make the predicted group delay close to a
natural group delay. The experimental results demonstrate that 1) the trained
DNN can predict group delay accurately more than phases themselves, and 2) our
phase reconstruction methods achieve better speech quality than the
conventional Griffin-Lim method.

Index Terms—  speech analysis, phase reconstruction, deep neural network, von
Mises distribution, group delay

##  1 Introduction

A variety of audio signal processing and machine learning-based methods, such
as audio source separation and speech enhancement, involve processing the
amplitude spectrogram obtained via short-term Fourier transform (STFT). Also,
statistical parametric text-to-speech synthesis [1] is shifting from vocoder-
based (source filter model-based) to STFT-based frameworks [2, 3, 4]. To
produce synthetic speech, we require the corresponding phase spectrogram, but
it is often unavailable. The Griffin-Lim method [5] is a well-known example
that iteratively estimates of the phase spectrogram through STFT and inverse
STFT while fixing the amplitude spectrogram. This signal-processing-based
method has high portability without a priori training but causes unnatural
artifacts in synthetic speech. Therefore, this paper proposes building a
trainable phase reconstruction method using generative models.

A deep neural network (DNN) is a powerful generative model. There are two
types of distributions: non-parametric [6, 7, 8, 9] and parametric ones. This
paper addresses the latter. There are many types of the parametric approaches
related to the Gaussian distribution, e.g., isotropic multivariate Gaussian
[10] and temporal-delta-constrained Gaussian [11, 12] (a.k.a., trajectory DNN
in statistical parametric speech synthesis). The loss function (e.g., mean
squared error) for training DNNs is often derived to minimize the negative
log-likelihood of the distribution. The straightforward way to predict a phase
spectrogram from an amplitude spectrogram is to use these models. However, the
Gaussian distribution is inappropriate for modeling distributions of a phase
that is a periodic variable with a period of $2\pi$.

This paper proposes phase reconstruction from amplitude spectrograms based on
the von-Mises-distribution DNN. The von Mises distribution is a probability
distribution on the circle [13], which is suitable for modeling periodic
variables. The von-Mises-distribution DNN is a generative model that has the
von Mises distribution as a conditional probability distribution. The von-
Mises-distribution shallow neural network was originally proposed by Nabney et
al. [14], and this paper utilizes it for predicting the phase spectrogram from
the amplitude spectrogram. The loss function for DNN training, named the phase
loss (see Fig. 1), is defined by minimizing likelihoods of the von Mises
distribution. Also, we propose another loss function named group-delay loss,
which has a stronger connection to the amplitude spectrum [15]. The group-
delay loss is used to make group delay of the predicted phase close to that of
the target phase. Since the group delay and group-delay loss are
differentiable by the phase, the DNN can be trained by a standard
backpropagation algorithm. We conduct objective and subjective evaluations to
evaluate the effectiveness of the proposed methods. The results demonstrate
that 1) the trained DNN can predict group delay accurately more than phases
themselves, and 2) our phase reconstruction method outperforms the
conventional Griffin-Lim method in terms of quality of synthesized speech.

Fig. 1: Overview of proposed phase reconstruction method. This figure shows
frame-by-frame phase prediction rather than multi-frame or sequence-wise
prediction for clear illustration.

##  2 Griffin-Lim phase reconstruction

This section briefly describes the conventional Griffin-Lim method [5]. The
Griffin-Lim method is a signal-processing-based iterative algorithm to
reconstruct a phase spectrogram from the amplitude spectrogram. Let
$\textrm{\boldmath$x$}=\left[\textrm{\boldmath$x$}_{1},\cdots,\textrm{\boldmath$x$}_{t},\cdots,\textrm{\boldmath$x$}_{T}\right]$
and
$\textrm{\boldmath$y$}=\left[\textrm{\boldmath$y$}_{1},\cdots,\textrm{\boldmath$y$}_{t},\cdots,\textrm{\boldmath$y$}_{T}\right]$
be amplitude and phase spectrograms, respectively.
$\textrm{\boldmath$x$}_{t}=\left[x_{t,0},\cdots,x_{t,f},\cdots,x_{t,F}\right]$
and
$\textrm{\boldmath$y$}_{t}=\left[y_{t,0},\cdots,y_{t,f},\cdots,y_{t,F}\right]$
are the amplitude and phase at frame $t$, respectively. $f$ is the frequency
index, and $F$ corresponds to the Nyquist frequency. Both $x_{t,f}$ and
$y_{t,f}$ are real-valued variables, but only $y_{t,f}$ is a variable with a
period of $2\pi$. The Griffin-Lim method randomly initializes $y$ first. 1)
Then it takes inverse STFT to obtain a waveform from $x$ and $y$. 2) The
method takes STFT to re-obtain $y$ from the waveform. 3) We substitute the
original $x$ for the re-obtained $x$ and then go back to step 1). These
inverse STFT and STFT are iteratively performed until they have converged. The
method can reconstruct the phase spectrogram consistent with the given
amplitude spectrogram but makes some artifacts in the synthesized speech,
e.g., extra reverberation and phasiness owing to inappropriate initialization
of $y$.

##  3 Phase reconstruction based on von-Mises-distribution DNNs

This section introduces the von-Mises-distribution DNN and proposes loss
functions for the DNN training.

###  3.1 von Mises distribution

The von Mises distribution $P^{(\rm vM)}\left(\cdot\right)$ [13] is a
probability distribution for a periodic variable $y_{t,f}$, given as

$\displaystyle P^{(\rm
vM)}\left(y_{t,f};\mu,\kappa\right)=\frac{\exp\left(\kappa\cos\left(y_{t,f}-\mu\right)\right)}{2\pi
I_{0}\left(\kappa\right)},$ (1)

where $\mu$ is the mean direction of the distribution (analogous to the mean
of the Gaussian distribution), $\kappa$ is the shape parameter (analogous to
the precision of the Gaussian distribution), and $I_{0}\left(\cdot\right)$ is
the modified Bessel function of the first kind of order 0. The negative log
likelihood given $\textrm{\boldmath$y$}_{t}$ is:

$\displaystyle-\log{P^{(\rm
vM)}\left(\textrm{\boldmath$y$}_{t};\mu,\kappa\right)}\propto-\sum\limits_{f=0}^{F}\cos\left(y_{t,f}-\mu\right)+\textrm{Const.},$
(2)

where Const. is a value constant to $\mu$. Not only $y_{t,f}$ but also a
maximum likelihood estimate of $\mu$ has a period of $2\pi$.

###  3.2 DNN training

We train a DNN that has the von Mises distribution as a conditional
probability distribution. The mean direction is predicted from $x$ at each
frame and frequency. Here, let $\textrm{\boldmath$G$}\left(\cdot\right)$ be
the DNN. The predicted phase (mean direction)
$\textrm{\boldmath$\hat{y}$}=\left[\textrm{\boldmath$\hat{y}$}_{1},\cdots,\textrm{\boldmath$\hat{y}$}_{t},\cdots,\textrm{\boldmath$\hat{y}$}_{T}\right]$
is given as
$\textrm{\boldmath$\hat{y}$}=\textrm{\boldmath$G$}\left(\textrm{\boldmath$x$}\right)$.
The following sections propose two loss functions for estimating model
parameters of $\textrm{\boldmath$G$}\left(\cdot\right)$: phase loss $L_{\rm
ph}\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)$ and
group-delay loss $L_{\rm
gd}\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)$.

####  3.2.1 Phase loss

The phase loss function is derived from Eq. (2) as follows:

$\displaystyle L_{\rm
ph}\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)=\sum\limits_{f=0}^{F}-\cos\left(y_{t,f}-{\hat{y}}_{t,f}\right).$
(3)

Model parameters of $\textrm{\boldmath$G$}\left(\cdot\right)$ are iteratively
updated by backpropagation algorithm to minimize this loss function. The
minimum point of ${\hat{y}}_{t,f}$ is periodic, i.e.,
${\hat{y}}_{t,f}=y_{t,f}\pm 2\pi N$, where $N$ is an arbitrary integer value.

####  3.2.2 Group-delay loss

Group delay of speech has a high potential in speech processing, such as
speech and speaker recognition [15, 16]. The group delay is defined as the
negative derivative of phase by frequency. In general, speech production via a
human vocal tract is well modeled as an all-pole filter defined by
$A(\omega)\exp(j\phi(\omega))$, where $A(\omega)$ and $\phi(\omega)$ are the
amplitude and phase functions, respectively, and $\omega=\pi f/F$ is the
angular frequency. If the filter has $P$ complex-valued poles
$z_{p}=r_{p}\exp(j\omega_{p})$ ($p=1,...,P$), we have $\log
A(\omega)=\log\prod_{p=1}^{P}A_{p}(\omega)=2\sum_{p=1}^{P}\sum_{n=1}^{\infty}r_{p}^{n}/n\cos
n(\omega-\omega_{p})$, where $A_{p}(\omega)$ is the amplitude of the single-
pole model for the specific $p$-th pole [15]. Then, the group delay can be
given by

$\displaystyle-\frac{d\phi(\omega)}{d\omega}=c\sum_{p=1}^{P}\sum_{n=1}^{\infty}n\cos
n(\omega-\omega_{p})\int_{-\pi}^{\pi}\log A_{p}(\omega)\cos(n\omega)d\omega,$
(4)

where $c$ is a constant value; this shows the strong correlation between the
group delay and the amplitude spectrum, which motivates us to utilize the
group delay as a regularization term.

In this paper, we approximate the group delay at frame $t$ and frequency bin
$f$ with the following equation:

$\displaystyle\Delta y_{t,f}=-\left(y_{t,f+1}-y_{t,f}\right).$ (5)

Since $\Delta y_{t,f}$ is also a periodic variable, the group-delay loss is
defined as similar to Eq. (3):

$\displaystyle L_{\rm
gd}\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)=\sum\limits_{f=0}^{F}-\cos\left(\Delta
y_{t,f}-\Delta{\hat{y}}_{t,f}\right).$ (6)

The group-delay loss makes $\Delta\hat{y}_{t,f}$ close to $\Delta y_{t,f}$.
Because Eq. (5) for $\textrm{\boldmath$\hat{y}$}_{t}$ is a linear
transformation of $\textrm{\boldmath$y$}_{t}$, the backpropagation algorithm
can be used as the same as in  Section 3.2.1.

####  3.2.3 Multi-task learning

On the basis of the multi-task learning formulation, the DNN can be trained
using both phase loss and group-delay loss. The loss function
$L\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)$ is:

$\displaystyle
L\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)=L_{\rm
ph}\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right)+\alpha
L_{\rm
gd}\left(\textrm{\boldmath$y$}_{t},\textrm{\boldmath$\hat{y}$}_{t}\right),$
(7)

where $\alpha$ is the weight of the secondary task (group-delay loss). Note
that, since ranges of Eq. (3) and Eq. (6) are the same, no scale normalization
factor is required.

###  3.3 Discussion

Fig. 2: Histogram of predicted phases. The target phases have a range of
$\left[0,2\pi\right]$, but the predicted phases have a range of
$\left[-4\pi,6\pi\right]$. All frames and frequency bins of the evaluation
data in  Section 4.1 are represented in this figure.

The generalized version of the von Mises distribution is the generalized
cardioid distribution [17] given as

$\displaystyle P^{(\rm GC)}\left(y_{t,f};\mu,\kappa,\psi\right)$
$\displaystyle=$
$\displaystyle\frac{\left(\cosh\left(\kappa\psi\right)\right)^{1/\psi}\left(1+\tanh\left(\kappa\psi\right)\cos\left(y_{t,f}-\mu\right)\right)^{1/\psi}}{2\pi
P_{1/\psi}\left(\cosh\left(\kappa\psi\right)\right)},$ (8)

where $P_{1/\psi}$ is the associated Legendre function of the first kind of
degree $1/\psi$ and order 0. The von Mises distribution is the special case
($\psi\rightarrow 0$) of this distribution. Also, this distribution is
equivalent to the cardioid distribution and the wrapped Cauchy distribution
for $\psi=1$ and $\psi=-1$, respectively. The negative log likelihood for
$\mu$ for the cardioid and wrapped Cauchy distributions are equal to Eq. (2).
Therefore, DNNs with these two distributions are trained in the same manner.
One possible way to extend our work is to model phases using this generalized
cardioid distribution. Other possible ways are to use an asymmetric
distribution [18] and mixture model.

As described in  Section 3.2.1, the phase loss is minimized when
${\hat{y}}_{t,f}=y_{t,f}\pm 2\pi N$ for arbitrary $N$. Therefore, von-Mises-
distribution DNNs suffer from an exploding value of ${\hat{y}}_{t,f}$. To
investigate this, Fig. 2 plots a histogram of the predicted phase
spectrograms. We can see that the predicted phase spectrograms have a wider
range than the target phase ($\left[0,2\pi\right]$), but the value does not
explode.

##  4 Experimental evaluation

###  4.1 Experimental setup

Evaluations were performed using the JSUT corpus [19], a free Japanese speech
corpus uttered by a female speaker. We used 5,000 utterances (approx. 6 hours)
of the subset BASIC5000 for training and 300 utterances of the subset
ONOMATOPEE300 for evaluation. Speech signals were downsampled at a rate of 16
kHz. The window length, shift length, and FFT length were set to 400 samples
(25 ms), 80 samples (5 ms), and 512 samples, respectively. The Hamming window
was used. Features fed to a DNN were the joint vectors of the log amplitude
spectra at current and $\pm 2$ frames, and they were normalized to have zero-
mean unit-variance. The DNN architectures were Feed-Forward networks that
included $3\times 1024$-unit gated linear unit [20] hidden layers. The
activation of the output layer was a linear function. We empirically explored
DNN architectures within Feed-Forward networks and found that the gated linear
unit is significantly better than a rectified linear unit (ReLU) [21] or
LeakyReLU [22] hidden units. The DNN parameters were randomly initialized and
the AdaGrad algorithm [23] with its learning rate set to 0.001 was used for
the optimization algorithm.

We compared the conventional Griffin-Lim and three proposed phase
reconstruction methods.

  * PH

only phase loss (Eq. (3))

  * GD

only group-delay loss (Eq. (6))

  * PH+GD

multi-task learning (Eq. (7))

In the Griffin-Lim method, phase spectrograms were randomly initialized. The
number of iterations was set to 100. The weight $\alpha$ for multi-task
learning was set to $0.1$. In the proposed methods, phases in the low
frequency band are first estimated by DNNs, and those in the remaining
frequency bins are randomly generated. We used three settings of frequency
bands of the predicted phase spectrograms: 0-2 (96 dim.), 0-4 (128 dim.), and
0-8 kHz (257 dim.). After predicting the phase spectrogram by DNNs, we further
applied the Griffin-Lim method to refine the phase. The number of iterations
of the phase refinement was 100.

###  4.2 Prediction accuracy

Fig. 3: Box plots of cosine distances between target and predicted phases (upper) and group delays (lower). The box indicates the first and third quartiles. Table 1: Results of preference tests: conventional Griffin-Lim method vs. proposed methods. Bold indicates preferred method that has a $p$-value smaller than $0.05$ Method A | Scores |  $p$-value | Method B
---|---|---|---
Griffin-Lim | 0.497 vs. 0.503 | 0.871 | PH (2 kHz)
Griffin-Lim |  0.280 vs. 0.720 | $<10^{-9}$ | PH (4 kHz)
Griffin-Lim |  0.277 vs. 0.723 | $<10^{-9}$ | PH (8 kHz)
Griffin-Lim |  0.453 vs. 0.547 | 0.022 | PH+GD (2 kHz)
Griffin-Lim |  0.233 vs. 0.767 | $<10^{-9}$ | PH+GD (4 kHz)
Griffin-Lim |  0.247 vs. 0.753 | $<10^{-9}$ | PH+GD (8 kHz)
Griffin-Lim |  0.447 vs. 0.553 | 0.009 | GD (2 kHz)
Griffin-Lim | 0.463 vs. 0.537 | 0.073 | GD (4 kHz)
Griffin-Lim | 0.490 vs. 0.510 | 0.619 | GD (8 kHz)

We evaluated prediction accuracies of phases and group delay. Fig. 3 shows
cosine distances between target and predicted phases and group delays. The
distances were averaged over all frames and frequency bins of the predicted
phases (0-2, 0-4, or 0-8 kHz).

The prediction accuracy of “PH (2 kHz)” ranges from $0.15$ to $0.31$, and the
distribution seems symmetric. Also, the accuracy becomes smaller as the
frequency band widens (“PH (4, 8 kHz)”). This result is natural because phases
at higher frequency bins are easily changed by the temporal position of frame
analysis. On the other hand, when using only group-delay loss (“GD”), group
delay can be predicted more accurately than phase for all settings of
frequency bands. This result shows us that the Feed-Forward DNN can predict
group delay better than phases themselves. Finally, combined phase loss and
group-delay loss (“PH+GD”) predicts phase more accurately than “GD” and group
delay more accurately than “PH” for all settings of frequency bands.
Therefore, we can demonstrate the effectiveness of the proposed multi-task
training.

###  4.3 Comparison of Griffin-Lim and proposed methods

To evaluate the effectiveness of the proposed methods, we compared the quality
of synthetic speech of Griffin-Lim and proposed methods. Preference AB tests
(listening tests) on speech quality were performed in our crowdsourcing
evaluation system. 30 listeners participated in each test. Approximately
$\$0.46$ were paid to each listener. Each listener preferred better-quality
speech and answered for ten pairs of samples, i.e., 300 answers were obtained
in each evaluation. Speech samples of pairs of methods were randomly presented
to the listeners. These settings are also used not only here but also in the
following sections.

Table 1 lists the results. The proposed methods outperform the conventional
Griffin-Lim method in all settings of loss functions and frequency bands. In
particular, “PH+GD” always has significantly better scores than the
conventional Griffin-Lim method. These results demonstrate the effectiveness
of the proposed methods.

###  4.4 Effect of phase refinements

Fig. 4: Box plots of cosine distances between predicted and refined phases (upper) and group delays (lower). The box indicates the first and third quartiles. Table 2: Results of preference tests: proposed methods with different frequency bands. Bold indicates preferred method that has a $p$-value smaller than $0.05$ Method A | Scores |  $p$-value | Method B
---|---|---|---
PH (2 kHz) |  0.270 vs. 0.730 | $<10^{-9}$ | PH (4 kHz)
PH (4 kHz) | 0.507 vs. 0.493 | 0.744 | PH (8 kHz)
PH+GD (2 kHz) |  0.223 vs. 0.777 | $<10^{-9}$ | PH+GD (4 kHz)
PH+GD (4 kHz) | 0.493 vs. 0.507 | 0.744 | PH+GD (8 kHz)
GD (2 kHz) | 0.513 vs. 0.487 | 0.514 | GD (4 kHz)
GD (4 kHz) |  0.567 vs. 0.433 | 0.001 | GD (8 kHz)

As explained in  Section 4.1, after phases were predicted by DNNs, they were
refined by the Griffin-Lim method. Here, we evaluated effects of the phase
refinements. Fig. 4 shows cosine distances between predicted and refined
phases and group delays. Tendencies were almost the same as those in Fig. 3.
When phase loss is used in training (“PH” and “PH+GD”), phase information is
comparably preserved. Similarly, when group-delay loss is used in training
(“GD” and “PH+GD”), the group-delay information is comparably preserved. In
the preliminary evaluation, we compared speech quality of refined and
unrefined phases (i.e., phases predicted by DNNs were directly used for
finally synthesized speech). The results demonstrated that unrefined phases
had significantly worse speech quality than refined phases.

###  4.5 Effect of frequency bands

We compared effects of frequency bands of predicted phases within one loss
function (“PH”, “PH+GD”, or “GD”). Table 2 shows the results of preference AB
tests on speech quality. In “PH” and “PH+GD,” speech quality for 0-4 kHz
frequency bands is significantly better than that for 0-2 kHz, and is
comparable with that for 0-8 kHz. These results suggest that at least 0-4 kHz
frequency bands needs to be predicted but the spectrograms in the higher
frequency bands may be excited by random phases (this tendency is similar to
the harmonics plus noise model [24]). A curious point is that “GD” has
different tendencies: speech quality for 0-8 kHz frequency bands was
significantly worse than that for 0-4 kHz. We will investigate the reason for
this.

###  4.6 Effect of group-delay loss

We investigated the effectiveness of group-delay loss compared with phase
loss. First, we evaluated convergence of phase refinements. Fig. 5 shows the
log spectral convergence [25] of “PH (4 kHz)” and “PH+GD (4 kHz).” For
comparison, results of randomized initialization of phases are also shown as
“Random.” The proposed methods (“PH (4 kHz)” and “PH+GD (4 kHz)”) have a
smaller value for the spectral convergence than “Random.” Also, “PH+GD (4
kHz)” has a smaller value than “PH (4 kHz)”, i.e., “PH+GD (4 kHz)” is the
closest to the perfect reconstruction.

Fig. 5: Log spectral convergence by phase refinements. When the value is
$-\infty$, perfect reconstruction through STFT and inverse STFT is achieved.
This is the result of one of the evaluation datasets, but the same tendency
was observed in all evaluation datasets.

In addition, we compared speech quality of “PH” and “PH+GD.” Table 3 lists the
results of the comparison with frequency bands of 0-2, 0-4, and 0-8 kHz. “PH”
is preferred more in 0-8 kHz with statistical significance, but “PH+GD” is
preferred more in 0-2 and 0-4 kHz (without statistical significance).
Therefore, we can demonstrate the effectiveness of group-delay loss in speech
quality.

Table 3: Results of preference tests: effects of group-delay loss. Bold indicates preferred method that has a $p$-value smaller than $0.05$ Method A | Scores |  $p$-value | Method B
---|---|---|---
PH (2 kHz) | 0.487 vs. 0.513 | 0.514 | PH+GD (2 kHz)
PH (4 kHz) | 0.486 vs. 0.514 | 0.500 | PH+GD (4 kHz)
PH (8 kHz) |  0.545 vs. 0.455 | 0.031 | PH+GD (8 kHz)

##  5 Conclusion

This paper presented DNN-based phase reconstruction from an amplitude
spectrogram. Based on maximum likelihood estimation of the von Mises
distribution, we introduced two loss functions for DNN training: phase loss
and group-delay loss. We demonstrated 1) the trained DNNs can predict group
delay more accurately than phases, and 2) our proposed phase reconstruction
methods achieve better speech quality than the conventional Griffin-Lim phase
reconstruction method. For future work, we will implement other probability
distributions for periodic variables, propose other approaches for phase
refinements, and integrate our method with text-to-speech synthesis that
generates amplitude spectrograms [4].

Acknowledgments: Part of this work was supported by SECOM Science and
Technology Foundation, and JSPS KAKENHI Grant Number 18K18100.

## References

  * [1] H. Zen, K. Tokuda, and A. Black,  “Statistical parametric speech synthesis,”  Speech Communication, vol. 51, no. 11, pp. 1039–1064, 2009.
  * [2] S. Takaki, H. Kameoka, and J. Yamagishi,  “Direct modeling of frequency spectra and waveform generation based on phase recovery for DNN-based speech synthesis,”  in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017.
  * [3] Y. Wang, RJ Skerry-Ryan, D. Stanton, Y. Wu, Ron J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous,  “Tacotron: Towards end-to-end speech synthesis,”  vol. abs/1609.03499, 2017.
  * [4] Y. Saito, S. Takamichi, and H. Saruwatari,  “Text-to-speech synthesis using stft spectra based on low-/multi-resolution generative adversarial networks,”  in Proc. ICASSP, Calgary, Canada, Apr. 2018, pp. 5299–5303.
  * [5] D. W. Griffin and J. S. Lim,  “Signal estimation from modified short-time fourier transform,”  IEEE Transactions on Acoustics, Speech and Signal Processing, vol. 32, no. 2, pp. 236–243, Apr. 1984.
  * [6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio,  “Generative adversarial nets,”  Proc. NIPS, pp. 2672––2680, 2014.
  * [7] Y. Saito, S. Takamichi, and H. Saruwatari,  “Statistical parametric speech synthesis incorporating generative adversarial networks,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 1, pp. 755–767, Jun. 2018.
  * [8] Y. Li, K. Swersky, and R. Zemel,  “Generative moment matching networks,”  in Proc. ICML, Lille, France, Jul. 2015, pp. 1718–1727.
  * [9] S. Takamichi, K. Tomoki, and H. Saruwatari,  “Sampling-based speech parameter generation using moment-matching network,”  in Proc. INTERSPEECH, Stockholm, Sweden, Aug. 2017, pp. 3961–3965.
  * [10] H. Zen, A. Senior, and M. Schuster,  “Statistical parametric speech synthesis using deep neural networks,”  in Proc. ICASSP. Vancouver, Canada, May 2013.
  * [11] Z. Wu and S. King,  “Minimum trajectory error training for deep neural networks, combined with stacked bottleneck features,”  in Proc. INTERSPEECH, Dresden, Germany, Sep. 2015, pp. 309–313.
  * [12] K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda,  “The effect of neural networks in statistical parametric speech synthesis,”  in Proc. ICASSP, Brisbane, Australia, Apr. 2015, pp. 4455–4459.
  * [13] K. V. Mardia and P. E. Jupp,  Directional Statistics,  John Wiley & Sons Ltd., 1999.
  * [14] I.T. Nabney, C.M. Bishop, and C. Legleye,  “Modelling conditional probability distributions for periodic variables,”  in 1995 Fourth International Conference on Artificial Neural Networks, Calgary, Canada, Jun. 1995, pp. 177–182.
  * [15] F. Itakura and T. Umezaki,  “Distance measure for speech recognition based on the smoothed group delay spectrum,”  in Proc. ICASSP, Dallas, U.S.A., Apr. 1987, pp. 1257–1260.
  * [16] R. Padmanabhan, S. H. K. Parthasarathi, and H. A. Murthy,  “Robustness of phase based features for speaker recognition,”  in Proc. INTERSPEECH, Brighton, U. K., Sep. 2009, pp. 2355–2358.
  * [17] M. C. Jones and Arthur Pewsey,  “A family of symmetric distributions on the circle,”  Journal of the American Statistical Association, vol. 100, no. 472, pp. 1422–1428, Dec. 2005.
  * [18] T. Abe and A. Pewsey,  “Sine-skewed circular distributions,”  Statistical Papers, vol. 52, no. 3, pp. 683–707, Aug. 2011.
  * [19] R. Sonobe, S. Takamichi, and H. Saruwatari,  “JSUT corpus: free large-scale japanese speech corpus for end-to-end speech synthesis,”  vol. abs/1711.00354, 2017.
  * [20] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier,  “Language modeling with gated convolutional networks,”  vol. abs/1612.08083, 2016.
  * [21] X. Glorot, A. Bordes, and Y. Bengio,  “Deep sparse rectifier neural networks,”  in Proc. AISTATS, Lauderdale, U.S.A., Apr. 2011, pp. 315–323.
  * [22] L. A. Maas, Y. A. Hannun, and Y. A. Ng,  “Rectifier nonlinearities improve neural network acoustic models,”  in Proc. ICML, 2013, vol. 30.
  * [23] J. Duchi, E. Hazan, and Y. Singer,  “Adaptive subgradient methods for online learning and stochastic optimization,”  EURASIP Journal on Applied Signal Processing, vol. 12, pp. 2121––2159, 2011.
  * [24] Y. Stylianou,  “Applying the harmonics plus noise model in concatenative speech synthesis,”  IEEE Transactions on Speech and Audio Processing, vol. 9, no. 1, pp. 21–29, Jun. 2001.
  * [25] N. Sturmel and L. Daudet,  “Signal reconstruction from STFT magnitude: A state of the art,”  in Proc. of 14th International Conference on Digital Audio Effects DAFx-11, Paris, France, Sep. 2011, pp. 177–182.


# Graph Learning based Recommender Systems: A Review

Shoujin Wang1 Liang Hu2,3 Yan Wang2111Corresponding author Xiangnan He4 Quan
Z. Sheng1
Mehmet A. Orgun1 Longbing Cao5 Francesco Ricci6 Philip S. Yu7 1 Macquarie
University, 2 DeepBlue Academy of Sciences, 3 Tongji University
4University of Science and Technology of China, 5University of Technology
Sydney
6Free University of Bozen-Bolzano, 7University of Illinons at Chicago
{shoujin.wang, yan.wang, michael.sheng<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recent years have witnessed the fast development of the emerging topic of
Graph Learning based Recommender Systems (GLRS). GLRS employ advanced graph
learning approaches to model users’ preferences and intentions as well as
items’ characteristics for recommendations. Differently from other RS
approaches, including content-based filtering and collaborative filtering,
GLRS are built on graphs where the important objects, e.g., users, items, and
attributes, are either explicitly or implicitly connected. With the rapid
development of graph learning techniques, exploring and exploiting homogeneous
or heterogeneous relations in graphs are a promising direction for building
more effective RS. In this paper, we provide a systematic review of GLRS, by
discussing how they extract important knowledge from graph-based
representations to improve the accuracy, reliability and explainability of the
recommendations. First, we characterize and formalize GLRS, and then summarize
and categorize the key challenges and main progress in this novel research
area. Finally, we share some new research directions in this vibrant area.

##  1 Introduction

Recommender Systems (RS) are one of the most popular and important
applications of Artificial Intelligence (AI). They have been widely adopted to
help the users of many popular content sharing and e-Commerce web sites to
more easily find relevant content, products or services. Meanwhile, Graph
Learning (GL), which relates to machine learning applied to graph structure
data, is an emerging technique of AI which is rapidly developing and has shown
its great capability in recent years Wu et al. (2021). In fact, by benefiting
from these capabilities to learn relational data, an emerging RS paradigm
built on GL, i.e., Graph Learning based Recommender Systems (GLRS), has been
proposed and studied extensively in the last few years Guo et al. (2020). In
this paper we offer a systematic review of the challenges and progresses in
this emerging area.

Figure 1: The demonstration of graph learning based recommender systems

####  1.0.1 Motivation: why graph learning for RS?

##### Most of the data in RS has essentially a graph structure.

In the real world, most of the objects around us are explicitly or implicitly
connected with each other; in other words, we are living in a world of graphs.
Such characteristic is even more obvious in RS where the objects here
considered including users, items, attributes, context, are tightly connected
with each other and influence each other via various relations Hu et al.
(2014), as shown in Figure 1. In practice, various kinds of graphs arise from
the data used by RS, and they can significantly contribute to the quality of
the recommendations.

##### Graph learning has the capability to learn complex relations.

As one of the most promising machine learning techniques, GL has shown great
potential in deriving knowledge embedded in different kinds of graphs.
Specifically, many GL techniques, such as random walk and graph neural
networks, have been developed to learn the particular type of relations
modeled by graphs, and have demonstrated to be quite effective Wu et al.
(2021). Consequently, employing GL to model various relations in RS is a
natural and compelling choice.

####  1.0.2 Formalization: how does graph learning can help RS?

To date, there is no unified formalization of GLRS. We generally formalize
GLRS from a high-level perspective.

We construct a graph $\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$ with the data
of an RS where the objects, e.g., users and items, are represented as nodes in
$\mathcal{V}$ and the relations between them, e.g., purchases, are represented
as edges in $\mathcal{E}$. Then, a GLRS model $M(\Theta)$ is constructed and
trained to generate optimal recommendation results $\mathcal{R}$ with
optimized model parameters $\Theta$ that are learned from the topological and
content information of $\mathcal{G}$. Formally,

$\mathcal{R}=\arg\max_{\Theta}f(M(\Theta)|\mathcal{G}).$ (1)

Depending on the specific recommendation data and scenarios, the graph
$\mathcal{G}$ and the recommendation target $\mathcal{R}$ can be defined in
various forms, e.g., $\mathcal{G}$ can be homogeneous sequences or
heterogeneous networks while $\mathcal{R}$ can be predicted ratings or ranking
over items. The objective function $f$ can be the maximum utility Wang et al.
(2019f) or the maximum probability to form links between nodes Verma et al.
(2019).

Contributions. The main contributions of this work are summarized below:

  * •

We systematically analyze the key challenges presented by various GLRS graphs
and categorize them from a data driven perspective, providing a useful view to
better understand the important characteristics of GLRS.

  * •

We summarize the current research progress in GLRS by systematically
categorizing the more technical state-of-the-art literature.

  * •

We share and discuss some open research directions of GLRS for giving
references to the community.

##  2 Data Characteristics and Challenges

Different objects are managed by an RS, e.g., users, items, attributes. All of
them are inter-connected with various types of relations Hu et al. (2019),
e.g., social relations between users, or interactions between users and items.
This results in different types of graphs that may be considered in an RS. In
this section, we first classify the various types of data used in RS by
considering their source and characteristics. For each class, we analyze its
characteristics, then discuss how to better represent it with graphs, and
finally indicate challenges that these characteristics pose to building GLRS.
A brief summary of these data types is provided in Table 1.

It is well known that the three key objects managed by an RS are user, item
and user-item interaction (interaction for short), and thus all the data
managed by an RS is related to them. There are two broad types of data: user-
item interaction data, e.g., clicks and ratings of users for items, and side
information data, e.g., users’ profiles and items’ attributes Shi et al.
(2014). Depending on whether the temporal order of the interactions is
recorded or not, interaction data can be classified into sequential
interaction data and general interaction data. Hence, we classify the data of
an RS into three classes: (1) general interaction data, (2) sequential
interaction data, and (3) side information data. Each class can be further
divided into multiple sub-classes (cf. Table 1).

Table 1: A summary of data in RS, the representing graph, and the
corresponding GLRS approach

Data Class | Data Subclass | Representing Graph | Representative Approach Category
---|---|---|---
General interaction | Explicit interaction | Weighted bipartite graph | Graph auto-encoder[1], GCN[2], GraphSage[3]
Implicit interaction | Unweighted bipartite graph | Random walk[4], graph embedding[5], GCN[6], GraphSage[7]
Sequential interaction | Single-type interactions | Directed homogeneous graph | GGNN[8], GraphSage[9], GAT[10]
Multi-type interactions | Directed heterogeneous graph | GraphSage[11]
Side information | Attribute information | Heterogeneous graph | Graph embedding[12], GAT[13]
Social information | Homogeneous graph | Random walk[14], graph embedding[15], GAT[16]
External knowledge | Tree or heterogeneous graph | Graph embedding[17], GCN[18]
[1]Berg et al. (2017);[2]Monti et al. (2017);[3]Zhang and Chen
(2020);[4]Nikolakopoulos and Karypis (2019);[5]Chen et al. (2019);[6]He et al.
(2020);[7]Zheng et al. (2018);[8]Wu et al. (2019b);[9]Ma et al. (2020)
[10]Qiu et al. (2019);[11]Wang et al. (2020b);[12]Shi et al. (2018);[13]Wang
et al. (2019f);[14]Jamali and Ester (2009);[15]Wen et al. (2018);[16]Fan et
al. (2019);[17]Gao et al. (2019);[18]Wang et al. (2019b)

###  2.1 GLRS Built on General Interaction Data

Interactions between users and items are usually represented as an interaction
matrix, where each row indicates one user and each column indicates one item.
Each entry in the matrix captures an information about the type of occurred
interaction. Depending on the interaction type, the interaction data can be
divided into the explicit (i.e., users’ ratings on items) and the implicit
(e.g., click, view) Zhang et al. (2019b). Then, the recommendation task based
on this general interaction data is usually formulated as a matrix completion
task Zhang and Chen (2020).

An interaction matrix can be naturally represented as a user-item bipartite
graph Zha et al. (2001). In this graph, the user nodes and the item nodes
constitute the two “parts” respectively, while the interactions are
represented as edges connecting two nodes in different parts. Furthermore, an
explicit interaction matrix can be represented as a weighted bipartite graph
where each edge is labeled with a weight to indicate the rating value. An
implicit interaction matrix can be represented as an unweighted bipartite
graph where en edge indicates a implicit interaction. Hence, from this graph
based perspective, the recommendation task is converted to link prediction on
the RS bipartite graph Li and Chen (2013).

The advantage of building a GLRS on a bipartite graph is obvious. Since most
users often interacted with only a small proportion of the large amount of
items, matrix completion methods generally face data sparsity related and
cold-start problems as discussed in Jamali and Ester (2009). A bipartite graph
based approach mitigates these issues by enabling the information propagating
widely among nodes to enrich the information of those users and items with
less interactions Wu et al. (2020b). However, it is challenging how to
effectively and efficiently propagate the information between users or items.
This is particularly challenging in a bipartite graph since no direct links
exist between users or items, and thus the information should be propagated
via multi-hop neighbour nodes. For instance, to propagate some information
from user $u_{1}$ to a similar user $u_{2}$ one needs to first propagate it to
a bridge item $v_{1}$ connecting both users, and then to $u_{2}$ from $v_{1}$.

While targeting this challenge, a variety of GLRS approaches have been
developed. For weighted bipartite graphs, there are mainly graph auto-encoder
(e.g., graph convolutional matrix completion Berg et al. (2017)), Graph
Convolutional Networks (GCN) (e.g., multi-graph convolutional neural networks
Monti et al. (2017), stacked and reconstructed GCN Zhang et al. (2019a)), and
Graph SAmple and aggreGatE (GraphSage) (e.g., inductive graph-based matrix
completion Zhang and Chen (2020)). For unweighted bipartite graphs, there are
mainly random walk (e.g., RecWalk Nikolakopoulos and Karypis (2019)), graph
embedding (e.g., high-order proximity for implicit recommendation Yang et al.
(2018), collaborative similarity embedding Chen et al. (2019)), GCN (e.g.,
spectral collaborative filtering Zheng et al. (2018), lightGCN He et al.
(2020), low-pass collaborative filter Yu and Qin (2020), multi-behavior GCN
Jin et al. (2020)), and GraphSage (e.g., neural graph collaborative filtering
Zheng et al. (2018)). The gist of these approaches will be discussed in
Section 3.

###  2.2 GLRS Built on Sequential Interaction Data

A sequential interaction data set is a collection of sequences of user-item
interactions (e.g., click, purchase) registered during a given time period,
and ordered by their time stamp. According to the number of interaction types
included in a sequence, a sequential interaction data set can be divided into
single-type interaction data set where only one type of interactions is
included, and multi-type interaction data set where multiple types of
interactions are included. Multi-type interactions like view, click and
purchase co-happening in one sequence are very common in practice Wang et al.
(2020b). For a given user $u$, a single-type interaction sequence is usually
recorded as a sequence of interacted with items (denoted as $v$), e.g.,
$\\{v_{1},...,v_{n}\\}$, while a multi-type interaction sequence is recorded
as a sequence of $\langle interaction\ type,item\rangle$ pairs, e.g.,
$\\{click\ v_{1},click\ v_{2},...,purchase\ v_{n}\\}$. An RS built on
sequential interaction data is formalized as a Sequential Recommender System
(SRS) which takes a sequence of historical interactions as input to predict
the possible next interaction(s) Quadrana et al. (2018); Wang et al. (2019e).

A sequential interaction data set can be represented as a directed graph where
each interaction sequence corresponds to one path in the graph Wu et al.
(2019b). In each path, the interactions serve as the nodes and a directed edge
between any adjacent nodes indicates the order of interactions. In a multi-
type interaction sequence, each element is a $\langle interaction\
type,item\rangle$ pair, which results in a compound node composed of two
parts. Note that in some cases one user may have multiple identical
interactions happening in a sequence (e.g., click the same items multiple
times), resulting in a path consisting of one or more loops Wu et al. (2019b).

The advantages of building SRS on directed graph lies in the strong capability
of graph learning to represent and model even the most complicated transitions
in a sequence of interactions. There are usually complicated transitions which
deviate from simple one-way consecutive time series patterns Wang et al.
(2020a) over sequential interactions, especially when there are multiple
identical interactions in one sequence Wu et al. (2019b). Such transitions can
be well represented by the multi-direction connections in a graph and well
learned by the information aggregation from neighbour nodes of different
directions in graph learning Wu et al. (2019b); Xu et al. (2019). However,
building SRS on a directed graph is still challenging. In particular it is
critical how to construct a graph to effectively represent the sequential
interaction data with minimal information loss, and how to propagate
information on the graph to effectively model even the most complicated
transitions.

While targeting these challenges, various SRS have been built based on graph
learning. Most of the studied approaches focus on single-type interaction
data, including Gated Graph Neural Networks (GGNN) (e.g., session-based
recommendation with GNN Wu et al. (2019b) and graph contextualized self-
attention networks Xu et al. (2019)), GraphSage (e.g., memory augmented GNN Ma
et al. (2020)), and Graph ATtention networks (GAT) (e.g., full graph neural
network Qiu et al. (2019)). Limited approaches for multi-type interaction data
include GraphSage (e.g., multi-relational GNN for session-based prediction
Wang et al. (2020b)).

###  2.3 GLRS Incorporating Side Information Data

Interaction data is often sparse Hu et al. (2019), thus is not sufficient for
correctly capturing the users’ preferences and item characteristics. Hence,
various types of side information, e.g., attribute information and social
information, have been used to alleviate such an issue. In this section, we
discuss three main types of side information: (1) attribute information, (2)
social information, and (3) external knowledge.

####  2.3.1 GLRS Incorporating Attribute Information

Attribute information mainly includes user attributes (e.g., gender, age), and
item attributes (e.g., category, price) Wang et al. (2017); Han et al. (2018).
A user (item) attribute data set is usually recorded as a user (item)
information table where each row indicates one user (item) and each column is
one attribute. Attribute information is often combined with general or
sequential interaction data to perform recommendations. Given a data set, the
combination of interaction data and attribute data naturally results in a
heterogeneous graph. In such a graph, three types of nodes, i.e., user node,
item node and attribute value node, and at least two types of edges exist.
Specifically, in the combination of general interaction data and attribute
data, in addition to user-item edges (cf. Sec. 2.1), there are user (or
item)-attribute value edge representing the relations between user (or item)
and attributes. In the combination of sequential interaction data and
attribute data, in addition to the directed interaction-interaction edges (cf.
Sec. 2.2), there are also item-attribute value edges. Consequently, the
recommendation task here becomes the prediction of the interactions by
learning the complex relations embedded in the above mentioned heterogeneous
graph.

Heterogeneous graphs combine two different types of information, i.e.,
interaction information and attribute information, hence enabling information
propagation among different types of nodes, and better coping with the
mentioned data sparsity problem. However, it is challenging to selectively
aggregate those useful attribute information to improve the recommendation
performance.

GLRS targeting such a challenge include (heterogeneous) graph embedding (e.g.,
entity2rec Palumbo et al. (2017) based on node2vec, heterogeneous preference
embedding Chen et al. (2016) and heterogeneous network embedding for
recommendation Shi et al. (2018)), and GAT (e.g., knowledge graph attention
network Wang et al. (2019f)).

####  2.3.2 GLRS Incorporating Social Information

Social information relates to the commonly existing social relations between
users. A particular type of social relation among the users in a data set
naturally forms a homogeneous social graph where each user corresponds to a
node and each social link (e.g., friend relation) between two users
corresponds to an edge. In an RS, the social graph can be mainly used for two
tasks: (1) social recommendation (recommending items to users by incorporating
social information) Fan et al. (2019), and (2) friend recommendation
(recommending users to a given user by predicting the possible social links)
Huang et al. (2015).

##### Social recommendation.

Social relations enable social influence diffusion among users Wu et al.
(2020a) and thus help better understand users’ preferences. The combination of
social information and general or sequential user-item interaction data
naturally results in a heterogeneous graph comprising two parts. The first is
the bipartite graph derived from the general interaction data (cf. Sec. 2.1)
or the directed graph extracted from the sequential interaction data (cf. Sec.
2.2), while the second part is the social graph connecting the users.
Obviously, two heterogeneous types of information (i.e., interaction
information and social information) are contained in the graph. Hence, the RS
must be able to effectively leverage this heterogeneous graph to predict the
unknown user-item interactions.

Such an approach helps better understand a user’s preference by considering
the influence of her neighbours in a social graph. However, on one hand, it is
not clear how many orders of neighbours should be considered to correctly
compute this influence on a given user. On the other hand, different
neighbours usually influence a user to different degrees Wu et al. (2020b).
Hence, it is a challenge to appropriately model the influence of other users
to a given user.

Typical approaches targeting this challenge include random walk (e.g., Trust-
walker Jamali and Ester (2009)), graph embedding Wen et al. (2018) and GAT
(e.g., GraphRec Fan et al. (2019) and improved diffusion network Wu et al.
(2020a)). All these works focus on combining social graph and general
interactions, while limited works combine social graph and sequential
interactions Song et al. (2019).

##### Friend recommendation.

By using the aforementioned homogeneous social graph, friend recommendation is
performed as a link prediction task on such graph Yin et al. (2010).
Specifically, given a target user and the known social links on the graph,
friend recommendation first infers the possible links between other users and
the target user and then recommends those users with high probabilities to
link with the target user to her. The main challenge lies in how to
appropriately model the mutual-influence between users. According to our
analysis, random walk based approaches Backstrom and Leskovec (2011); Bagci
and Karagoz (2016) are more common in order to address this challenge. Other
approaches include graph embedding Verma et al. (2019).

####  2.3.3 GLRS Incorporating External Knowledge

External knowledge, e.g., item taxonomy and semantic relations between
concepts, related to users and items usually contributes to a deeper
understanding of the users’ preference and item characteristics Wang et al.
(2018a), and ultimately improving recommendation performance. Such knowledge
is usually represented as a knowledge graph where various types of objects
(e.g., users, movies, movie directors) are represented as nodes and the
relations between them (e.g., movie-director relation) are represented as
edges Wang et al. (2019f). This graph is often combined with the graph
composed by the general or sequential interaction data, giving rise to a more
complex and heterogeneous graph. There are mainly two types of external
knowledge commonly utilized in RS: item/user ontology and common knowledge.

##### GLRS incorporating ontology knowledge.

The ontology of users or items is usually represented as a hierarchical tree-
like graph where the hierarchical relations between users or items are
recorded. A type of commonly utilized ontology knowledge for recommendations
is item taxonomy information Huang et al. (2019). An example of such a tree
graph is used in Amazon.com, where the category information of products is
used to organize all the items offered by the platform. In that graph, the
root node corresponds to the coarsest-grained category and the leaf nodes
represent specific items.

The incorporation of item ontology knowledge enables a better understanding of
the users’ multi-level preferences towards items, and thus helps improving the
explainability of the recommendations Gao et al. (2019). However, it remains a
challenge to propagate users’ preferences over items along the hierarchy tree
graph to extract the multi-level preferences.

Representative works targeting such a challenge include graph embedding based
approaches Wang et al. (2018b); Gao et al. (2019), aimed at learning more
informative item embedding for general recommendations, and memory network on
graphs to learn coarse-grained-preference representation for sequential
recommendations Huang et al. (2019).

##### GLRS incorporating common knowledge.

Common knowledge refers to the wide range of relations between the various
entities managed by an RS. It includes, but is not limited to, general
semantic relations between entities (e.g., the relations among bread, food,
bakery item from Microsoft Concept
Graph222https://concept.research.microsoft.com/) Sheu and Li (2020), and
domain-specific relations between entities (e.g., the relations between
movies, directors, genre) Gao et al. (2020). Due to the diversity of these
entities and their relations, common knowledge is usually represented as a
heterogeneous and complex graph where different types of nodes and edges exist
Guo et al. (2020).

The incorporation of common knowledge benefits the exploration and
exploitation of various external implicit relations between users and/or
items, improving recommendation performance. However, it remains a challenge
to effectively propagate information between different types of entities via
different types of links between them, to obtain coherent and useful
information for the recommendations.

Representative works targeting this challenge include graph embedding methods
Wang et al. (2019c) (especially meta-path based embedding Zhao et al. (2017);
Sun et al. (2018); Shi et al. (2018); Wang et al. (2019g)) to wisely learn the
embedding of heterogeneous entities and relations, and GNN based methods
(especially GCN Wang et al. (2019b) an GAT Wang et al. (2019f)) to iteratively
aggregate the information from neighbour nodes.

##  3 Graph Learning Approaches for RS

In this section, we introduce graph learning based techniques, which offer
solutions to the challenges faced by GLRS, which were discussed in Section 2.
We first provide a technical categorization of the solutions, and then we
discuss the gist of each solution together with the achieved progresses.

The categorization of the approaches to GLRS is presented in Figure 2. GLRS
are divided into three categories, and some categories are further divided
into sub-categories.

###  3.1 Random Walk Approach

Random walk based RS have been extensively studied in the past years and have
been widely employed on various types of graphs (e.g., social graphs, sequence
graphs). Generally, a random walk based RS first let a random walker to walk
on a given graph with a predefined transition probability for each step, in
order to model the implicit preference or interaction propagation among users
and/or items, and then takes the probability the walker lands on nodes after
certain steps to rank these candidate nodes for recommendations. Random walk
based RS are particularly suitable for capturing the complex, higher-order and
indirect relations among various types of nodes (e.g., users and items) on the
graph, and thus, can address important challenges for GLRS especially those
built on heterogeneous graphs.

There are different variants of random walk based RS. Besides the basic random
walk based RS Baluja et al. (2008), random walk with restart based RS Bagci
and Karagoz (2016); Jiang et al. (2018) is a representative type of several
variants. It sets a constant probability to jump back to the starting node in
each transition and it is generally used in graphs containing many nodes to
avoid leaving the particular context of the starting node.

Although widely applied, the drawbacks of random walk based RS are clear: (1)
they need to generate ranking scores on all candidate items at each step for
each user, leading to low efficiency; (2) unlike most of the learning-based
paradigms, they are heuristic-based, lacking model parameters to optimize the
recommendation objective.

###  3.2 Graph Embedding Approach

Graph embedding is an effective technique to analyze the complex relations
embedded on graphs and has been rapidly developing in recent years. It maps
each node into a low-dimension embedding vector which encodes the graph
structure information. Researchers introduced graph embedding to model the
complex relations between various nodes (e.g., users, items) and they came up
with the novel approach of Graph Embedding based RS (GERS). Depending to the
specific embedding approach that is used, GERS can be divided into three
classes: (1) Graph Factorization based RS (GFRS), (2) Graph Distributed
Representation based RS (GDRRS), and (3) Graph Neural Embedding based RS
(GNERS).

Figure 2:  Classifying GLRS approaches from technical perspective

##### Graph Factorization based RS (GFRS).

GFRS first factorizes the inter-node commuting matrix based on meta-path on
the graph in order to obtain the embedding of each node (e.g., a user or an
item), which are then used as input of the subsequent recommendation task Wang
et al. (2019h). By doing so, the complex relations between nodes in the graph
are encoded into the embedding to improve the recommendations. Due to their
capability to handle the heterogeneity of the nodes, GFRS have been widely
applied to capture relations between different types of nodes, e.g., users and
items. However, although being simple and effective, such models may easily
suffer from the sparsity of the observed data.

##### Graph Distributed Representation based RS (GDRRS).

Differently from GFRS, GDRRS usually follow Skip-gram model Mikolov et al.
(2013) in order to learn a distributed representation of each user or item in
a graph. They encode information about the user or item and its adjacent
relations into a low-dimensional vector Shi et al. (2018), which is then used
for the subsequent recommendation step. Specifically, GDRRS usually first use
random walk to generate a sequence of nodes that co-occurred in one meta-path
and then employ the skip-gram or similar models to generate node
representations for recommendations. By exploiting its powerful capability to
encode the inter-node connections in a graph, GDRRS are widely applied to both
homogeneous and heterogeneous graphs for capturing the relations between the
objects managed by the RS Cen et al. (2019). GDRRS have shown their great
potential in recent years due to their simplicity, efficiency and efficacy.

##### Graph Neural Embedding based RS (GNERS).

GNERS utilize neural networks, like Multilayer-perceptron, auto-encoder, to
learn users or items embedding. Neural embedding models are easy to integrated
with other downstream neural recommendation models (e.g., RNN based ones) to
build an end-to-end RS Han et al. (2018). To this end, GNERS have been widely
applied to a variety of graphs like attributed graphs Han et al. (2018),
interaction combined with knowledge graphs Hu et al. (2018); Cen et al.
(2019).

###  3.3 Graph Neural Network Approach

Graph Neural Networks (GNN) apply neural networks techniques on graph data.
Leveraging the strength of GNN in learning informative representations,
several RS have used GNN to address the most important challenges posed by
GLRS. By considering a model perspective, GNN based RS can be mainly
categorized into three classes: (1) Graph ATtention network based RS (GATRS),
(2) Gated Graph Neural Network based RS (GGNNRS), and (3) Graph Convolutional
Network (including GraphSage) based RS (GCNRS).

Table 2: A list of representative open-source GLRS algorithms

Algorithm | Input Data | Learning Task | Learning Approach | Venue | Link
---|---|---|---|---|---
GC-MC[1] | Explicit interaction | Rating prediction | Graph auto-encoder | KDD’2018 DL | https://github.com/riannevdberg/gc-mc
MGCNN[2] | Explicit interaction | Rating prediction | GCN | NIPS’2017 | https://github.com/fmonti/mgcnn
IGMC[3] | Explicit interaction | Rating prediction | GraphSage | ICLR’2020 | https://github.com/muhanzhang/IGMC
RecWalk[4] | Implicit interaction | Click prediction | Random walk | WSDM ’2019 | https://github.com/nikolakopoulos/RecWalk
PinSage[5] | Implicit interaction | Click prediction | GraphSage | KDD’2018 | https://github.com/yoonjong12/pinsage
CSE[6] | Implicit interaction | Click prediction | Graph embedding | WWW’2019 | https://github.com/cnclabs/proNet-core
LightGCN[7] | Implicit interaction | Click prediction | GCN | SIGIR’2020 | https://github.com/kuandeng/LightGCN
SpectralCF[8] | Implicit interaction | Click prediction | GraphSage | RecSys’2018 | https://github.com/lzheng21/SpectralCF
SR-GNN[9] | Single-type sequential interaction | Next-item prediction | GGNN | AAAI’2019 | https://github.com/CRIPAC-DIG/SR-GNN
MA-GNN[10] | Single-type sequential interaction | Next-item prediction | GraphSage | AAAI’2020 | https://github.com/cynricfu/MAGNN
FGNN[11] | Single-type sequential interaction | Next-item prediction | GAT | CIKM’2019 | https://github.com/RuihongQiu/FGNN
MGNN-SPred[12] | Multi-type sequential interaction | Next-item prediction | GraphSage | WWW’2020 | https://github.com/Autumn945/MGNN-SPred
HERec[13] | Explicit interaction + Attribute information | Rating prediction | Graph embedding | TKDE’2018 | https://github.com/librahu/HERec
KGAT[14] | Implicit interaction + Attribute information | Click prediction | GAT | KDD’2019 | https://github.com/xiangwang1223/
TrustWalker[15] | Explicit interaction + Trust relation | Rating prediction | Random walk | KDD’2009 | https://github.com/Antili/TrustWalker
GraphRec[16] | Explicit interaction + Social relation | Rating prediction | GAT | WWW’2019 | https://github.com/wenqifan03/GraphRec-WWW19
KGNN-LS[17] | Implicit interaction + External knowledge | Click prediction | GAT | KDD’2019 | https://github.com/hwwang55/KGNN-LS
[1] Berg et al. (2017);[2]Monti et al. (2017);[3]Zhang and Chen
(2020);[4]Nikolakopoulos and Karypis (2019);[5]Ying et al. (2018); [6]Chen et
al. (2019);[7]He et al. (2020);[8]Zheng et al. (2018);[9]Wu et al. (2019b);
[10]Ma et al. (2020);[11]Qiu et al. (2019);[12]Wang et al. (2020b);[13]Shi et
al. (2018);[14]Wang et al. (2019f);[15]Jamali and Ester (2009);[16]Fan et al.
(2019);[17]Wang et al. (2019b)

  * 2

2\. More up-to-date resource on GLRS can be found at:
https://github.com/shoujin88/graph-learning-based-recommender-systems-GLRS-.

##### Graph ATtention network based RS (GATRS).

Graph ATtention networks (GAT) introduce attention mechanisms into GNN to
discriminatively learn the different relevance and influence degree of other
users (items) w.r.t. the target user (item) on a given graph. GATRS are based
on GAT for precisely learning inter-user or item relations. In such a case,
the influence of the more important users or items, w.r.t. a specific user or
item, is emphasized, which is more in line with the real-world cases and this
has been shown to be beneficial for the recommendations. Due to their good
discrimination capability, GAT are widely used in different kinds of graphs
including social graphs Fan et al. (2019), item session graphs Xu et al.
(2019), and knowledge graphs Wang et al. (2019f).

##### Gated Graph Neural Network based RS (GGNNRS).

Gated graph neural networks (GGNN) introduce the Gated Recurrent Unit (GRU)
into GNN to learn the optimized node representations by iteratively absorbing
the influence of other nodes in a graph to comprehensively capture the inter-
node relations. GGNNRS are built on GGNN to learn the user or item embeddings
for recommendations by comprehensively considering the complex inter-user or
inter-item relations. Due to their capability to capture the complex relations
between nodes, GGNN are widely used to model the complex transitions between
items in a sequence graph for sequential recommendations Wu et al. (2019b), or
to model the complex interactions between different categories of fashion
products for fashion recommendations Cui et al. (2019), and they have achieved
superior recommendation performance.

##### Graph Convolutional Network based RS (GCNRS).

Graph Convolutional Networks (GCN) generally learn how to iteratively
aggregate feature information from local graph neighbor nodes by leveraging
both graph structure and node feature information. In general, by utilizing
the convolution and pooling operations, GCNs are capable of learning
informative embeddings of users and items by effectively aggregating
information from their neighborhoods in graphs. GCNRS are built on GCN to
learn the user or item embeddings in a graph while exploiting both the complex
relations between users or/and items and their own content information for
recommendations Ying et al. (2018). Thanks to the powerful feature extraction
and learning capability, particularly their strength in combining the graph
structure and node content information, GCN are widely applied to a variety of
graphs in RS to build GCNRS and are demonstrated to be very effective. For
instance, GCN are used for influence diffusion on social graphs in social
recommendations Wu et al. (2019a), for mining the hidden user-item connection
information on user-item interaction graphs, for alleviating the data sparsity
problemn in collaborative filtering Wang et al. (2019a), and for capturing
inter-item relatedness by mining their associated attributes on knowledge
graphs Wang et al. (2019b).

##  4 GLRS Algorithms and Datasets

The source code of most of the representative GLRS algorithms is publicly
accessible. In Table 2, to facilitate the access for empirical analysis, we
summarize source codes of algorithms for GLRS which take various input data
and use different learning approaches for different learning tasks. The listed
algorithms are carefully selected and are commonly used as baselines in
existing work.

In addition to algorithms, datasets are another important part for empirical
analysis of GLRS approaches. In order to facilitate the empirical analysis of
the surveyed algorithms, in Table 3 we also list public and real-world
datasets with different characteristics from various domains. These datasets
are commonly used for evaluating GLRS algorithms.

Table 3: A list of commonly used and publicly accessible real-world datasets
for GLRS

Dataset | Domain | Information Included | # Interactions | Reference | Link
---|---|---|---|---|---
MovieLens-1M | Movie | Explicit interaction | 1,000,209 | Zheng et al. (2018) | https://grouplens.org/datasets/
HetRec | Movie | Explicit interaction | 855,598 | Zheng et al. (2018) | https://grouplens.org/datasets/
Amazon instant video | Video | Explicit interaction | 583,933 | Zheng et al. (2018) | http://jmcauley.ucsd.edu/data/amazon/
Gowalla | POI | Implicit interaction | 1,027,370 | He et al. (2020) | http://snap.stanford.edu/data/loc-gowalla.html
Yelp 2018 | POI | Implicit interaction | 1,561,406 | He et al. (2020) | https://www.yelp.com/dataset
Amazon-book | E-commerce | Implicit interaction | 2,984,108 | He et al. (2020) | https://github.com/uchidalab/book-dataset
Yoochoose 1/4 | E-commerce | Stream of clicks | 8,326,407 | Wu et al. (2019b) | http://2015.recsyschallenge.com/challege.html
Diginetica | E-commerce | Stream of clicks | 982,961 | Wu et al. (2019b) | https://competitions.codalab.org/competitions/11161
Book-crossing | Book | Ratings and attribute information | 1,000,000 | Wang et al. (2019b) | http://www2.informatik.uni-freiburg.de/~cziegler/BX/
Last.FM | Music | Implicit interaction, social and tag | 92,834 | Wang et al. (2019b) | https://grouplens.org/datasets/hetrec-2011/
Epinions | E-commerce | Rating, trust relation | 764,352 | Fan et al. (2019) | http://alchemy.cs.washington.edu/data/epinions/
Ciao | E-commerce | Rating, trust relation | 283,319 | Fan et al. (2019) | https://www.cse.msu.edu/~tangjili/datasetcode/truststudy.htm
Amazon-toys and games | E-commerce | Implicit interaction | 167,597 | Gao et al. (2019) | http://jmcauley.ucsd.edu/data/amazon
Amazon-digital music | Music | Implicit interaction | 64,706 | Gao et al. (2019) | http://jmcauley.ucsd.edu/data/amazon

##  5 Open Research Directions

GLRS are fast developing. Although substantial results have been achieved,
some challenges still remain. By matching the demonstrated challenges to the
research progress already achieved, we have identified some open research
directions.

##### Self-evolutionary RS with dynamic-graph learning.

In real-world RS, users, items and the interactions between them, keep
evolving over time Wang et al. (2019d). This originates graphs with dynamic
topology, and such dynamics could have direct impacts on the user and
requirement modeling, causing even a clear change of recommendation results
over time. However, this issue is still underestimated in existing GLRS.
Therefore, it is a promising future research direction to design self-
evolutionary RS over dynamic graphs.

##### Explainable RS with causal graph learning.

Causal inference is a major technique used to discover the causal relations
between objects or actions. Although some progress has been achieved in
explainable RS, we are is still far away from achieving a complete
understanding of the reasons and intents behind user choice behaviours, which
is a critical step to make reliable and explainable recommendations Zhang and
Chen (2018). To this end, it is another promising direction to construct
explainable RS with causal graph learning.

##### Cross-domain RS with multiplex graph learning.

In reality, the data and interactions for recommendation could be derived from
multiple domains, including various sources, systems, and modalities Zhu et
al. (2019). These are inter-correlated and must collaboratively contribute to
the recommendations Zhu et al. (2021). Consequently, the interactions in
cross-domain RS can be represented by multiplex networks where nodes may or
may not be interconnected with other nodes in other layers. As a result, the
new generation cross-domain RS potentially works with multiplex graph
learning.

##### High-efficiency online RS with large-scale graph learning.

An inevitable issue in real RS is the scale of data, which is often large and
leads to high cost in terms of both time and space. This issue is even more
important in GLRS since the graph structure data is usually even larger and
requires more time and space to be processed, let alone to perform complex
machine learning techniques on it to generate recommendations. Therefore, it
is necessary to study more efficient algorithms to speed up large-scale online
processing and learning to keep updating model to generate timely
recommendations.

##  6 Conclusions

As one of the most important applications of Artificial Intelligence (AI),
Recommender Systems (RS) can be found nearly at every corner of our daily
lives. Graph Learning (GL), as one of the most promising AI techniques, has
shown a great capability to learn the complex relations among the various
objects managed by an RS. This has launched a totally new RS paradigm: Graph
Learning based Recommender Systems (GLRS), which is of great potential to be
the next-generation of RS. It is our hope that this review has provided a
comprehensive and self contained overview of the recent progress, challenges
as well as future research directions in GLRS to both the academia and
industry.

## Acknowledgements

This work was supported by ARC Discovery Project DP180102378.

## References

  * Backstrom and Leskovec [2011] Lars Backstrom and Jure Leskovec.  Supervised random walks: predicting and recommending links in social networks.  In WSDM, pages 635–644, 2011.
  * Bagci and Karagoz [2016] Hakan Bagci and Pinar Karagoz.  Context-aware friend recommendation for location based social networks using random walk.  In WWW, pages 531–536, 2016.
  * Baluja et al. [2008] Shumeet Baluja, Rohan Seth, Dharshi Sivakumar, and et al.  Video suggestion and discovery for youtube: taking random walks through the view graph.  In WWW, pages 895–904, 2008.
  * Berg et al. [2017] Rianne van den Berg, Thomas N Kipf, and Max Welling.  Graph convolutional matrix completion.  arXiv preprint arXiv:1706.02263, 2017.
  * Cen et al. [2019] Yukuo Cen, Xu Zou, Jianwei Zhang, Hongxia Yang, Jingren Zhou, and Jie Tang.  Representation learning for attributed multiplex heterogeneous network.  In SIGKDD, pages 1358–1368, 2019.
  * Chen et al. [2016] Chih-Ming Chen, Ming-Feng Tsai, Yu-Ching Lin, and Yi-Hsuan Yang.  Query-based music recommendations via preference embedding.  In RecSys, pages 79–82, 2016.
  * Chen et al. [2019] Chih-Ming Chen, Chuan-Ju Wang, Ming-Feng Tsai, and Yi-Hsuan Yang.  Collaborative similarity embedding for recommender systems.  In WWW, pages 2637–2643, 2019.
  * Cui et al. [2019] Zeyu Cui, Zekun Li, Shu Wu, Xiao-Yu Zhang, and Liang Wang.  Dressing as a whole: Outfit compatibility learning based on node-wise graph neural networks.  In WWW, pages 307–317, 2019.
  * Fan et al. [2019] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, and et al.  Graph neural networks for social recommendation.  In WWW, pages 417–426, 2019.
  * Gao et al. [2019] Jingyue Gao, Xiting Wang, Yasha Wang, and et al.  Explainable recommendation through attentive multi-view learning.  In AAAI, pages 3622–3629, 2019.
  * Gao et al. [2020] Yang Gao, Yi-Fan Li, Yu Lin, Hang Gao, and Latifur Khan.  Deep learning on knowledge graph for recommender system: A survey.  arXiv preprint arXiv:2004.00387, 2020.
  * Guo et al. [2020] Qingyu Guo, Fuzhen Zhuang, Chuan Qin, and et al.  A survey on knowledge graph-based recommender systems.  IEEE Transactions on Knowledge and Data Engineering, 2020.  doi: 10.1109/TKDE.2020.3028705.
  * Han et al. [2018] Xiaotian Han, Chuan Shi, Senzhang Wang, S Yu Philip, and Li Song.  Aspect-level deep collaborative filtering via heterogeneous information networks.  In IJCAI, pages 3393–3399, 2018.
  * He et al. [2020] Xiangnan He, Kuan Deng, Xiang Wang, and et al.  Lightgcn: Simplifying and powering graph convolution network for recommendation.  In SIGIR, pages 639–648, 2020.
  * Hu et al. [2014] Liang Hu, Jian Cao, Guandong Xu, and et al.  Deep modeling of group preferences for group-based recommendation.  In AAAI, pages 1861–1867, 2014.
  * Hu et al. [2018] Binbin Hu, Chuan Shi, Wayne Xin Zhao, and Philip S Yu.  Leveraging meta-path based context for top-n recommendation with a neural co-attention model.  In SIGKDD, pages 1531–1540, 2018.
  * Hu et al. [2019] Liang Hu, Songlei Jian, Longbing Cao, Zhiping Gu, and et al.  Hers: Modeling influential contexts with heterogeneous relations for sparse and cold-start recommendation.  In AAAI, pages 3830–3837, 2019.
  * Huang et al. [2015] Shangrong Huang, Jian Zhang, Lei Wang, and Xian-Sheng Hua.  Social friend recommendation based on multiple network correlation.  IEEE transactions on multimedia, 18(2):287–299, 2015.
  * Huang et al. [2019] Jin Huang, Zhaochun Ren, Wayne Xin Zhao, Gaole He, Ji-Rong Wen, and et al.  Taxonomy-aware multi-hop reasoning networks for sequential recommendation.  In WSDM, pages 573–581, 2019.
  * Jamali and Ester [2009] Mohsen Jamali and Martin Ester.  Trustwalker: a random walk model for combining trust-based and item-based recommendation.  In SIGKDD, pages 397–406, 2009.
  * Jiang et al. [2018] Zhengshen Jiang, Hongzhi Liu, Bin Fu, Zhonghai Wu, and Tao Zhang.  Recommendation in heterogeneous information networks based on generalized random walk model and bayesian personalized ranking.  In WSDM, pages 288–296, 2018.
  * Jin et al. [2020] Bowen Jin, Chen Gao, Xiangnan He, Depeng Jin, and Yong Li.  Multi-behavior recommendation with graph convolutional networks.  In SIGIR, pages 659–668, 2020.
  * Li and Chen [2013] Xin Li and Hsinchun Chen.  Recommendation as link prediction in bipartite graphs: A graph kernel-based machine learning approach.  Decision Support Systems, 54(2):880–890, 2013.
  * Ma et al. [2020] Chen Ma, Liheng Ma, Yingxue Zhang, Jianing Sun, Xue Liu, and Mark Coates.  Memory augmented graph neural networks for sequential recommendation.  In AAAI, pages 5045–5052, 2020.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In NIPS, pages 3111–3119, 2013.
  * Monti et al. [2017] Federico Monti, Michael M Bronstein, and Xavier Bresson.  Geometric matrix completion with recurrent multi-graph neural networks.  In NIPS, pages 3700–3710, 2017.
  * Nikolakopoulos and Karypis [2019] Athanasios N Nikolakopoulos and George Karypis.  Recwalk: Nearly uncoupled random walks for top-n recommendation.  In WSDM, pages 150–158, 2019.
  * Palumbo et al. [2017] Enrico Palumbo, Giuseppe Rizzo, and Raphaël Troncy.  Entity2rec: Learning user-item relatedness from knowledge graphs for top-n item recommendation.  In RecSys, pages 32–36, 2017.
  * Qiu et al. [2019] Ruihong Qiu, Jingjing Li, Zi Huang, and Hongzhi Yin.  Rethinking the item order in session-based recommendation with graph neural networks.  In CIKM, pages 579–588, 2019.
  * Quadrana et al. [2018] Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach.  Sequence-aware recommender systems.  ACM Computing Surveys (CSUR), 51(4):1–36, 2018.
  * Sheu and Li [2020] Heng-Shiou Sheu and Sheng Li.  Context-aware graph embedding for session-based news recommendation.  In RecSys, pages 657–662, 2020.
  * Shi et al. [2014] Yue Shi, Martha Larson, and Alan Hanjalic.  Collaborative filtering beyond the user-item matrix: A survey of the state of the art and future challenges.  ACM Computing Surveys (CSUR), 47(1):1–45, 2014.
  * Shi et al. [2018] Chuan Shi, Binbin Hu, Wayne Xin Zhao, and S Yu Philip.  Heterogeneous information network embedding for recommendation.  IEEE Transactions on Knowledge and Data Engineering, 31(2):357–370, 2018.
  * Song et al. [2019] Weiping Song, Zhiping Xiao, Yifan Wang, Laurent Charlin, Ming Zhang, and Jian Tang.  Session-based social recommendation via dynamic graph attention networks.  In WSDM, pages 555–563, 2019.
  * Sun et al. [2018] Zhu Sun, Jie Yang, Jie Zhang, Alessandro Bozzon, Long-Kai Huang, and Chi Xu.  Recurrent knowledge graph embedding for effective recommendation.  In RecSys, pages 297–305, 2018.
  * Verma et al. [2019] Janu Verma, Srishti Gupta, Debdoot Mukherjee, and et al.  Heterogeneous edge embedding for friend recommendation.  In ECIR, pages 172–179, 2019.
  * Wang et al. [2017] Shoujin Wang, Liang Hu, and Longbing Cao.  Perceiving the next choice with comprehensive transaction embeddings for online recommendation.  In ECML-PKDD, pages 285–302, 2017.
  * Wang et al. [2018a] Hongwei Wang, Fuzheng Zhang, Jialin Wang, Miao Zhao, Wenjie Li, and et al.  Ripplenet: Propagating user preferences on the knowledge graph for recommender systems.  In CIKM, pages 417–426, 2018.
  * Wang et al. [2018b] Xiang Wang, Xiangnan He, Fuli Feng, Liqiang Nie, and Tat-Seng Chua.  Tem: Tree-enhanced embedding model for explainable recommendation.  In WWW, pages 1543–1552, 2018.
  * Wang et al. [2019a] Haoyu Wang, Defu Lian, and Yong Ge.  Binarized collaborative filtering with distilling graph convolutional networks.  In IJCAI, pages 4802–4808, 2019.
  * Wang et al. [2019b] Hongwei Wang, Fuzheng Zhang, and et al.  Knowledge-aware graph neural networks with label smoothness regularization for recommender systems.  In SIGKDD, pages 968–977, 2019.
  * Wang et al. [2019c] Hongwei Wang, Fuzheng Zhang, Miao Zhao, and et al.  Multi-task feature learning for knowledge graph enhanced recommendation.  In WWW, pages 2000–2010, 2019.
  * Wang et al. [2019d] Shoujin Wang, Longbing Cao, Yan Wang, and et al.  A survey on session-based recommender systems.  arXiv preprint arXiv:1902.04864, 2019.
  * Wang et al. [2019e] Shoujin Wang, Liang Hu, Yan Wang, and et al.  Sequential recommender systems: challenges, progress and prospects.  In IJCAI, pages 6332–6338, 2019.
  * Wang et al. [2019f] Xiang Wang, Xiangnan He, Yixin Cao, and et al.  Kgat: Knowledge graph attention network for recommendation.  In SIGKDD, pages 950–958, 2019.
  * Wang et al. [2019g] Xiang Wang, Dingxian Wang, Canran Xu, Xiangnan He, and et al.  Explainable reasoning over knowledge graphs for recommendation.  In AAAI, pages 5329–5336, 2019.
  * Wang et al. [2019h] Zekai Wang, Hongzhi Liu, Yingpeng Du, and et al.  Unified embedding model over heterogeneous information network for personalized recommendation.  In IJCAI, pages 3813–3819, 2019.
  * Wang et al. [2020a] Nan Wang, Shoujin Wang, Yan Wang, and et al.  Modelling local and global dependencies for next-item recommendations.  In WISE, pages 285–300, 2020.
  * Wang et al. [2020b] Wen Wang, Wei Zhang, Shukai Liu, Qi Liu, Bo Zhang, Leyu Lin, and Hongyuan Zha.  Beyond clicks: Modeling multi-relational item graph for session-based target behavior prediction.  In The Web Conference, pages 3056–3062, 2020.
  * Wen et al. [2018] Yufei Wen, Lei Guo, Zhumin Chen, and Jun Ma.  Network embedding based recommendation method in social networks.  In WWW, pages 11–12, 2018.
  * Wu et al. [2019a] Le Wu, Peijie Sun, Yanjie Fu, and et al.  A neural influence diffusion model for social recommendation.  In SIGIR, pages 235–244, 2019.
  * Wu et al. [2019b] Shu Wu, Yuyuan Tang, Yanqiao Zhu, and et al.  Session-based recommendation with graph neural networks.  In AAAI, pages 346–353, 2019.
  * Wu et al. [2020a] Le Wu, Junwei Li, Peijie Sun, and et al.  Diffnet++: A neural influence and interest diffusion network for social recommendation.  IEEE Transactions on Knowledge and Data Engineering, 2020.  doi: 10.1109/TKDE.2020.3048414.
  * Wu et al. [2020b] Shiwen Wu, Wentao Zhang, Fei Sun, and Bin Cui.  Graph neural networks in recommender systems: A survey.  arXiv preprint arXiv:2011.02260, 2020.
  * Wu et al. [2021] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip.  A comprehensive survey on graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems, 32(1):4–24, 2021.
  * Xu et al. [2019] Chengfeng Xu, Pengpeng Zhao, and et al.  Graph contextualized self-attention network for session-based recommendation.  In IJCAI, pages 3940–3946, 2019.
  * Yang et al. [2018] Jheng-Hong Yang, Chih-Ming Chen, Chuan-Ju Wang, and Ming-Feng Tsai.  Hop-rec: high-order proximity for implicit recommendation.  In RecSys, pages 140–144, 2018.
  * Yin et al. [2010] Zhijun Yin, Manish Gupta, Tim Weninger, and Jiawei Han.  Linkrec: a unified framework for link recommendation with user attributes and graph structure.  In WWW, pages 1211–1212, 2010.
  * Ying et al. [2018] Rex Ying, Ruining He, and et al.  Graph convolutional neural networks for web-scale recommender systems.  In SIGKDD, pages 974–983, 2018.
  * Yu and Qin [2020] Wenhui Yu and Zheng Qin.  Graph convolutional network for recommendation with low-pass collaborative filters.  In ICML, pages 10936–10945, 2020.
  * Zha et al. [2001] Hongyuan Zha, Xiaofeng He, Chris Ding, Horst Simon, and Ming Gu.  Bipartite graph partitioning and data clustering.  In CIKM, pages 25–32, 2001.
  * Zhang and Chen [2018] Yongfeng Zhang and Xu Chen.  Explainable recommendation: A survey and new perspectives.  arXiv preprint arXiv:1804.11192, 2018.
  * Zhang and Chen [2020] Muhan Zhang and Yixin Chen.  Inductive matrix completion based on graph neural networks.  In ICLR, 2020.
  * Zhang et al. [2019a] Jiani Zhang, Xingjian Shi, Shenglin Zhao, and Irwin King.  Star-gcn: Stacked and reconstructed graph convolutional networks for recommender systems.  In IJCAI, 2019.
  * Zhang et al. [2019b] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay.  Deep learning based recommender system: A survey and new perspectives.  ACM Computing Surveys, 52(1):1–38, 2019.
  * Zhao et al. [2017] Huan Zhao, Quanming Yao, Jianda Li, and et al.  Meta-graph based recommendation fusion over heterogeneous information networks.  In SIGKDD, pages 635–644, 2017.
  * Zheng et al. [2018] Lei Zheng, Chun-Ta Lu, Fei Jiang, and et al.  Spectral collaborative filtering.  In RecSys, pages 311–319, 2018.
  * Zhu et al. [2019] Feng Zhu, Chaochao Chen, Yan Wang, and et al.  Dtcdr: A framework for dual-target cross-domain recommendation.  In CIKM, pages 1533–1542, 2019.
  * Zhu et al. [2021] Feng Zhu, Yan Wang, Chaochao Chen, and et al.  Cross-domain recommendation: Challenges, progress, and prospects.  arXiv preprint arXiv:2103.01696, 2021.