# Malakai: Music That Adapts to the Shape of Emotions

Zack Harris, Liam Atticus Clarke, Pietro Gagliano,
Dante Camarena, Manal Siddiqui
Transitional Forms
https://transforms.ai/
<EMAIL_ADDRESS>
Pablo S. Castro
Google Brain
<EMAIL_ADDRESS>

##  1 Background

This is a strange and exciting time for computer-generated music. The idea of
computer-generated musical composition has captured the public imagination, as
far back as Kurzweil’s demonstration of a pattern-based composer on live TV in
1965[1]. Since then, improvements in technology and composition tools have
created whole musical genres based around computer-generated compositions, and
have resulted in a vast library of algorithmic compositional techniques.
Furthermore, in the past few decades, interactive media such as games and
virtual reality have resulted in a demand for music that can adapt to dynamic
circumstances presented within the interactive medium. Finally, the advent of
ML music models such as Google Magenta’s MusicVAE[6] now allow us to extract
and replicate compositional features from otherwise complex datasets. These
models allow computational composers to parameterize abstract variables such
as style and mood.

By leveraging these models and combining them with procedural algorithms from
the last few decades, it is possible to create a dynamic song that composes
music in real-time to accompany interactive experiences [10]. Malakai is a
tool that helps users of varying skill levels create, listen to, remix and
share such dynamic songs. Using Malakai, a Composer can create a dynamic song
that can be interacted with by a Listener.

##  2 Project Malakai

Malakai is comprised of three main components, The Curves, The Graph and The
Blocks:

#### The Curves

A dynamic music player (Figure 3). As opposed to conventional music tracks,
Malakai tracks are dynamically composed based on certain input parameters. We
have developed a simple player interface that allows listeners to define the
value of these parameters over the course of the song by modifying a curve.
This interface takes inspiration from storytelling curves.[8, 9] The intended
user of this interface is the Listener. By default, tracks that we developed
internally are based on emotional values. These values correspond with the
three-variable model introduced by [4]. Listeners are able to modulate Energy,
Valence and Complexity (Matching Greenberg’s Arousal, Valence and Depth). For
example, higher valence produces happier or brighter sounding chord
progressions, while lower valence leads to progressions with a darker mood.

#### The Blocks

A Block (Figure 1) is a musical process, which represents a transformation or
generation of musical elements. Each Block has a set of inputs and outputs.
One example would be a block that improvises a melody according to an input
chord progression that was generated by another block. To ensure that the user
cannot make systematic mistakes (plugging incompatible blocks together),
blocks show coloured ports to represent the type of date expected. E.g.
Symbolic Music (Notes, rhythm, progressions, etc), Audio data (raw sound,
filters, etc.) or others. Blocks can vary between simple transformations to
implementations of Machine Learning models.

Figure 1: A MusicVAE Block

In order to effectively create dynamic songs, the Composer must become
familiar with the output of the different ML models available through Malakai.
Each AI model has a block that allows configuration of its options. We found
that this kind of low level control, combined with real-time interaction has
allowed our musicians to create music that accentuates each model’s
peculiarities. Furthermore, we created a series of procedural emotional music
generators for different musical elements, such as chord progressions, rhythms
and melodies. These generators can have their emotional parameters set
manually, or can receive the emotional curve values as inputs which change
over the duration of the song.

#### The Graph

An authoring tool for dynamic songs.(Figure 2) This tool allows a user to
combine Machine Learning and algorithmic music generators to author the
dynamic songs that can be heard in the player. This goal is achieved through a
node graph interface inspired by other no-code programming tools such as Node-
Based Shader editors [5, 7]. The intended user of this interface is the
Composer.

Using the graph, Composers can drag, drop, and connect blocks together to
build a dynamic song. In doing so, the Composer can provide their artistic
intent by selecting the instrumentation, progressions, melodies and structure.
Composers may provide these statically (through an instrument or midi file) or
by using the Algorithmic or ML generators.

By using this system, the Composer is encouraged to construct an abstraction
of their music in such a way that can be interacted with by the listener
through the Curve interface. In a way, this form of composition is uniquely
distinct from traditional composition, as it requires collaboration with both
the user and the AI models (in which performance varies due to stochasticity).
As a result, the composer is invited to merely set the stage for the
performance, allowing the user and AI models to participate in the
composition, a process that we label Three-Way Authorship [3].

##  3 Example

In our example below (Figure 2), the melody was provided with MidiMe [2],
which trained a MusicVAE model within a given latent space. This provided the
leading melody of the performance. That output was then also mapped to a
separate MusicVAE that provided the countermelody. That countermelody made use
of MusicVAE’s underlying chords feature to match the chords that were created
by an algorithmic progression generator. That generator was in turn fed by the
user input of the curve parameters. This performance was then able to vary in
tension by modulating the counter-melody and chord progressions by using the
valence and complexity parameters. We’ve also created a preliminary system
which sets the foundation for allowing dynamic tracks to be embedded in games,
with the parameters tied to the state of the game. This will result in an
atmospheric track that mirrors the user’s emotional state within the context
of the game. This kind of interactive composition will be more important as
interactive media continues to increase in popularity.

Figure 2: A simple dynamic song

##  Appendix A Appendix

###  A.1 Ethical Considerations

There are foundational ethical questions related to machine learning and
creativity in music. For example, it is not clear if machine-generated works
could, should and will be protected by Intellectual Property, and, if they do,
who should be the owner of the related rights. In addition, other problems
related to copyright should be considered in our opinion, such as if training
over protected work is admitted and what happens if the results are similar to
some of the training data. Another ongoing important debate is about
authorship and the human role in creative fields in the era of AI.

###  A.2 Technical description

Malakai is powered by Tensorflow.js and renders audio using the WebAudio API.
It can also be used as a standalone composition library. Using the Malakai
API, it can be extended to support new blocks. While the focus on the
application is the use of AI blocks, Procedural blocks allow for more
controllable transformations of data and the opportunity for creative coders
to create new musical algorithmic generators as new blocks in our modular
system. As such, we leverage the existing high level audio library Tone.js and
music theory library tonal.js, which provide an easy to use way for users to
write compositional elements.

Malakai is currently in a functional state, with a few remaining bugs. If
selected, Malakai will be made available to NeurIPS audience as a browser-run
application. If chosen for a presentation, the presentation will musical
samples.

###  A.3 Acknowledgements

Thanks to Kory Mathewson, Liam Clarke (unrelated to author), Alexander
Bakogeorge, Anna Huang, Erin Ray, and the rest of the Transforms.ai team for
their help and/or advice in conducting this research.

###  A.4 The curves interface

Figure 3: The curves interface

## References

  * American Academy of Achievement [2019] American Academy of Achievement.  Ray kurzweil, May 2019.  URL https://achievement.org/achiever/ray-kurzwell/.

  * Dinculescu et al. [2019] M. Dinculescu, J. Engel, and A. Roberts, editors.  _MidiMe: Personalizing a MusicVAE model with user data_ , 2019.
  * Gagliano et al. [2021] P. Gagliano, D. Camarena, C. Blustein, and D. Oppenheim.  Agence, a dynamic film about (and with) artificial intelligence.  In _ACM SIGGRAPH 2021 Immersive Pavilion_ , SIGGRAPH ’21, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450383684.  doi: 10.1145/3450615.3464538.  URL https://doi.org/10.1145/3450615.3464538.
  * Greenberg et al. [2016] D. Greenberg, M. Kosinski, D. Stillwell, B. Monteiro, D. Levitin, and P. Rentfrow.  The song is you: Preferences for musical attribute dimensions reflect personality.  _Social Psychological and Personality Science_ , 7, 05 2016.  doi: 10.1177/1948550616641473.
  * Nintendo [2021] Nintendo.  Game builder garage.  [Switch Game], 2021.
  * Roberts et al. [2018] A. Roberts, J. H. Engel, C. Raffel, C. Hawthorne, and D. Eck.  A hierarchical latent vector model for learning long-term structure in music.  In J. G. Dy and A. Krause, editors, _ICML_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4361–4370. PMLR, 2018\.  URL http://dblp.uni-trier.de/db/conf/icml/icml2018.html#RobertsERHE18.
  * Unity Technologies [2019] Unity Technologies.  Shader Graph, 2019.  URL https://unity.com/shader-graph.
  * Vonnegut [1987] K. Vonnegut.  _Palm Sunday an autobiographical collage_.  Grafton Books, 1987.
  * Vonnegut [2010] K. Vonnegut.  Shape of stories, Oct 2010.  URL https://www.youtube.com/watch?v=oP3c1h8v2ZQ.
  * Zulić et al. [2019] H. Zulić et al.  How ai can change/improve/influence music composition, performance and education: three case studies.  _INSAM Journal of Contemporary Music, Art and Technology_ , 1(2):100–114, 2019.


∎

11institutetext:  Yuan Jin 22institutetext: Faculty of Information Technology,
Monash University, Clayton, Victoria, 3168, Australia
22email<EMAIL_ADDRESS>33institutetext: He Zhao 44institutetext: Faculty
of Information Technology, Monash University, Clayton, Victoria, 3168,
Australia
44email<EMAIL_ADDRESS>55institutetext: Ming Liu 66institutetext:
School of Information Technology, Deakin University, Burwood, Victoria,
Australia 3125.
66email<EMAIL_ADDRESS>77institutetext: Ye Zhu 88institutetext: School
of Information Technology, Deakin University, Burwood, Victoria, Australia
3125.
88email<EMAIL_ADDRESS>99institutetext: Lan Du 1010institutetext: Faculty of
Information Technology, Monash University, Clayton, Victoria, 3168, Australia
1010email<EMAIL_ADDRESS>1111institutetext: Longxiang Gao
1212institutetext: School of Information Technology, Deakin University,
Burwood, Victoria, Australia 3125.
1212email<EMAIL_ADDRESS>1313institutetext: Wray Buntine
1414institutetext: Faculty of Information Technology, Monash University,
Clayton, Victoria, 3168, Australia
1414email<EMAIL_ADDRESS>
# Leveraging Cross Feedback of User and Item Embeddings with Attention for
Variational Autoencoder based Collaborative Filtering

Yuan Jin  He Zhao  Ming Liu  Ye Zhu  Lan Du  Longxiang Gao  Wray Buntine

(Received: date / Accepted: date)

###### Abstract

Matrix factorization (MF) has been widely applied to collaborative filtering
in recommendation systems. Its Bayesian variants can derive posterior
distributions of user and item embeddings, and are more robust to sparse
ratings. However, the Bayesian methods are restricted by their update rules
for the posterior parameters due to the conjugacy of the priors and the
likelihood. Variational autoencoders (VAE) can address this issue by capturing
complex mappings between the posterior parameters and the data. However,
current research on VAEs for collaborative filtering only considers the
mappings based on the explicit data information while the implicit embedding
information is overlooked.

In this paper, we first derive evidence lower bounds (ELBO) for Bayesian MF
models from two viewpoints: user-oriented and item-oriented. Based on the
ELBOs, we propose a VAE-based Bayesian MF framework. It leverages not only the
data but also the embedding information to approximate the user-item joint
distribution. As suggested by the ELBOs, the approximation is iterative with
cross feedback of user and item embeddings into each other’s encoders. More
specifically, user embeddings sampled at the previous iteration are fed to the
item-side encoders to estimate the posterior parameters for the item
embeddings at the current iteration, and vice versa. The estimation also
attends to the cross-fed embeddings to further exploit useful information. The
decoder then reconstructs the data via the matrix factorization over the
currently re-sampled user and item embeddings. We show the effectiveness of
our framework in lowering rating prediction errors, improving relevant item
ranking and handling rating sparsity across five real-world datasets. We also
perform ablation studies and convergence analysis to illustrate the importance
of the cross-feedback and attention components of our framework in the
performance and the model convergence.

###### Keywords:

Recommendation Collaborative filtering Matrix factorization Variational
autoencoders

##  1 Introduction

Collaborative filtering (CF) approaches (Koren and Bell, 2015) have been the
backbone of modern recommendation systems. Their goal is to suggest new items
to a particular user that are of interest to him/her based on ratings from
other like-minded users or various types of side information (e.g. rating
timestamps and locations, user demographics, item content). Among them, matrix
factorization (MF) has been the most widely applied approach (Koren et al,
2009). It aims to reconstruct observed ratings from two sets of multi-
dimensional embeddings: one for users and the other for items. The user and
item embeddings are usually estimated by minimizing the regularized squared
(reconstruction) errors via stochastic gradient descent. During the
estimation, the embeddings of like-minded users tend to be mapped into close
proximity. As a result, a user tend to be recommended with new items that are
liked by those with similar embeddings.

From a probabilistic standpoint, the regularization in the above minimization
problem is equivalent to imposing Normal priors over the embeddings. The
squared errors between the observed and reconstructed ratings are equivalent
to Normal likelihood. Thus, the entire problem can also be viewed as
maximizing a posteriori over the embeddings (Mnih and Salakhutdinov, 2008). A
further treatment is to make fully Bayesian modelling and inference of the
embeddings in matrix factorization. This is referred to as the Bayesian matrix
factorization (BMF) for collaborative filtering.

In this case, the BMF models impose appropriate prior distributions over the
user and item embeddings (Salakhutdinov and Mnih, 2008; Lakshminarayanan et
al, 2011; Harvey et al, 2011). These embeddings then construct a likelihood
term over the rating data. Training BMF models amounts to infer the joint
posterior distribution of the embeddings analytically or by approximation. In
most cases, distributions involving multiple variables cannot be derived
analytically, and thus approximation inference techniques are used instead.

BMF models adopt two types of approximation inference frameworks: Markov Chain
Monte Carlo (MCMC) methods and Variational Inference. Both of them rely on the
conjugacy of the priors and the likelihood over the embeddings. The BMF models
have mostly exploited Normal-Normal and Poisson-Gamma conjugacy for their
likelihood and priors. Despite its algebraic convenience, the conjugate
updates on the posterior parameters of each embedding are independent and thus
fail to utilize the information from each other. This limits the performance
of the BMF models. In this case, we want the data information for different
embeddings to be shared to refine the inference of their posterior
distributions.

To enable the sharing of the data information, it is common to model the
posterior parameters as regression over the data. The regression weights are
learned to map similar data patterns into values in close proximity in the
latent space. Naturally, the mapping is non-linear. This motivates us to use
artificial neural networks (NN), which can fit arbitrarily complex mapping
functions, to predict the posterior parameters for the embeddings. In
particular, variational auto-encoders (VAEs) provide the foundation to achieve
all of the above. It connects variational inference of posterior distributions
with neural networks.

Motivated by our derivation of two ELBOs for the variational inference of BMF
models in collaborative filtering, we propose a novel BMF framework based on
variational autoencoders. It conducts mean-field variational inference for
user and item embeddings by predicting their posterior parameters using multi-
layer perceptron (MLP) encoders. These encoders take in not only the data but
also the information from the embeddings. Such an input arrangement is
evidenced by the inter-dependence of variables in the derived ELBOs; that is a
user’s embeddings dependent on both his/her ratings and the embeddings of the
rated items, and vice versa.

Our framework leverages both the data information and the embedding
information in an iterative manner. In each iteration, user and item
embeddings are sampled from their respective encoders. The user-side encoders
take in the rating inputs and latent inputs popluated by the item embedding
samples which are fed back from the last iteration, and vice versa. In
addition, we enable the encoders to attend to certain parts of the latent
inputs to further exploit useful information from the embeddings. Finally, the
decoder reconstructs the ratings using the matrix factorization over the
currently (re-)sampled user and item embeddings.

The contributions made by our paper are listed as follows:

  * •

We derive user-oriented and item-oriented ELBOs for the varitional inference
of BMF models in collaborative filtering.

  * •

We propose a VAE-based BMF framework that unifies the modelling of both
derived ELBOs. It achieves overall lower reconstruction errors and higher
relevant item ranking measures compared to six state-of-the-art NN-based CF
methods across five real-world rating datasets.

  * •

When the datasets are subsampled to emulate the sparsity that might occur
across both users and items, we found that our framework which leverages the
implicit embedding information is overall more robust than the baseline
models.

  * •

The cross-feedback and attention components clearly improve the performance
and the convergence of our framework, as revealed by an ablation study and a
model convergence analysis.

##  2 Related Work

Matrix factorization (MF) (Koren and Bell, 2015) is the most widely applied
approach in collaborative filtering (CF). It assumes that a matrix can be well
approximated by the multiplication of two low-rank matrices. In the context of
CF, the target matrix stores the user-item ratings, while the two low-rank
matrices comprise the latent representations, called the embeddings, of users
and items. Based on the MF modelling, regularized squared errors between the
observed entries (stored in the rating matrix) and the approximated ones are
minimized with respect to the user and item embeddings. This minimization is
usually conducted via stochastic gradient descent (SGD) (Koren et al, 2009;
Koren and Bell, 2015). Among various MF models, the probabilistic matrix
factorization (PMF) (Mnih and Salakhutdinov, 2008) proposed to assign Normal
priors to the user and item embeddings. These embeddings further form a Normal
likelihood over the ratings. Each rating is modelled to follow a Normal
distribution centered on a user-item embedding dot product. The above
minimization problem is now formulated as maximum a posteriori (MAP)
estimation of the embeddings via SGD. However, both of these cases yield point
estimates for the embeddings.

###  2.1 Bayesian Matrix Factorization

Extending from the PMF model are the Bayesian matrix factorization (BMF)
models (Salakhutdinov and Mnih, 2008; Gopalan et al, 2015a, b). They impose
various priors over the user and item embeddings. These priors are usually
conjugate to the likelihood over the rating data. In (Salakhutdinov and Mnih,
2008), a fully Bayesian treatment was applied in which Normal-Wishart priors
were imposed over the embeddings. Due to the conjugacy of such priors and a
Normal likelihood, the conditional posterior distributions of the embeddings
can be sampled in a closed form via the Gibbs sampling.

Lakshminarayanan et al (2011) replaced the Normal priors with some heavy-
tailed distributions such as the Student-t distributions to increase the
robustness of BMF to outlier or atypical ratings. Gopalan et al (2015a, b)
proposed to impose Gamma distributions as the priors over the embeddings. In
this case, the likelihood is assumed to follow a Poisson distribution which
exploits the Poisson-Gamma conjugacy for the convenience of following
inference. All these work leverages the variational inference (Blei et al,
2017) to estimate the posterior distribution parameters for the embeddings.

###  2.2 Variational Inference

According to Blei et al (2017), the goal of variational inference is to
approximate some intractable conditional distribution of latent variables
$\boldsymbol{Z}$ given observations $\boldsymbol{X}$:
$p(\boldsymbol{Z}|\boldsymbol{X})$. The approximation amounts to solving an
optimization problem over a family of tractable distribution
$q(\boldsymbol{Z})$. The objective function in this case is the Kullback-
Leibler (KL) divergence between the true intractable distribution
$p(\boldsymbol{Z}|\boldsymbol{X})$ and the tractable variational one
$q(\boldsymbol{Z})$:
$\text{KL}\big{[}q(\boldsymbol{Z})||p(\boldsymbol{Z}|\boldsymbol{X})\big{]}$.
It is minimized with respect to the distribution parameters of
$q(\boldsymbol{Z})$, which turns out to be equivalent to maximizing the
following evidence lower bound (ELBO) Q:

$\begin{split}\text{Q}&=\mathbbm{E}_{q}\big{[}\log{p(\boldsymbol{X},\boldsymbol{Z})}\big{]}-\mathbbm{E}_{q}\big{[}\log{q(\boldsymbol{Z})}\big{]}\hskip
10.0pt=\mathbbm{E}_{q}\big{[}\log{p(\boldsymbol{X}|\boldsymbol{Z})}\big{]}-\text{KL}\big{[}q(\boldsymbol{Z})||p(\boldsymbol{Z})\big{]}\end{split}$
(1)

In Eq. 1,
$\mathbbm{E}_{q}\big{[}\log{p(\boldsymbol{X},\boldsymbol{Z})}\big{]}$ and
$\mathbbm{E}_{q}\big{[}\log{p(\boldsymbol{x}|\boldsymbol{Z})}\big{]}$ are
respectively the expected joint distribution and data log-likelihood with
respect to (the parameters of) $q(\boldsymbol{Z})$ and
$\text{KL}\big{[}q(\boldsymbol{Z})||p(\boldsymbol{Z})\big{]}$ is the KL
divergence between $q(\boldsymbol{Z})$ and the prior distribution
$p(\boldsymbol{Z})$.

The variational distribution $q(\boldsymbol{Z})$ needs to be easy to optimize
over. One such family of distributions is the mean-field distributions which
are the set of fully factored independent distributions from
$q(\boldsymbol{Z})$: $q(\boldsymbol{Z})=\prod_{n=1}^{N}q(\boldsymbol{z}_{n})$
where $N$ is the size of the data. In this case, the ELBO function Q in Eq. 1
can be simplified and expanded as follows:

$\text{Q}=\sum_{n=1}^{N}\mathbbm{E}_{q}\big{[}\log{p(\boldsymbol{x}_{n}|\boldsymbol{z}_{n})}\big{]}-\sum_{n=1}^{N}\text{KL}\big{[}q(\boldsymbol{z}_{n})||p(\boldsymbol{z}_{n})\big{]}$
(2)

In collaborative filtering, it is conventional to set each
$q(\boldsymbol{z}_{n})$ and $p(z_{n})$ to be Normal distributions:
$q(\boldsymbol{z}_{n})=\mathcal{N}(\boldsymbol{\mu}_{n},\text{diag}(\boldsymbol{\sigma}^{2}_{n}))$
and
$p(\boldsymbol{z}_{1})=p(\boldsymbol{z}_{2})=...=p(\boldsymbol{z}_{n})=\mathcal{N}(\boldsymbol{\mu},\text{diag}(\boldsymbol{\sigma}^{2}))$.
Maximizing the ELBO with respect to each parameter
$\\{\boldsymbol{\mu}_{n},\text{diag}(\boldsymbol{\sigma}^{2}_{n})\\}_{n=1,...,N}$
via the coordinate ascent (Blei et al, 2017) yields iterative posterior
updates on the parameters. This means that updates on $\boldsymbol{\mu}_{n}$
and $\text{diag}(\boldsymbol{\sigma}^{2}_{n})$ take in the data information
either directly from $\boldsymbol{x}_{n}$ or indirectly from each other.

One major problem of the update rules based on the variational inference is
their lack of flexibility to capture the full complexity of the true posterior
distribution $p(\boldsymbol{Z}|\boldsymbol{X})$ (Zhang et al, 2018). This is
largely due to the conjugate pairs (e.g. Normal-Normal, Poisson-Gamma,
Multinomial-Dirichlet) imposed by the variational models on the likelihood and
priors. Furthermore, the update for the variable $\boldsymbol{z}_{n}$ fails to
leverage the update information for the other variables
$\\{\boldsymbol{z}_{n^{\prime}}\\}_{1\leq n^{\prime}\leq N,n^{\prime}\neq n}$.

###  2.3 Variational Autoencoder Based Bayesian Matrix Factorization

Variational autoencoders (VAE) (Kingma and Welling, 2014) can solve both of
the above problems. They are extended from autoencoders (AE) (Baldi, 2012)
which are unsupervised generative neural networks. They aim to reconstruct the
input data and meanwhile generate multi-dimensional embeddings of the data.
VAEs are known to be able to approximate intractable posterior distributions
of the embeddings. This is achieved via computing the variational posterior
parameters of the embeddings using particular neural networks (e.g. MLPs).
These neural networks are referred to as the encoders of the VAEs. More
specifically, the variational posterior distribution $q(\boldsymbol{z}_{n})$
of the multi-dimensional embedding $\boldsymbol{z}_{n}$ are:

$q(\boldsymbol{z}_{n})=\mathcal{N}\big{(}\boldsymbol{\mu}_{n},\text{diag}(\boldsymbol{\sigma}^{2}_{n})\big{)}=\mathcal{N}\big{(}\boldsymbol{F}(\boldsymbol{x}_{n}),\boldsymbol{F}^{\prime}(\boldsymbol{x}_{n})\big{)}~{}$
(3)

In Eq. 3, $\boldsymbol{F}$ and $\boldsymbol{F}^{\prime}$ are the non-linear
functions modelled by the encoder neural networks. In this case, the neural
networks take in each data point $\boldsymbol{x}_{n}$ and output the
corresponding mean vector $\boldsymbol{\mu}_{n}$ and the variance vector
$\text{diag}(\boldsymbol{\sigma}^{2}_{n})$.

VAEs can capture arbitrarily complex mapping relationships between the
posterior parameters and the data, which failed to be captured by the
conjugate updates of the BMF models. VAEs also allow the sharing of the
posterior update information through the learned encoder weights while the
update information are independent in the BMF models.

In recent years, research has started to emerge which integrates VAEs into the
BMF in collaborative filtering. Li and She (2017) proposed to leverage VAEs
for estimating the posterior means and variances of item embeddings from the
item content. Meanwhile, the generated item embeddings are involved in the
traditional BMF modelling. Liang et al (2018) only estimated the posterior
means and variances of the user embeddings using the VAEs from the implicit
data (e.g. click data). Item-specific weights of a MLP network are learned to
map the user embeddings non-linearly into the multinomial predictions for the
implicit data. Its extension (Shenbin et al, 2020) leverages users’ embedding
estimates from the previous iteration to construct a composite prior (with a
standard Normal prior) for the user embeddings in the current iteration. This
work also proposes to use a user-specific rescaling factor to control the KL-
divergence for each user. However, to our best knowledge, there has not been
any work which estimates the posterior parameters for both the user and the
item embeddings using the VAEs.

###  2.4 Other Neural Network Based Matrix Factorization Models

Apart from the VAEs, there have been many other types of neural networks (NN)
applied to the matrix factorization for collaborative filtering. Liang et al
(2018) also proposed a denoising autoencoder (DAE) based user-oriented MF
model. It randomly adds noise to the input data and then learns user
embeddings to recover the original data. This model is very similar to another
major DAE based user-oriented model proposed by Wu et al (2016). The main
difference is that the former model adopts the multinomial log-likelihood loss
which is more robust than the logistic loss employed by the latter. Earlier
work has mostly used vanilla AEs to replace the MF approaches. One of the
pioneering work (Sedhain et al, 2015) proposed two variants of the vanilla AEs
for separately encoding users and items from the user-item and the item-user
rating matrices respectively. Their decoders then reconstruct the above rating
matrices from the resulting user and item embeddings respectively.

Unlike AE based approaches which take in the rating data, there are other NN
based approaches that take in the one-hot encodings of the user and item IDs.
He et al (2017) proposed to capture non-linear interactions between the user
and the item embeddings (mapped from their encoded IDs) via a MLP network. The
two embeddings are concatenated before passed through the MLP. Different from
the concatenation, He et al (2018) proposed to perform outer product between
the user and the item embeddings to better capture their interactive patterns.
The resulting latent interaction maps are passed through a convolutional
neural network (CNN) which learns high-order correlations among the embedding
dimensions.

Wang et al (2019) proposed a graph neural network which leverages the message
passing and aggregation of the node embeddings of items rated by a user for
estimating his/her node embedding, and vice versa. This message-passing
structure corresponds to the cross-feedback component of our framework.
Despite this, our framework is fundamentally different from theirs as a fully
probabilistic approach which learns a joint distribution with VAE based on a
principled evidence lower bound.

##  3 Proposed Framework

In this paper, we use the symbols $I$ and $J$ to respectively denote the
number of users and items involved in collaborative filtering. We denote the
embedding of the $i$-th user as $\boldsymbol{u}_{i}$ and that of the $j$-th
item as $\boldsymbol{v}_{j}$ where $i=1,...,I$ and $j=1,...,J$. The number of
dimensions of $\boldsymbol{u}_{i}$ and $\boldsymbol{v}_{j}$ are $K$. The user
and item embedding matrices are expressed as
$\boldsymbol{U}\in\mathbbm{R}^{I\times K}$ and
$\boldsymbol{V}\in\mathbbm{R}^{J\times K}$ respectively. The ratings of all
the users on all the items are formulated as a matrix
$\boldsymbol{X}\in\mathbbm{R}^{I\times J}$. Its ($i$,$j$)-th entry stores the
rating $x_{ij}$ given by the $i$-th user to the $j$-th item. If the rating is
missing, we set $x_{ij}=0$ for convenience of mathematical expression.

###  3.1 Deriving ELBOs for BMF Models

Our framework is motivated by the derivation of two ELBOs for the BMF models
in collaborative filtering which are user-oriented and item-oriented. More
specifically, in the context of collaborative filtering, the ELBO specified in
Eq. 1 is modified as follows:

$\begin{split}Q=\mathbbm{E}_{q}\big{[}\log{p(\boldsymbol{X},\boldsymbol{U},\boldsymbol{V})}\big{]}-\mathbbm{E}_{q}\big{[}\log{q(\boldsymbol{U},\boldsymbol{V})}\big{]}~{}\end{split}$
(4)

From a user-oriented viewpoint, Eq. 4 can be further expanded as follows:

$\begin{split}Q_{\boldsymbol{U}}=\sum\nolimits^{I}_{i}\Big{(}\mathbbm{E}[\log
p(\boldsymbol{u}_{i}|\boldsymbol{V}_{i},\boldsymbol{X}_{i})]-\mathbbm{E}_{i}[\log
q(\boldsymbol{u}_{i})]\Big{)}+\text{C}_{1}~{}\end{split}$ (5)

Eq. 5 is derived under the mean-field assumption. The symbols
$\boldsymbol{V}_{i}$ and $\boldsymbol{X}_{i}$ correspond to the embeddings of
items rated by the $i$-th user and their received ratings. For
$\boldsymbol{V}_{i}$, they are typically obtained from the one-hot encoding
vector of the items’ IDs via an embedding layer (Sedhain et al, 2015; He et
al, 2017) and are fixed when Eq. 5 is maximized with respect to
$q(\boldsymbol{u}_{i})$.
$p(\boldsymbol{u}_{i}|\boldsymbol{V}_{i},\boldsymbol{X}_{i})$ is the true
posterior distribution to be approximated by the variational distribution
$q(\boldsymbol{u}_{i})$; that is to construct $q(\boldsymbol{u}_{i})$ with the
same form of $p(\boldsymbol{u}_{i}|\boldsymbol{V}_{i},\boldsymbol{X}_{i})$ and
make it close enough with respect to its parameters. Terms that do not depend
on $q(\boldsymbol{u}_{i})$ are treated as the constant $\text{C}_{1}$.
Likewise, Eq. 4 can be expanded from an item-oriented viewpoint as well:

$\begin{split}Q_{\boldsymbol{V}}=\sum\nolimits^{J}_{j}\Big{(}\mathbbm{E}[\log
p(\boldsymbol{v}_{j}|\boldsymbol{U}_{j},\boldsymbol{X}_{j})]-\mathbbm{E}_{j}[\log
q(\boldsymbol{v}_{j})]\Big{)}+\text{C}_{2}~{}\end{split}$ (6)

Eq. 6 focuses on approximation with respect to $q(\boldsymbol{v}_{j})$. All
its terms have the same meanings as their counterparts in Eq. 5. For
$\boldsymbol{U}_{j}$, they are usually derived from the users’ IDs via another
embedding layer and are fixed when Eq. 6 is maximized with respect to
$q(\boldsymbol{v}_{j})$. The true posterior distribution
$p(\boldsymbol{v}_{j}|\boldsymbol{U}_{i},\boldsymbol{X}_{j})$, in this case,
is approximated by the variational distribution $q(\boldsymbol{v}_{j})$.

From Eq. 5 and Eq. 6, we see that neither user-oriented nor item-oriented
viewpoints capture the full inter-dependency among users, items and ratings.
To do so, we consider a unified viewpoint to expand Eq. 4; that is to
construct $q(\boldsymbol{u}_{i})$ and $q(\boldsymbol{v}_{j})$ to depend on
both the data and each other for KL-divergence:

$\begin{split}Q=\mathbbm{E}\big{[}\log{p(\boldsymbol{X}|\boldsymbol{U},\boldsymbol{V})}\big{]}&-\beta_{\boldsymbol{u}}\sum\nolimits^{I}_{i}\text{KL}\big{[}q(\boldsymbol{u}_{i}|\boldsymbol{V}_{i},\boldsymbol{X}_{i})||p(\boldsymbol{u}_{i})\big{]}\\\
&-\beta_{\boldsymbol{v}}\sum\nolimits^{J}_{j}\text{KL}\big{[}q(\boldsymbol{v}_{j}|\boldsymbol{U}_{j},\boldsymbol{X}_{j})||p(\boldsymbol{v}_{j})\big{]}~{}\end{split}$
(7)

Eq. 7 can be interpreted as an autoencoder. It “encodes” data
$\boldsymbol{X}_{i}$ and variables $\boldsymbol{V}_{i}$ into
$\boldsymbol{u}_{i}$, and meanwhile, data $\boldsymbol{X}_{j}$ and variables
$\boldsymbol{U}_{j}$ into $\boldsymbol{v}_{i}$. Then, it “decodes” the user
and item variables to reconstruct the data. $\beta_{\boldsymbol{u}}$ and
$\beta_{\boldsymbol{v}}$ denote the hyper-parameters that respectively control
the regularization effect of the user-specific and item-specific KL terms
according to (Liang et al, 2018).

###  3.2 Framework Architecture

To optimize the ELBO specified by Eq. 7, we propose an VAE-based BMF framework
for collaborative filtering. This framework is characterized by a cross-
feedback connectivity component which captures the mutual dependency between
user and item variables in the equation.

Figure 1: The architecture of our framework.

Figure 1 shows the architecture of our framework. Note that for simplicity,
the posterior means and variances are shown to be computed by the same set of
feed-forward networks. In the experiments, we implement separate MLP feed-
forward networks to compute these two sets of parameters.

Our framework inherits the encoder-decoder structure from the VAEs. In terms
of the decoder, it reconstructs the rating matrix $\boldsymbol{X}$ as the
multiplication of the user and item embedding matrices $\boldsymbol{UV}^{T}$.
In this paper, we focus on reconstructing the observed data during the
training phase. Therefore, the log-likelihood term
$p(\boldsymbol{X}|\boldsymbol{U},\boldsymbol{V})$ in Eq. 7 is calculated
solely based on each observed entry of $\boldsymbol{X}$ (i.e. $x_{ij}\neq 0$).
As for the encoder part, there are user-side and item-side encoders which
respectively model the variational distributions
$q(\boldsymbol{u}_{i}|\boldsymbol{V}_{i},\boldsymbol{X}_{i})$ and
$q(\boldsymbol{v}_{j}|\boldsymbol{U}_{j},\boldsymbol{X}_{j})$ to generate the
matrices $\boldsymbol{U}$ and $\boldsymbol{V}$. Both encoders possess two
types of inputs: the observed rating input and the latent embedding input. The
user-side observed input (layer) takes in the matrix $\boldsymbol{X}$, while
the item-side input receives its transpose $\boldsymbol{X}^{T}$.

As for the latent input (layer) of the user-side encoder, it is constructed
based on the item embeddings. More specifically, it is constructed as a matrix
$\boldsymbol{Z}_{\boldsymbol{U}}\in\mathbbm{R}^{I\times(JK)}$. Its $i$-th row
is a concatenation of the embeddings of all the items after the embeddings of
those unrated by the $i$-th user are masked by zeros. As an example, suppose
the $i$-th user has rated the first of three items and the embedding dimension
$K=2$. Then, the input row corresponding to this user stores a vector
$\boldsymbol{z}_{\boldsymbol{u}_{i}}=[v_{11},v_{12},0,0,0,0]$. Likewise, the
latent input layer of the item-side encoder takes in a matrix
$\boldsymbol{Z}_{\boldsymbol{V}}\in\mathbbm{R}^{J\times(IK)}$ which is
constructed in the same way as $\boldsymbol{Z}_{\boldsymbol{U}}$.

Both the observed and latent inputs at each side are passed through and
transformed by their own MLP networks to compute the corresponding posterior
parameters. To make the framework description succinct, we only describe the
MLP networks of the user-side encoder that compute the posterior means for the
user embeddings. Initially, the rating input of the user-side encoder is
transformed by an MLP network as follows:

$\displaystyle\begin{split}\boldsymbol{H}_{1}=\text{ReLU}(\boldsymbol{X}\boldsymbol{W}_{0}+\boldsymbol{{1}}\boldsymbol{\alpha}_{0})\\\
\boldsymbol{H}_{2}=\text{ReLU}(\boldsymbol{H}_{1}\boldsymbol{W}_{1}+\boldsymbol{{1}}\boldsymbol{\alpha}_{1})\\\
\ldots\hskip 50.0pt\\\
\boldsymbol{H}_{L^{\prime}+1}=\text{ReLU}(\boldsymbol{H}_{L^{\prime}}\boldsymbol{W}_{L^{\prime}}+\boldsymbol{{1}}\boldsymbol{\alpha}_{L^{\prime}})\\\
\end{split}~{}$ (8)

In Eq. 8, $\boldsymbol{W}_{0}\in\mathbbm{R}^{J\times M^{(1)}}$ and
$\boldsymbol{\alpha}_{0}\in\mathbbm{R}^{1\times M^{(1)}}$ are respectively the
weight matrix and the bias vector specific to the observed input layer.
$M^{(1)}$ is the number of neurons for the first hidden layer subsequent to
the observed inputs. The term $\boldsymbol{{1}}$ denotes a column vector of
size $I\times 1$ with its entries being all ones. The matrix output of this
input layer is then transformed by the activation function ReLU. The result
$\boldsymbol{H}_{1}\in\mathbbm{R}^{I\times M^{(1)}}$ serves as the input to
the first hidden layer. There are in total $L^{\prime}$ hidden layers in the
user-side encoder for the observed inputs. The output of the $L^{\prime}$-th
hidden layer, $\boldsymbol{H}_{L^{\prime}+1}\in\mathbbm{R}^{I\times
M^{(L^{\prime}+1)}}$, is the intermediate user embedding matrices derived from
the rating input $\boldsymbol{X}$. For convenience, we use
$K^{\prime}=M^{(L^{\prime}+1)}$ to denote the dimension of each intermediate
user embedding.

Similar to the observed input layer, the latent embedding input
$\boldsymbol{Z}_{\boldsymbol{U}}$ is passed through its dedicated MLP network.
Correspondingly, we denote the weight matrices and the bias vectors of each
hidden layer of this MLP network as
$\\{\boldsymbol{\Phi}_{l^{\prime}},\boldsymbol{\beta}_{l^{\prime}}\\}_{l^{\prime}=1,...,L^{\prime}}$.
Note that the weight matrix of the latent input layer
$\boldsymbol{\Phi}_{0}\in\mathbbm{R}^{JK\times M^{(1)}}$ is much larger than
its counterpart $\boldsymbol{W}_{0}$ for the observed input layer. Other than
this, we let the two MLP encoders have the same number of hidden layers
$L^{\prime}$, the same number of hidden neurons $M^{(l^{\prime})}$ at the
$l^{\prime}$-th layer, the same setting of the activation functions and the
same dimension $K^{\prime}$ for the intermediate embeddings111In the
experiment, we did not find notable improvements in the framework performance
by making the two MLP networks have different numbers for these parameters..
Finally, we denote the output of the $L^{\prime}$-th hidden layer over the
latent inputs as $\boldsymbol{G}_{L^{\prime}+1}$.

After generating the intermediate embedding matrices
$\boldsymbol{H}_{L^{\prime}+1}$ and $\boldsymbol{G}_{L^{\prime}+1}$, we
concatenate them in a column-wise manner. This creates the intermediate input
(layer)
$\boldsymbol{S}=[\boldsymbol{H}_{L^{\prime}+1};~{}\boldsymbol{G}_{L^{\prime}+1}]\in\mathbbm{R}^{I\times
2K^{\prime}}$ to the final MLP network. It computes the posterior means with
another set of weight matrics and bias vectors:
$\\{\boldsymbol{\Psi}_{l},\boldsymbol{\gamma}_{l}\\}_{l=1,...,L}$. This
network can have different numbers of hidden layers $L$ and neurons $M^{(l)}$
at each layer from the previous networks. Furthermore, the linear function is
used for activation of its output layer to compute the posterior means
$\\{\boldsymbol{\mu}_{i}\\}_{i=1,...,I}$.

The posterior variances
$\\{\text{diag}(\boldsymbol{\sigma}^{2}_{i})\\}_{i=1,...,I}$ is computed in
the same way as the posterior means but with different sets of weights and
biases and with the acitivation of the MLP output being the Sigmoid function.
Finally, each user embedding $\boldsymbol{u}_{i}$ in the matrix
$\boldsymbol{U}$ is generated based on the corresponding
$\boldsymbol{\mu}_{i}$ and $\text{diag}(\boldsymbol{\sigma}^{2}_{i})$ using
the reparametrization trick (Kingma and Welling, 2014). In addition, we have
also tried to clip the embeddings to make them non-negative. However, in the
experiment, we have not found any difference in performance resulted from the
clipping operation.

###  3.3 Cross-Feedback Connectivity

The cross-feedback connectivity provides a solution to effectively modelling
the variational distributions
$q(\boldsymbol{u}_{i}|\boldsymbol{V}_{i},\boldsymbol{X}_{i})$ and
$q(\boldsymbol{v}_{j}|\boldsymbol{U}_{j},\boldsymbol{X}_{j})$ in Eq. 7. With
such connectivity, we are able to optimize an ELBO that aims to capture the
full inter-dependency among users, items and ratings. More specifically, in
our framework, there exist two cross-feedback loops. As shown by Figure 1, one
feeds the user-side (encoder) output back into the item-side latent input
layer. The other feeds the item-side output back into the user-side latent
input layer. According to Algorithm 1, this cross feedback mechanism is
conducted iteratively alongside the variational inference for the embeddings
$\boldsymbol{U}$ and $\boldsymbol{V}$. In our algorithm, steps 6 and 15
specify the cross feedback loops between the latent inputs at one side and the
encoder outputs at the other side.

In Algorithm 1, users and items are batched according to the predefined user
and item batch sizes $B_{\boldsymbol{U}}$ and $B_{\boldsymbol{V}}$. As a
result, the observed and latent inputs of the encoders at both sides need to
be sliced by the corresponding batches before passed through the networks. In
addition, our algorithm employs nested iteration across the item and the user
batches rather than sequential iteration. The latter alternates the update of
the user-side and the item-side weights and biases while the nested iteration
updates these parameters all at once. We found in the experiment that the
nested iteration allows our framework to achieve higher prediction accuracy at
the cost of more running time.

1 Input: ratings $\boldsymbol{X}$, user batch size $B_{\boldsymbol{U}}$, item
batch size $B_{\boldsymbol{V}}$

2 Initialize entries of $\boldsymbol{U},\boldsymbol{V}$ by sampling them from
$\mathcal{N}(\mu,\sigma^{2})$

3 Sample $\left\lceil{\frac{I}{B_{\boldsymbol{U}}}}\right\rceil$ user batches
and $\left\lceil{\frac{I}{B_{\boldsymbol{V}}}}\right\rceil$ item batches

4 Randomly initialize user and item-side encoder parameters

5 for  _not converged or not max iterations_ do

6 Build user-side latent inputs $\boldsymbol{Z}_{\boldsymbol{U}}$ from
$\boldsymbol{V}$ and item-side latent inputs $\boldsymbol{Z}_{\boldsymbol{V}}$
from $\boldsymbol{U}$

7 for  _each user batch_ do

8 Slice $\boldsymbol{X},\boldsymbol{Z}_{\boldsymbol{U}}$ by the user batch

9 for  _each item batch_ do

10 Slice $\boldsymbol{X}^{T},\boldsymbol{Z}_{\boldsymbol{V}}$ by the item
batch

11 Compute gradients $\nabla\text{Q}$ specified in Eq. 7 w.r.t. all the
weights and biases over the slices

12 Update the weights and biases with the computed gradients

13 end for

14

15 end for

16 Resample $\boldsymbol{U},\boldsymbol{V}$ using reparametrization trick

17 end for

Algorithm 1 VAE-BMF with Cross-Feedback Connectivity

###  3.4 User-Side and Item-Side Attention

To further strengthen the effect of the cross-feedback connectivity, we
propose a user-side and item-side attention component based on the latent
inputs of each side. The basic idea is that some items tend to be more
representative of a user’s interest or taste, while some users’ tastes might
better represent the quality or characteristics of an item. Furthermore, the
encoders of our framework transform the entire latent inputs
$\boldsymbol{Z}_{\boldsymbol{U}}$ and $\boldsymbol{Z}_{\boldsymbol{V}}$
without recognizing the individual embeddings underneath and their
relationships with the target embeddings. In contrast, the two attention
mechanisms capture such relationships for the user and item embeddings
respectively.

To make a succinct description, only the user-side attention is specified here
while the item-side attention, which has the same format and a counterpart set
of parameters, is omitted. In this case, the posterior mean vector
$\boldsymbol{\mu}_{i}$ of the $i$-th user is attended to the embedding of the
$j$-th item with a weight $a_{ij}$ calculated as follows:

$a_{ij}=\frac{\mathbbm{1}_{\\{\boldsymbol{v}_{j}\in\boldsymbol{V}_{i}\\}}\boldsymbol{\mu}_{i}\boldsymbol{\Theta}\boldsymbol{v}^{T}_{j}}{\sum\nolimits^{J}_{j^{\prime}=1}\mathbbm{1}_{\\{\boldsymbol{v}_{j^{\prime}}\in\boldsymbol{V}_{i}\\}}\boldsymbol{\mu}_{i}\boldsymbol{\Theta}\boldsymbol{v}^{T}_{j^{\prime}}}$
(9)

where $\boldsymbol{\Theta}\in\mathbbm{R}^{K\times K}$ are the weight matrix to
be learned at the user side. Then, we compute the weighted average of the item
embeddings according to their weights to obtain the attention vector
$\boldsymbol{c}_{i}$ for the $i$-th user:

$\boldsymbol{c}_{i}=\sum\nolimits^{J}_{j=1}\mathbbm{1}_{\\{\boldsymbol{v}_{j}\in\boldsymbol{V}_{i}\\}}a_{ij}\boldsymbol{v}_{j}$
(10)

Note that in Eq. 9 and Eq. 10, only the items rated by the $i$-th user (as
shown by the indicator functions) are attended to by $\boldsymbol{\mu}_{i}$.
We find in the experiment that this attention strategy improves the
performance of our framework when responses are abundant but harm the
performance when responses are sparse. In the latter case, we instead attend
$\boldsymbol{\mu}_{i}$ to all the item embeddings. By doing so, the attention
of a user embedding moves from being local to being global by extracting
useful information from the entire item population. Finally, the attention
vector $\boldsymbol{c}_{i}$ can be integrated with the original vector
$\boldsymbol{\mu}_{i}$ in different ways to form a new user embedding, such as
element-wise addition:
$\boldsymbol{\mu}_{i}:=\boldsymbol{\mu}_{i}+\boldsymbol{c}_{i}$; or
concatenation:
$\boldsymbol{\mu}_{i}:=[\boldsymbol{\mu}_{i};\boldsymbol{c}_{i}]$. In this
paper, we choose to use the concatenation strategy. The new user embedding
$\boldsymbol{\mu}_{i}$ thus has a dimension of $2K$ and we transform it back
to be $K$-dimensional by simply placing a linear layer on top of it which we
omit in Figure 1 for the simplicity of presentation. Likewise, the variance
vector $\text{diag}(\boldsymbol{\sigma}^{2}_{i})$ for the $i$-th user is also
attended to the item embeddings by following the same strategy with a
different attention weight matrix $\boldsymbol{\Lambda}$.

##  4 Experiments and Results

In this section, we compare our framework with various state-of-the-art CF
models across five real-world datasets. Furthermore, we study the importance
of the cross-feedback and attention mechanisms in improving the model
performance and accelerating the model convergence. This includes an ablation
study and a convergence analysis in which we compare the performance and the
convergence (rate) of different variants of the framework by taking away its
different components.

###  4.1 Datasets

We investigate five medium to large real-world datasets. The major information
of these datasets is summarized in Table 1.

Datasets | # users | # items | # ratings
---|---|---|---
ML-1M | 6,040 | 3,900 | 1,000,209
ML-10M | 69,878 | 10,631 | 9,999,989
ML-20M | 135,543 | 21,651 | 19,972,967
Amazon Pet | 46,403 | 61,673 | 1,008,336
Amazon Kindle | 92,351 | 55,924 | 1,619,155
Table 1: The summary of the key attributes of the datasets used in the
experiment.

MovieLens 1M, 10M, 20M (ML-1M, ML-10M, ML-20M): These are medium to large
datasets of user-movie ratings collected from
MovieLens222https://movielens.org/, a movie recommendation service, over
different periods of time.

Amazon pet supplies and Kindle stores (Amazon-Pet, Amazon-Kindle): These
datasets contain ratings from Amazon on different types of
products333http://jmcauley.ucsd.edu/data/amazon/. For these two datasets, we
only preserve users and items that have at least 10 ratings.

###  4.2 Experimental setup

To evaluate the performance of our framework, we randomly split the observed
rating data into the training, validation and testing data subsets with a
70%-15%-15% ratio. The evaluation metrics employed in the experiment include
the RMSE, Recall@N and NDCG@N where N = 20 or 50. The RMSE measures the
difference between the observed and predicted ratings irrespective of the
specific users or items. The Recall and NDCG metrics are measured specific to
each user and are originally designed for binary ratings (e.g. clicks). In
this case, we binarize the ratings accordingly (i.e. 1 if the rating is
greater than 3, suggesting that a user is interested; 0 otherwise, for not
being interested). The difference between the Recall and the NDCG metrics is
that the former accounts for all the top-N ranked items as being equally
important, while the latter assigns more weights to items ranked higher. Here,
we calculate the average Recall and NDCG scores across all the users who rated
at least two items by following the formulas specified in (Liang et al, 2018).
Note that in (Liang et al, 2018), test datasets were held out specific to
users while in this work, they are held out regardless of users and items.

We perform hyper-parameter optimization for our framework based on its RMSE
scores on the validation datesets. More specifically, a grid search is
conducted on the embedding dimensions $K$ and $K^{\prime}$ among
$\\{10,15,25,50\\}$, the number of hidden layers $L$ and $L^{\prime}$ among
$\\{0,1,2\\}$, the number of hidden neurons $M^{(l)}$ and $M^{(l^{\prime})}$
at each layer among $\\{10,25,50,100\\}$ and the controlling parameters
$\beta_{\boldsymbol{u}},\beta_{\boldsymbol{v}}$ for the KL terms among
$\\{10^{-5},10^{-4},...,0.1,1\\}$. We restrict the grid search such that
$K\leq K^{\prime}$ and $K,K^{\prime}\leq M^{(l)},M^{(l^{\prime})}$ at any
hidden layer. As for the user and item batch sizes $B_{\boldsymbol{U}}$ and
$B_{\boldsymbol{V}}$, we enlarge them when the numbers of users and items in
the datasets are larger. This accelerates the framework training at the cost
of more memory consumption and the degraded framework performance in RMSE. In
the experiment, we carefully tuned the batch sizes considering this trade-off.
For the ML-1M dataset, we set $B_{\boldsymbol{U}},B_{\boldsymbol{V}}$ to be
100 given its relatively small numbers of users and items, while for the other
datasets, $B_{\boldsymbol{U}},B_{\boldsymbol{V}}$ are set to be 1,000.

###  4.3 Baselines

We compare our framework with the following baselines.

  * •

Neural Collaborative Filtering (NCF) (He et al, 2017): This model uses an MLP
network (with embedding layers for user and item ID one-hot encoding vectors)
to replace the traditional matrix factorization over the user and item
embeddings. We adhere to the choice of ReLU as the activation function from
the original paper.

  * •

Convolutional Neural Network based Collaborative Filtering (CNN-CF) (He et al,
2018): This model applies a CNN network to the outer product of the user and
the item embeddings to capture the user-item interactive patterns. We adopt
the default kernel size $2\times 2$ and the ReLU activation function from the
original paper. Furthermore, we change the original Bayesian personalized
ranking loss on the implicit data to the MSE loss on our explicit rating data.

  * •

Autoencoder based User-side & Item-side Collaborative Filtering (U-Rec, I-Rec)
(Sedhain et al, 2015): U-Rec encodes and decodes the rating matrix
$\boldsymbol{X}$ with a standard autoencoder. The encoder in this case can
only generate user embeddings. Likewise, I-Rec encodes and decodes
$\boldsymbol{X}^{T}$ with the encoder only generating item embeddings. We
adopt the ReLU function for the hidden layers and the Sigmoid function for the
output layer as described in the paper.

  * •

Variational & Denoising Autoencoder based Collaborative Filtering (VAE-CF,
DAE-CF) (Liang et al, 2018): These models are essentially user-side
autoencoders that only generate embeddings for the users. We change their
multinomial likelihood loss functions on the implicit data to the MSE loss
over our rating data. The other parts of the models remain unchanged. These
include the controlling parameter $\beta$ for the KL term and its decay
strategy in VAE-CF, the weight decay strategies and the dropout probabilities
in both models.

The optimization of the hyper-parameters for each of these models is based on
the validation RMSE. Table 2 summarizes the ranges of the hyper-parameter
values of each model for the optimization. All the models, except for the VAE-
CF, are regularized with $\text{L}_{2}$ norms on their parameters for which
the controlling hyper-parameters are selected among
$\\{0,10^{-2},10^{-1},1\\}$. For the VAE-CF model, the controlling parameter
$\beta$ for its KL term is selected among
$\\{10^{-5},10^{-4},...,10^{-1},1\\}$.

Models | $K$ | # hidden layers | # neurons | $\text{L}_{2}$
---|---|---|---|---
NCF | $\\{5,10,25,50,100\\}$ | 0 to 2 | $\\{10,25,50,100\\}$ | {0,0.01,0.1,1}
CNN-CF | $\\{8,16,32,64,128\\}$ | 1 to 6 | $\\{64\times 64,32\times 32,...\\}$ | {0,0.01,0.1,1}
U/I-Rec | $\\{5,10,25,50,100\\}$ | 0 to 2 | $\\{10,25,50,100\\}$ | {0,0.01,0.1,1}
VAE-CF | $\\{10,25,50,100,200\\}$ | 0 to 2 | $\\{50,100,200,400\\}$ | N/A
DAE-CF | $\\{10,25,50,100,200\\}$ | 0 to 2 | $\\{50,100,200,400\\}$ | {0,0.01,0.1,1}
Table 2: The hyper-parameter value ranges of each baseline model for the hyper-parameter optimization on the validation RMSE. Metrics | Models | ML-1M | ML-10M | ML-20M | Amazon-Pet | Amazon-Kindle
---|---|---|---|---|---|---
| NCF | 0.870 | 0.831 | 0.822 | 1.098 | 0.734
| CNN-CF | 0.875 | 0.844 | 0.839 | 1.119 | 0.770
| U-Rec | 0.883 | 0.893 | 0.833 | 1.136 | 0.772
| I-Rec | 0.859 | 0.823 | 0.818 | 1.213 | 0.787
RMSE | DAE-CF | 0.911 | 0.852 | 0.841 | 1.247 | 0.762
| VAE-CF | 0.903 | 0.834 | 0.836 | 1.211 | 0.747
| Ours | 0.842 | 0.804 | 0.798 | 1.074 | 0.702
| NCF | 0.797 | 0.799 | 0.797 | 0.908 | 0.963
| CNN-CF | 0.774 | 0.778 | 0.796 | 0.886 | 0.948
| U-Rec | 0.796 | 0.794 | 0.802 | 0.896 | 0.963
| I-Rec | 0.767 | 0.759 | 0.776 | 0.883 | 0.942
NDCG@20 | DAE-CF | 0.797 | 0.801 | 0.799 | 0.899 | 0.961
| VAE-CF | 0.798 | 0.801 | 0.803 | 0.909 | 0.962
| Ours | 0.802 | 0.806 | 0.809 | 0.911 | 0.966
| NCF | 0.788 | 0.791 | 0.791 | 0.907 | 0.963
| CNN-CF | 0.758 | 0.762 | 0.790 | 0.882 | 0.946
| U-Rec | 0.788 | 0.788 | 0.796 | 0.894 | 0.963
| I-Rec | 0.754 | 0.744 | 0.771 | 0.879 | 0.941
NDCG@50 | DAE-CF | 0.789 | 0.793 | 0.795 | 0.898 | 0.960
| VAE-CF | 0.787 | 0.794 | 0.793 | 0.908 | 0.960
| Ours | 0.793 | 0.797 | 0.800 | 0.910 | 0.965
| NCF | 0.775 | 0.778 | 0.790 | 0.905 | 0.959
| CNN-CF | 0.756 | 0.761 | 0.792 | 0.882 | 0.939
| U-Rec | 0.776 | 0.781 | 0.784 | 0.893 | 0.958
| I-Rec | 0.749 | 0.740 | 0.755 | 0.879 | 0.933
Recall@20 | DAE-CF | 0.778 | 0.782 | 0.785 | 0.896 | 0.958
| VAE-CF | 0.777 | 0.782 | 0.786 | 0.906 | 0.958
| Ours | 0.781 | 0.784 | 0.790 | 0.906 | 0.961
| NCF | 0.763 | 0.768 | 0.782 | 0.904 | 0.958
| CNN-CF | 0.742 | 0.749 | 0.779 | 0.880 | 0.937
| U-Rec | 0.764 | 0.763 | 0.775 | 0.892 | 0.957
| I-Rec | 0.738 | 0.726 | 0.743 | 0.876 | 0.932
Recall@50 | DAE-CF | 0.766 | 0.772 | 0.782 | 0.896 | 0.957
| VAE-CF | 0.767 | 0.773 | 0.783 | 0.905 | 0.958
| Ours | 0.770 | 0.774 | 0.783 | 0.906 | 0.961
Table 3: Evaluation results of each model across the five datasets with
different metrics.

###  4.4 Predictive Analysis

In this section, we evaluate the effectiveness of our framework by comparing
its performance against that of the baseline models on predicting the observed
data from the testing datasets. Table 3 summarizes the testing RMSE, Recall@N
and NDCG@N, where N = 20 or 50, of each model over the five benchmark
datasets. It can be observed that our framework has achieved lower testing
RMSE than all the baselines on these datasets. More specifically, it manages
to outperform the second-best model by at least 0.017 (on the ML-1M dataset)
and at most 0.032 (on the Amazon-Kindle dataset). It is also interesting to
see that the I-Rec model holds the second place across all the Movielens
datasets while it is the NCF model for the Amazon datasets. We conjecture that
this has to do with how the I-Rec model works. It performs better when ratings
are abundant across the items which is the case in the Movielens data.
However, we can see from Table 1 that the ratings in the Amazon datasets are
much sparser across the users and the items. The sparsity at both sides cause
the I-Rec and the U-Rec to degrade in their performance. In comparison, our
framework appears to be more robust when the sparsity occurs in the Amazon
datasets.

To further compare the multiple models over the five datasets, we use critical
difference (CD) diagram which is one of the standard evaluation tools in
machine learning research (Demšar, 2006; Benavoli et al, 2016). We apply the
Friedman test to compare the ranks of multiple models (Demšar, 2006). In this
case, the null hypothesis states that there is no significant difference in
the mean rankings of the multiple models (at a statistical significant level
0.1444The level is set to be 0.1 to account for a small number of datasets in
our experiment.). Figure 2(e) shows the mean ranks of different models on
RMSE; ours leading at 1, while NCF and I-Rec rivals at 2.58 and 2.79
respectively (statistically no different). It can also be seen that our
framework has statistically different RMSE results from VAE-CF555We compare
only the top 4 models to keep the number of models smaller than the number of
datasets..

In terms of the Recall and the NDCG metrics, our framework performs better
than or equal to any other baseline models across all the datasets. More
specifically, on the ML-1M, ML-10M and Amazon-Kindle datasets, our framework
consistently outperforms all the baseline models in all these metrics. On the
ML-20M dataset, our framework ties with the VAE-CF model in the Recall@50
while on the Amazon-Pet dataset, it ties with the same model in the Recall@20.
Another interesting finding is that the I-Rec model, despite its superiority
in the RMSE on the Movielens datasets, is significantly inferior to the other
models across all the datasets under these user-oriented metrics. This is not
unusual as the I-Rec model focuses on modelling the characteristics of each
item rather than those of the individual users. In comparison, the performance
of models like the U-Rec, VAE-CF and DAE-CF, which are designed to capture
users’ underlying characteristics, is much less affected when evaluated under
the user-oriented metrics. It can also be observed that models which have much
more advantage in performance under the RMSE metric (e.g. our framework and
the NCF) generally have less to no advantage compared to user-oriented models
(e.g. VAE-CF and DAE-CF) under the NDCG and Recall metrics (see Figures 2(a),
2(c) and 2(d)). This is largely due to the fact that these models are trained
to minimize the MSE loss rather than optimize the ranking of relevant items.
With that being said, our framework manages to yield significantly different
NDCG@50 performance from the other top models (see Figure 2(b)).

(a) The CD diagram for NDCG@20

(b) The CD diagram for NDCG@50

(c) The CD diagram for Recall@20

(d) The CD diagram for Recall@50

(e) The CD diagram for RMSE

Figure 2: The critical difference (CD) diagrams showing the average ranks of
the top 4 models on different evaluation metrics across the five datasets. The
lower the rank (further to the left) the better performance of a model on the
particular metric compared to the others on average. A line in each diagram
indicates that there is no significant difference in performance among the
models crossed by that particular line in terms of the Friedman test that
compares the ranks of multiple classifiers (Demšar, 2006).

As a further evaluation, we would like to see how robust our framework is
towards the data sparsity issue, i.e., small numbers of responses per user and
item, which often happens in the cold-start scenario (Rashid et al, 2008). We
sub-sampled all the datasets randomly (irrespective of users and items) to
make them sparse across both users and items. The sub-sampling percentages we
have taken are $1\%,2\%,3\%,5\%$ and $10\%$. The sub-sampled datasets are used
to train each model. The rest of the data is split equally for optimizing the
model hyper-parameters (with the validation datasets) and testing. Figure 3
shows the changes of the testing RMSE, NDCG and Recall scores of each model
across different sub-sampling percentages. It can be observed that overall,
our framework either outperforms or performs equally well against all the
baselines in terms of these metrics on each dataset. The results in this case
are largely consistent with the previous results on the full datasets; Our
framework clearly performs better than the baselines in terms of the RMSE. The
smallest margin in this case is 0.005 on the Amazon-Pet dataset when the sub-
sampling percentage is 3%. Its advantage in performance becomes relatively
less under the ranking-type metrics most likely because of it minimizing the
MSE loss. As for the baselines, the NCF model ranks at the second place in
terms of the testing RMSE, while it is the VAE-CF model that ranks at this
position in terms of the testing NDCG and Recall. It also appears that the
I-Rec model suffers the most from the sparsity issue when evaluated by the
ranking-type metrics.

Figure 3: The evaluation results of each model on the sub-sampled datasets
under different metrics; The X-axis shows the sampling percentages which are
1%, 2%, 3%, 5% and 10%. The Y-axis shows the testing RMSE.

###  4.5 Ablation Study

In this section, we conduct an ablation study to illustrate the importance of
the three main components of our framework: the observed data inputs, the
cross-feedback and the attention mechanisms. The observed data inputs include
the rating matrix $\boldsymbol{X}$ and its transpose $\boldsymbol{X}^{T}$ that
are fed to the user-side and item-side encoders respectively. Removing them
and the entire MLP encoders that generate the corresponding intermediate
embeddings from the framework means that the explicit data information is no
longer exploited. The cross-feedback and attention components are based on the
two latent inputs $\boldsymbol{Z}_{\boldsymbol{U}}$ and
$\boldsymbol{Z}_{\boldsymbol{V}}$. Removing these components from our
framework means that it no longer leverages the implicit data information for
the variational inference.

With each of its components taken away, we train and optimize the hyper-
parameters of our framework with the same training and validation datasets.
Afterwards, we evaluate it on the same testing datasets in terms of all the
metrics. Table 4 shows the study results. It can be observed that all the
components are important to our framework with respect to all the metrics.
This is because when each component is removed, the corresponding performance
of our framework degrades notably. Especially, the degradation caused by
removing the data inputs is much more greater than that caused by removing the
cross-feedback and attention mechanisms. This is expected as the explicit data
information is more useful in improving the model performance compared to the
implicit information. In addition, removing the attention mechanism in general
causes the least degradation in performance; which still shows its importance
as it further boosts the performance of our framework to surpass different
baselines on different datasets. Overall, the ablation study shows the
efficacy of the cross-feedback and attention mechanisms in improving the
performance of our framework.

| Components | ML-1M | ML-10M | ML-20M | ML-Pet | ML-Kindle
---|---|---|---|---|---|---
| No data input | 0.917 | 0.902 | 0.878 | 1.241 | 0.813
| No cross feedback | 0.864 | 0.829 | 0.824 | 1.110 | 0.739
RMSE | No attention | 0.852 | 0.812 | 0.809 | 1.087 | 0.718
| Full Model | 0.842 | 0.804 | 0.798 | 1.074 | 0.702
| No data input | 0.795 | 0.797 | 0.804 | 0.907 | 0.961
| No cross feedback | 0.798 | 0.801 | 0.807 | 0.908 | 0.963
NDCG@20 | No attention | 0.800 | 0.803 | 0.808 | 0.910 | 0.965
| Full Model | 0.802 | 0.806 | 0.809 | 0.911 | 0.966
| No data input | 0.787 | 0.790 | 0.793 | 0.906 | 0.961
| No cross feedback | 0.790 | 0.793 | 0.796 | 0.907 | 0.963
NDCG@50 | No attention | 0.792 | 0.795 | 0.798 | 0.909 | 0.964
| Full Model | 0.793 | 0.797 | 0.800 | 0.910 | 0.965
| No data input | 0.775 | 0.776 | 0.786 | 0.902 | 0.956
| No cross feedback | 0.778 | 0.779 | 0.788 | 0.904 | 0.958
Recall@20 | No attention | 0.780 | 0.782 | 0.789 | 0.905 | 0.960
| Full Model | 0.781 | 0.784 | 0.790 | 0.906 | 0.961
| No data input | 0.764 | 0.767 | 0.781 | 0.901 | 0.955
| No cross feedback | 0.767 | 0.771 | 0.782 | 0.903 | 0.957
Recall@50 | No attention | 0.768 | 0.773 | 0.782 | 0.905 | 0.959
| Full Model | 0.770 | 0.774 | 0.783 | 0.906 | 0.961
Table 4: The ablation study shows that all components contribute significantly
to the framework performance in terms of the RMSE.

###  4.6 Convergence analysis

To further investigate the importance of the cross-feedback and attention
mechanism, we inspect how the framework performance (i.e. the testing RMSE
scores) converges with and without the mechanism. More specifically, we
inspect the performance of the following five variants of our framework:

  * •

Variant A: our framework without the cross-feedback mechanism, thereby no
latent input, and the number of hidden layers is $L^{\prime}=0$ for the MLP
networks over the observed input.

  * •

Variant B: our framework with the cross-feedback (but no attention) mechanism
and the number of hidden layers (before “concat”) is $L^{\prime}=0$ for the
MLP networks over both the observed and the latent inputs. The number of
hidden layers (after “concat”) is $L=0$.

  * •

Variant C: our framework without the cross-feedback mechanism and the number
of hidden layers is $L^{\prime}=1$ for the MLP networks over the observed
input.

  * •

Variant D: our framework with the cross-feedback (but no attention) mechanism
and the number of hidden layers (before “concat”) is $L^{\prime}=1$ for the
MLP networks over both the observed and the latent inputs. The number of
hidden layers (after “concat”) is $L=0$.

  * •

Variant E: our framework with the cross-feedback and the attention mechanisms.
The number of hidden layers (before “concat”) is $L^{\prime}=1$ for the MLP
networks over both the observed and the latent inputs. The number of hidden
layers (after “concat”) is $L=0$.

Figure 4: The testing RMSE scores of the five variants across the first 100
training iterations over the ML-1M data. The X-axis shows the iteration
number, while the Y axis shows the testing RMSE scores.

Figure 4 shows how the testing RMSE scores of the five variants change during
the first 100 training iterations over the ML-1M dataset. It can be observed
that the three variants (i.e. variants B and D) with the cross-feedback
mechanism constantly outperform their counterparts without the mechanism (i.e.
variants A and C). Especially, the RMSE score of variant D declines much more
quickly than that of variant C at the early stage of the training. This
indicates that the cross-feedback mechanism can quickly improve the framework
performance even when the posterior inference for the embeddings has just
begun. We also notice that the line of variant B is much smoother than that of
variant A which still fluctuates after 50 iterations. Considering the two
variants have the same optimization configuration, this clearly shows that the
cross-feedback mechanism smooths and accelerates the convergence of our
framework. Furthermore, there is a huge performance gap between variants A and
B. This gap reconfirms that the cross-feedback mechanism can significantly
improve the framework performance when the explicit data information is not
sufficiently exploited (due to no hidden layer in the corresponding MLPs).
Finally, when the attention mechanism is integrated into our framework (see
variant E in the figure), the framework’s performance increases significantly
and converges almost steadily after 20 iterations. It is also worth noting
that the attention mechanism slows down the performance improvement (created
by the cross-feedback mechanism) at the early stage of the training. This is
most likely because the learning of the additional model parameters
$\boldsymbol{\Theta}$ and $\boldsymbol{\Lambda}$ (at each side) is tangled
with the learning of the embeddings, which is more difficult than the learning
of the embeddings alone.

As for the running time of our framework, which includes the model induction
and prediction time, it is largely determined by the input dimension which is
$\mathcal{O}\big{(}(K+1)\times(I+J)\big{)}$ for our framework. Inevitably,
this results in our framework being notably slower than simpler models such as
VAE-CF, DAE-CF, U-Rec and I-Rec in exchange for improved performance. With
that being said, thanks to the generally fast convergence, our framework is in
general just 4-5 times slower than these models on large-scale datasets such
as ML-20M and Amazon datasets; which have large numbers of users and items. We
found empirically that the running time of our model overall rivals that of
the NCF and CNN-CF models. We have also experimented with an accelerated
variant of our framework which, instead of taking in a concatenation of the
embeddings, uses their average as the latent input. However, we observed that
this variant constantly yielded worse performance, indicating that it might
fail to fully exploit the implicit information.

##  5 Conclusion

In this paper, we propose a variational auto-encoder based Bayesian matrix
factorization framework for collaborative filtering. Compared to the previous
work, this framework leverages not only the explicit data information from the
ratings but also the implicit information from the user and item embeddings
for improving the variational approximation of their posterior parameters.

Our framework is characterized by the iterative cross feedback of the user and
item embeddings to each other’s encoder input layers. The cross-feedback
inputs provide useful implicit information regarding the counterparts. Both
the explicit and implicit information is learned by dedicated MLP networks
whose final outputs are concatenated. Then, the result is fed into another MLP
network which fuses and map the two types of information into the posterior
parameters of the embeddings. To better exploit the implicit information
provided by the cross-feedback component, we propose to further leverage the
attention of the posterior parameters of one side to the embeddings cross fed
from the other side.

Experimental results show that our framework outperforms six state-of-the-art
NN based collaborative filtering approaches in terms of the reconstruction
error over both the full and sub-sampled data. They also show that our
framework performs no worse than those baselines with respect to the ranking
of relevant items in both scenarios. It is further found from the ablation
study and the convergence analysis that both the cross-feedback and the
attention mechanisms not only improve the framework performance but also
accelerate its convergence even though the attention mechanism slows down such
improvement at the beginning of the training. As the future work, we would
like to investigate how to accelerate the running time of our framework by
improving its latent input structure.

## References

  * Baldi (2012) Baldi P (2012) Autoencoders, unsupervised learning, and deep architectures. In: Proceedings of ICML Workshop on Unsupervised and Transfer Learning, PMLR, Proceedings of Machine Learning Research, vol 27, pp 37–49
  * Benavoli et al (2016) Benavoli A, Corani G, Mangili F (2016) Should we really use post-hoc tests based on mean-ranks? The Journal of Machine Learning Research 17(1):152–161
  * Blei et al (2017) Blei DM, Kucukelbir A, McAuliffe JD (2017) Variational inference: A review for statisticians. Journal of the American statistical Association 112(518):859–877
  * Demšar (2006) Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research 7(Jan):1–30
  * Gopalan et al (2015a) Gopalan P, Hofman JM, Blei DM (2015a) Scalable recommendation with hierarchical poisson factorization. In: Proceedings of the Thirty-First Conference on Uncertainty in Artificial Intelligence, AUAI Press, Arlington, Virginia, United States, UAI’15, pp 326–335, URL http://dl.acm.org/citation.cfm?id=3020847.3020882
  * Gopalan et al (2015b) Gopalan P, Hofman JM, Blei DM (2015b) Scalable recommendation with hierarchical poisson factorization. In: Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence, AUAI Press, UAI’15, pp 326–335
  * Harvey et al (2011) Harvey M, Carman MJ, Ruthven I, Crestani F (2011) Bayesian latent variable models for collaborative item rating prediction. In: Proceedings of the 20th ACM international conference on Information and knowledge management, pp 699–708
  * He et al (2017) He X, Liao L, Zhang H, Nie L, Hu X, Chua TS (2017) Neural collaborative filtering. In: Proceedings of the 26th international conference on world wide web, pp 173–182
  * He et al (2018) He X, Du X, Wang X, Tian F, Tang J, Chua TS (2018) Outer product-based neural collaborative filtering. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp 2227–2233
  * Kingma and Welling (2014) Kingma DP, Welling M (2014) Auto-encoding variational bayes. In: Proceedings of the 2nd International Conference on Learning Representations (ICLR)
  * Koren and Bell (2015) Koren Y, Bell R (2015) Advances in collaborative filtering. In: Recommender systems handbook, Springer, pp 77–118
  * Koren et al (2009) Koren Y, Bell R, Volinsky C (2009) Matrix factorization techniques for recommender systems. Computer 42(8):30–37
  * Lakshminarayanan et al (2011) Lakshminarayanan B, Bouchard G, Archambeau C (2011) Robust bayesian matrix factorisation. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp 425–433
  * Li and She (2017) Li X, She J (2017) Collaborative variational autoencoder for recommender systems. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp 305–314
  * Liang et al (2018) Liang D, Krishnan RG, Hoffman MD, Jebara T (2018) Variational autoencoders for collaborative filtering. In: Proceedings of the 2018 World Wide Web Conference, International World Wide Web Conferences Steering Committee, p 689–698
  * Mnih and Salakhutdinov (2008) Mnih A, Salakhutdinov RR (2008) Probabilistic matrix factorization. In: Advances in neural information processing systems, pp 1257–1264
  * Rashid et al (2008) Rashid AM, Karypis G, Riedl J (2008) Learning preferences of new users in recommender systems: an information theoretic approach. ACM SIGKDD Explorations Newsletter 10(2):90–100
  * Salakhutdinov and Mnih (2008) Salakhutdinov R, Mnih A (2008) Bayesian probabilistic matrix factorization using markov chain monte carlo. In: Proceedings of the 25th international conference on Machine learning, pp 880–887
  * Sedhain et al (2015) Sedhain S, Menon AK, Sanner S, Xie L (2015) Autorec: Autoencoders meet collaborative filtering. In: Proceedings of the 24th international conference on World Wide Web, pp 111–112
  * Shenbin et al (2020) Shenbin I, Alekseev A, Tutubalina E, Malykh V, Nikolenko SI (2020) Recvae: A new variational autoencoder for top-n recommendations with implicit feedback. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp 528–536
  * Wang et al (2019) Wang X, He X, Wang M, Feng F, Chua TS (2019) Neural graph collaborative filtering. In: Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval, pp 165–174
  * Wu et al (2016) Wu Y, DuBois C, Zheng AX, Ester M (2016) Collaborative denoising auto-encoders for top-n recommender systems. In: Proceedings of the 9th ACM International Conference on Web Search and Data Mining, pp 153–162
  * Zhang et al (2018) Zhang C, Bütepage J, Kjellström H, Mandt S (2018) Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence 41(8):2008–2026