figurec

# Variational Bandwidth Auto-encoder for Hybrid Recommender Systems

Yaochen Zhu  School of Remote Sensing and Information Engineering, Wuhan
University  Zhenzhong Chen<EMAIL_ADDRESS>School of Remote Sensing and
Information Engineering, Wuhan University

###### Abstract

Hybrid recommendations have recently attracted a lot of attention where user
features are utilized as auxiliary information to address the sparsity problem
caused by insufficient user-item interactions. However, extracted user
features generally contain rich multimodal information, and most of them are
irrelevant to the recommendation purpose. Therefore, excessive reliance on
these features will make the model overfit on noise and difficult to
generalize. In this article, we propose a variational bandwidth auto-encoder
(VBAE) for recommendations, aiming to address the sparsity and noise problems
simultaneously. VBAE first encodes user collaborative and feature information
into Gaussian latent variables via deep neural networks to capture non-linear
user similarities. Moreover, by considering the fusion of collaborative and
feature variables as a virtual communication channel from an information-
theoretic perspective, we introduce a user-dependent channel to dynamically
control the information allowed to be accessed from the feature embeddings. A
quantum-inspired uncertainty measurement of the hidden rating embeddings is
proposed accordingly to infer the channel bandwidth by disentangling the
uncertainty information in the ratings from the semantic information. Through
this mechanism, VBAE incorporates adequate auxiliary information from user
features if collaborative information is insufficient, while avoiding
excessive reliance on noisy user features to improve its generalization
ability to new users. Extensive experiments conducted on three real-world
datasets demonstrate the effectiveness of the proposed method. Codes and
datasets are released at https://github.com/yaochenzhu/vbae.

##  1 Introduction

††Corresponding author: Zhenzhong Chen, E-mail<EMAIL_ADDRESS>
In the era of information overload, people have been inundated by large
amounts of online content, and it becomes increasingly difficult for them to
discover interesting information. Consequently, recommender systems play a
pivotal role in modern web applications due to their ability to help users
discover items that they may be interested in from a large collection of
candidates. Based on how recommendations are made, existing recommender
systems can be categorized into three classes [1]: collaborative-based
methods, content-based methods, and hybrid methods. Collaborative-based
methods [2, 3] predict user preferences by exploiting their past activities,
such as clicks or ratings, where the recommendation quality relies heavily on
peers with similar behavior patterns. Content-based methods [4, 5], on the
other hand, make recommendations based on users or items that share similar
features. Hybrid methods [6, 7, 8] combine the advantages of both worlds where
the collaborative information and user/item features are comprehensively
considered to generate more precise recommendations.

Recent years have witnessed an upsurge of interest in employing auto-encoders
[9] to both collaborative and content-based recommender systems, where compact
representations of sparse ratings [10, 11] or high-dimensional user/item
features [12, 13] can be learned to more effectively exploit the similarity
patterns between users or items for recommendation. As a Bayesian version of
auto-encoder where the encoded latent representations are modeled as random
variables, VAE has demonstrated superiority compared to other forms of auto-
encoders, such as the contractive auto-encoder [14] and the denoising auto-
encoder [15]. Among them, Collaborative variational auto-encoder (CVAE) [16]
first used a VAE to infer latent item content embeddings from tf-idf textual
features, and then iteratively fine-tune the embeddings with rating
information via matrix factorization. Multi-VAE [10], in contrast, used the
VAE in the collaborative setting to learn compact user embeddings from
discrete user ratings. Recently, Macrid-VAE [17, 18] further extended Multi-
VAE, where learned user representations were constrained to disentangle at
both the macro and the micro -levels to improve the robustness and
interpretability of learned embeddings.

However, it is difficult to generalize the VAE to a hybrid recommender system,
due to various challenges from both the collaborative and content-based
components (Fig. 1). As users tend to vary in their activity levels and
tastes, user embeddings learned through collaborative filtering bear different
degrees of uncertainty, which hinders good recommendations for users with
unreliable collaborative embeddings (e.g. user #4 and #5 in Fig. 1). The
uncertainty mainly comes from three aspects: (1) Sparsity: First, for a user
with sparser interactions (user #4), her associated embedding is more
unreliable due to the information insufficiency in her historical
interactions, which makes the similarity measure induced by the embedding
space less informative compared to users with denser interactions. (2)
Diversity: Even if a user has denser interactions, we cannot safely conclude
that we can estimate her preferences with more confidence, because her ratings
may focus on a few types of items, which makes the collaborative information
conveyed by different items correlated. However, if she rated items of more
diverse types, we can have more confidence in the estimate of her preferences
based on the rating information because the item space is more thoroughly
explored. (3) Overlapping: In addition, the uncertainty of a user embedding
may also be large if the items the user has interacted with are seldomly
visited by other users (user #5). Considering two users who click the same
number of items, the items that the former user click are also clicked by many
other users, while the latter user only clicks items that no other users have
yet clicked, the embedding uncertainty of the latter user would be larger than
that of the former user.

Figure 1: Challenges associated with collaborative and feature components of a
hybrid recommendation system: (left) users whose ratings are sparse or who
rate only rarely-rated items have large uncertainty in their collaborative
embeddings; (right) user/item features contain large amounts of information
that are irrelevant for recommendations.

Although the user/item features could be exploited to reduce the uncertainty
incurred by the sparsity of ratings, the main obstacle for utilizing them lies
in the heavy noise that may outweigh the useful information. Here, by noise,
we mean any pattern that is irrelevant for the recommendation purpose, which
should be distinguished from the low-level noises such as image blur, audio
aliasing, and textual typos. Considering, for example, recommending academic
articles to researchers, the related work and the empirical study sections are
less informative compared to the abstract of methodology, and generally they
should be regarded as noise in recommendations, although they both contain
valuable information once a researcher gets attracted by the abstract and
decides to delve deeper into the article. Moreover, since different users may
consider varied aspects when rating an item, such noise exhibits a
personalized characteristic that makes it difficult to eliminate [19, 20]. A
similar analysis could be made for the user features, since the collection of
certain user attributes, such as location, etc., may raise up privacy issues,
and a widely adopted surrogate strategy is to empirically combine the features
of the items that the users have interacted with to build up the user profiles
for recommendations. In fact, the consensus in the community is that
collaborative filtering is more reliable than feature-based methods for large-
scale web recommendations if user interactions are sufficient to leverage
[21]. Therefore, a good hybrid recommender system should manage to avoid
unnecessary reliance on the noisy user/item features depending upon the
sufficiency level of the collaborative information in the ratings (e.g. user
#1 #2, #3 in Fig. 1), such that the noise in these features would not outweigh
the useful information and hurt the model generalization ability.

To address the above challenges, we propose an information-driven Bayesian
generative model called variational bandwidth auto-encoder (VBAE) for hybrid
recommendations. The model first jointly learns the generation of user
features and ratings from latent collaborative and user-feature embeddings.
These embeddings are modeled as Gaussian random variables and inferred via
deep neural networks through the auto-encoding variational Bayes (AEVB)
algorithm [22]. Furthermore, observing that the extracted user features could
be extremely noisy, we start by considering the fusion of the collaborative
and feature embeddings from an information-theoretic perspective, i.e., as a
virtual communication channel. We then introduce a novel user-dependent
channel that dynamically controls the amount of information that is allowed to
be accessed from the user features based on the collaborative information
already contained in the user ratings. A quantum-inspired uncertainty
measurement of the hidden rating embeddings is proposed accordingly to infer
the channel bandwidth by disentangling the uncertainty information in the
ratings from the semantic information. Through this mechanism, sufficient
auxiliary information can be accessed from user features if collaborative
information is inadequate, while unnecessary dependence of the model on noisy
user features can be avoided otherwise to make it generalize better to the
user features. The main contribution of this article is summarized as follows:

  * •

We present VBAE, a unified information-driven recommendation framework where
the generation of user ratings and features is parameterized via deep Bayesian
networks and their fusion is modeled as a personalized virtual communication
channel, such that the rating sparsity and feature noise problems can be
simultaneously addressed.

  * •

A novel quantum-inspired uncertainty measurement of the hidden rating
embedding is proposed to infer the bandwidth of the user-dependent channel,
which enhances the model generalization ability by dynamically controlling the
information allowed to be accessed from user features based on the sufficiency
level of collaborative information.

  * •

Two kinds of channel implementations with different desired properties, i.e.,
Bernoulli and Beta channels, are thoroughly discussed, with the corresponding
optimization objectives derived with distribution approximation and
variational inference to make them amenable to stochastic gradient descent.

  * •

The proposed VBAE empirically out-performs state-of-the-art hybrid
recommendation baselines. We also discover that the inferred bandwidth of the
channel variable can well distinguish users with different sufficiency levels
of collaborative information.

##  2 Related Work

As a special kind of deep neural network, auto-encoders aim to learn compact
representations of inputs by reconstruction [23]. Since both user ratings and
features are high-dimensional sparse vectors that make the direct manipulation
of them in the original space difficult, much effort has been dedicated to
improve representation learning strategies with auto-encoders by researchers
recently [9]. Generally, auto-encoder-based recommenders can be categorized
into two main classes: user-oriented auto-encoders (UAEs) [24, 15, 10, 25] and
item-oriented auto-encoders (IAEs) [9, 26], based on whether the auto-encoder
is used to tackle the user or the item side information.

###  2.1 Item-oriented Auto-encoders

The advent of IAE predates that of UAE, where item content auto-encoders are
built on top of matrix factorization (MF) -based collaborative backbones, such
as weighted matrix factorization [27], to incorporate auxiliary content
information into the factorized item collaborative embeddings. Two exemplar
methods from this category are CDL [28] and CVAE [16], where an item offset
variable is introduced to tightly couple the Bayesian stacked denoising auto-
encoder (SDAE) [29] or variational auto-encoder (VAE) [22] with MF to enhance
its performance. The MF and item content auto-encoder are trained in an
iterative manner. Recently, auto-encoders have been exploited to capture the
item collaborative information. Among them, DICER [26] was proposed to capture
non-linear item similarity based on their user ratings, and from which
disentangles the content information for a better generalization. Since in
collaborative IAEs, the input dimension and the number of trainable weights is
proportional to the number of users, and the number of training samples equals
the number of items, these methods generally require a large items-to-user
ratio such that a good representation of items can be learned for satisfactory
recommendations.

###  2.2 User-oriented Auto-encoders

Compared with IAEs, UAEs have attracted more attention among researchers
because they break the long-standing bottleneck of the linear collaborative
modeling limitation of MF and allow modeling users in a deeper manner [24, 15,
30]. Instead of factorizing the rating matrix into user and item embeddings,
UAE-based recommenders take the historical ratings of users as the inputs,
embed them into hidden user representations with a deep encoder network, and
from which reconstruct the ratings with a deep decoder network. The
reconstructed ratings for unrated items are then ranked for recommendations.
Since UAE-based recommenders eliminate the need of modeling item latent
variables and reconstruct the whole ratings directly from the user latent
embedding, another advantage of UAEs over MFs is that they are efficient to
fold in new users for whom historical ratings have been recorded, since
recommendations can be made with a single forward propagation. The first UAE-
based recommender system is the collaborative denoise auto-encoder (CDAE)
[15], where the input ratings are randomly masked with zeros to simulate the
rating missing process. Afterwards, Multi-VAE [10] was proposed where a VAE
with multinomial likelihood on ratings is used instead of the DAE, which
demonstrates clear advantages. However, one key problem for these
collaborative UAEs is that if the ratings of certain users are sparse, the
recommendation performance could be severely degenerated due to the lack of
collaborative information.

###  2.3 Hybrid Recommendation Techniques

Due to the wide availability of user features and various methods that build
user profiles with item features, hybriding UAE with auxiliary user features
to address the sparsity problem has become a new trend in the recommendation
community. A simple but effective method to incorporate user feature
information into UAE is to adopt the early fusion strategy similar to [31],
where the user features are concatenated with ratings as the input for the UAE
where only the ratings are reconstructed for recommendations. In this way, the
first dense layer of the UAE can be viewed as calculating the weighted
combination of user rating and features, which may be over-parameterized and
is susceptible to overfitting. A more sophisticated approach is the
conditional VAE (CondVAE) [32], where the user features are exploited to
calculate the conditional prior of user latent variables, which are then
updated into posterior by the collaborative information, i.e., ratings, for
recommendations. However, all these methods treat the relative importance of
collaborative and content information as fixed for all users, ignoring the
individual differences both in the reliability of their extracted features and
in the sufficiency level of historical rating information [20]. This is
problematic, since generally, user features contain much irrelevant
information and noise, and a good recommender system should avoid unnecessary
dependence on these features when collaborative information is sufficient to
improve generalization. Therefore, it motivates us to design the information-
driven variational bandwidth auto-encoder with a user-dependent channel to
fuse the user feature and collaborative information.

##  3 Methodology

Figure 2: (Left): the probabilistic graphical model (PGM) of the proposed
VBAE. (Right): the zoomed-in view of the collaborative and user feature
networks. User $i$ has sparser ratings than user $i^{\prime}$, which leads to
a larger uncertainty of her collaborative embedding, and VBAE infers a larger
bandwidth to allow more information of $\mathbf{z}_{t}$ to flow into
$\mathbf{v}$. For user i’, on the other hand, a smaller bandwidth is inferred
to avoid overfitting on noise in user features.

###  3.1 Problem Formulation

The focus of this article is on recommendations with implicit feedback [33].
We define the rating matrix as $\mathbf{R}\in\mathbb{R}^{I\times J}$, where
each row $\mathbf{r}_{i}$ is the bag-of-words vector denoting the whether user
$i$ visit each of the $J$ items. $\mathbf{R}$ is obtained by keeping track of
user activities for a certain amount of time. In addition, the users profiles
are represented by matrix $\mathbf{X}\in\mathbb{R}^{I\times S}$, where the row
vector $\mathbf{x}_{i}$ is the extracted feature for the $i$th user.
$\mathbf{x}_{i}$ can contain user inherent attributes such as her age,
location, and self-description, etc., or be built from the feature of items
that the user has interacted with when such information is not available. The
capital non-boldface letters $R_{i}$ and $X_{i}$ are used to denote the
corresponding random variables, respectively222The subscript $i$ will be
omitted for simplicity if no ambiguity exists.. Note that the density of
$r_{i}$, which is defined as $\sum_{j}r_{ij}$, could vary dramatically for
different $i$, and $x_{i}$ is generally high-dimensional and noisy. Given the
partial observation of the records in $\mathbf{R}$ and the user features
$\mathbf{X}$, the problem is to make predictions of the remaining ratings in
$\mathbf{R}$ so as to recommend new relevant items to users.

###  3.2 Model Overview

The PGM in the left part of Fig. 2 shows the overall generative and inference
process of VBAE. The details of VBAE are discussed in the following sections.

###  3.3 Generative Process

####  3.3.1 User Embeddings

In this article, the collaborative and feature embedding of users are assumed
to lie in $K$-dimensional latent spaces. Therefore, VBAE starts by generating
for each user the collaborative-based latent variable from a normal prior:

$\mathbf{z}_{b}\sim\mathcal{N}\left(\mathbf{0},\lambda_{b}^{-1}\mathbf{I}_{K}\right).$

Since user features are extracted as the auxiliary information to complement
the collaborative information, they are embedded in a user feature latent
variable $\mathbf{z}_{t}$, which is drawn from another Gaussian distribution
as follows:

$\mathbf{z}_{t}\sim\mathcal{N}\left(\mathbf{0},\lambda_{t}^{-1}\mathbf{I}_{K}\right).$

####  3.3.2 Channel with User-dependent Bandwidth

Traditional generative models for hybrid recommendation [28, 16, 34] directly
add $\mathbf{z}_{t}$ and $\mathbf{z}_{b}$ via an offset variable to form the
latent user embedding $\mathbf{v}$, or learn their fixed relative weights by
direct concatenation, and do not consider that the uncertainty of
$\mathbf{z}_{b}$ usually varies for different users due to the both explicit
(difference in the sparsity, diversity of ratings) and implicit (difference in
the number of overlapped ratings) factors, nor that $\mathbf{z}_{t}$ contains
lots of irrelevant information that may distract the recommendation model.
This could be problematic, since for users who have denser, more diverse
ratings and rate items that are also rated by many other users, their
associated collaborative embeddings $\mathbf{z}_{b}$ are more reliable, and
$\mathbf{v}$ can afford to access less information from $\mathbf{z}_{t}$ such
that unnecessary dependence of the model on noisy user features can be
avoided. For users with sparser ratings, however, $\mathbf{v}$ has to access
more information from $\mathbf{z}_{t}$ to enable a more personalized
recommendation even if the user features are noisy.

From an information-theoretic perspective, if we view the fusion of
$\mathbf{z}_{t}$ and $\mathbf{z}_{b}$ into $\mathbf{v}$ as a virtual
communication channel, the issue sources from the assumption that the channel
is deterministic and independent of the information already contained in the
collaborative embedding $\mathbf{z}_{b}$, where the individual difference in
sufficiency level of collaborative information is ignored. Therefore, to
address such a problem, we design a user-dependent channel in VBAE by
introducing a latent capacity variable $\alpha$ that determines the bandwidth
of the channel from $\mathbf{z}_{t}$ to $\mathbf{v}$. The bandwidth variable
$\alpha$ encodes for each user our belief towards how much extra information
is required from the user features given the information already contained in
the observed interactions. Through this mechanism, the channel dynamically
allocates the amount of information that is allowed to flow from
$\mathbf{z}_{t}$ to $\mathbf{v}$ conditional on $\mathbf{z}_{b}$. Two
strategies are explored to implement the user-dependent channel with bandwidth
$\alpha$. The first strategy we consider is called the "hard" channel, where
the bandwidth $\alpha$ is achieved when $\mathbf{v}$ losslessly accesses
$\mathbf{z}_{t}$ with probability $\alpha$, and accesses no information
otherwise [35]. In the generative case, we can introduce an auxiliary channel
variable $d$, and draw $d$ from a Bernoulli distribution with probability
$\alpha$:

$d\sim Bernoulli(\alpha).$

Although the hard channel complies more strictly with the definition of
bandwidth in information theory, it may results in training instability
because the bandwidth $\alpha$ only appears as a statistical property, i.e.,
an expectation when the feature and collaborative embeddings of a user are
repeatedly fused for multiple times. However, for one user, the user latent
embedding either access the user feature information or not in one
recommendation, which is coarse in granularity to distinguish user with
different uncertainty levels of collaborative information. Therefore, we
consider a second strategy, the "soft" channel, which is a relaxed version of
the hard channel and resembles more to the variational attention approach [36]
than the variational information bandwidth theory [35]. This strategy assumes
that the channel variable $d$ is drawn from a Beta distribution and the
bandwidth $\alpha$ determines the mean of the Beta:

$d\sim Beta(\alpha_{1},\alpha_{2}),\ \text{where}\
\alpha=\alpha_{1}/(\alpha_{1}+\alpha_{2}),$

where the channel curtails or amplifies the weights of user feature
information based on the bandwidth $\alpha$. Given only $\alpha$, however, the
Beta distribution for the channel $d$ is undetermined, as its variance remains
to be specified to calculate both $\alpha_{1}$ and $\alpha_{2}$. However,
since we care primarily about the bandwidth itself than its uncertainty, we
fix the variance of the Beta, which is then treated as a nuisance parameter,
to a small value $\delta^{2}_{fixed}$ for simplicity. Hereafter, we use the
mean-variance parameterization of Beta distribution unless otherwise
specified, since it explicitly contains the bandwidth as its first parameter.
We use VBAE-hard and VBAE-soft to distinguish the two channel implementation
strategies. The detailed comparisons between the soft and hard channels are
summarized in Fig. 3. After drawing $d$, the user latent variable $\mathbf{v}$
is deterministically calculated as

$\mathbf{v}=\mathbf{z}_{b}+d\cdot\mathbf{z}_{t},$

which defines the fusion process of $\mathbf{z}_{t}$ into $\mathbf{z}_{b}$ via
the user-dependent channel of VBAE-hard and VBAE-soft.

Figure 3: Comparison between the soft and hard channels.

####  3.3.3 Neural Network Implementations

To model the non-linear generation process of user features and ratings from
the corresponding latent variables, we parameterize the generative
distributions as deep neural networks. The user feature $\mathbf{x}$ is
generated from the user-feature latent variable $\mathbf{z}_{t}$ via a
multilayer perceptron (MLP) $\operatorname{Gen}_{t}(\mathbf{z}_{t})$: If
$\mathbf{x}$ is binary, we squashed the output of $\operatorname{Gen}_{t}$ by
the sigmoid function, and draw $\mathbf{x}$ from
$Bernoulli(\operatorname{sigmoid}(\operatorname{Gen}_{t}(\mathbf{z}_{t})))$,
or if $\mathbf{x}$ is real-value, we take the raw outputs of
$\operatorname{Gen}_{t}$ as the mean of a Gaussian distribution and draw
$\mathbf{x}$ from
$\mathcal{N}\left(\operatorname{Gen}_{t}(\mathbf{z}_{t}),\lambda_{x}^{-1}\mathbf{I}_{S}\right)$.
Finally, we put a multinomial likelihood on the ratings $\mathbf{r}$ as [10],
and generate $\mathbf{r}$ from the latent user variable $\mathbf{v}$ via
$p(\mathbf{r}|\mathbf{v})$ parameterized as
$Multi(N,\operatorname{softmax}(\operatorname{Gen}_{b}(\mathbf{v})))$, where
$\operatorname{Gen}_{b}$ is another MLP-based generative neural network. The
generation process of $\mathbf{x}$, $\mathbf{r}$ from $\mathbf{z}_{t}$,
$\mathbf{v}$ is given as follows:

(1) For each layer $l$ of the collaborative and the user-feature modules of
the generation network:

(a) For each column $n$ of the weight matrices, draw

$\mathbf{W}^{(l)}_{\\{b,t\\},:n}\sim\mathcal{N}\left(\mathbf{0},\lambda_{w}^{-1}\mathbf{I}_{K_{l}}\right);$

(b) Draw the bias vector from
$\mathbf{b}^{(l)}_{\\{b,t\\}}\sim\mathcal{N}\left(\mathbf{0},\lambda_{w}^{-1}\mathbf{I}_{K_{l}}\right)$;

(c) For ${\mathbf{h}}^{(l)}_{\\{b,t\\}}$ of a user $u$, draw

$\mathbf{h}^{(l)}_{\\{b,t\\}}\sim\delta\left(\operatorname{f_{act}}\left(\mathbf{h}^{(l-1)}_{\\{b,t\\}}\mathbf{W}^{(l)}_{\\{b,t\\}}+\mathbf{b}^{(l)}_{\\{b,t\\}}\right)\right).$

(2) For user features that are binary, draw

$\mathbf{x}\sim
Bernoulli\left(\operatorname{sigmoid}\left(\mathbf{h}_{t}^{(L)}\mathbf{W}^{(L+1)}_{t}+\mathbf{b}^{(L+1)}_{t}\right)\right);$

For user-item interactions, draw

$\mathbf{r}\sim
Multi\left(N,\operatorname{softmax}\left(\mathbf{h}_{b}^{(L)}\mathbf{W}^{(L+1)}_{b}+\mathbf{b}^{(L+1)}_{b}\right)\right),$

where $h_{t}^{(0)}=\mathbf{z}_{t}$, $h_{b}^{(0)}=\mathbf{v}$, $\lambda_{w}$ is
a hyperparameter, $f_{act}$ is the intermediate activation function and
$\delta$ is the Dirac Delta function. Step 1.c can be alternatively viewed as
putting Gaussian priors on the intermediate activations and setting the
precision to infinity.

The generative model of VBAE is described by the joint distribution of all
observed and hidden variables:

$\displaystyle p_{\bm{\theta}}(R,X,Z_{b},Z_{t},D)=$ $\displaystyle
p\left(R|Z_{b},Z_{t},D\right)p\left(X|Z_{t}\right)$ (1) $\displaystyle
p\left(Z_{b}\right)p\left(Z_{t}\right)p\left(D\right),$

where the symbol $\theta$ denotes the set of trainable parameters that pertain
to the generation network.

###  3.4 Inferential Process

Given Eq. (1), however, it is intractable to calculate the posterior
$p_{\theta}(Z_{b},Z_{t},D|R,X)$ exactly, as the non-linearity of generative
process precludes us from integrating over the latent space and calculating
the marginal distribution of the observed evidence $p_{\theta}(R,X)$.
Therefore, we resort to the amortized variational inference [37], where we
introduce a variational posterior $q_{\phi}(Z_{b},Z_{t},D|R,X)$ parameterized
by an inference neural network as an approximation to the true but intractable
posterior. Using the conditional independence assumptions implied by VBAE, the
joint variational posterior can be decomposed into the compact product of two
factors that make up two modules of the inference network: the collaborative
module $q_{\phi}(Z_{b},D|R)=q_{\phi}(Z_{b}|R)\times q_{\phi}(D|R)$ and the
user feature module $q_{\phi}(Z_{t}|X)$, which infer the user collaborative
embeddings, the bandwidth of the user-dependent channel, and user feature
embeddings from the user ratings and features, respectively.

####  3.4.1 Quantum-inspired Semantic and Uncertainty Measurement of
Collaborative Information

The collaborative module infers the collaborative-based latent user variable
and the personalized latent channel variable from the observed user-item
interactions. Since the bandwidth of the channel denotes the sufficiency level
of the collaborative information, an important role that this module should
play is to disentangle uncertainty and semantic information from the user
ratings. To achieve this objective, we first use an MLP to embed the raw,
sparse rating vector into a compact hidden representation
$\mathbf{h}^{inf}_{b}$ for each user. Inspired by [38], we then use the length
(L2-norm) of the hidden representation as the uncertainty measurement to
calculate the channel bandwidth and the direction (L2-normalized hidden
embedding) as the representation of semantic information to infer the latent
collaborative embedding. The details of the introduced semantic and
uncertainty measurement of the hidden rating embeddings is illustrated in Fig.
4, which draws inspirations from theoretic quantum mechanics. To see the
links, we first liken the hidden rating embedding $\mathbf{h}^{inf}_{b}$ to a
quantum superposition state of a physical system, which is represented by a
complex vector in $\mathbb{C}^{K_{h}}$. Then, the superposition vector has the
property that the norm of the vector is positively correlated with the
probability that this superposition is observed when measuring the system
(which depicts the fundamental uncertainty of quantum physics), and the
direction of the vector distinguishes this superposition state with other
states (which carries the state semantic information).

Figure 4: Illustration of the proposed quantum-inspired semantic and
uncertainty measurement of collaborative information from user interactions.

Such an uncertainty-semantic interpretation of the length and direction of a
vector also works for hidden rating embedding $\mathbf{h}^{inf}_{b}$ in
recommendation, where the norm and direction of $\mathbf{h}^{inf}_{b}$ can be
associated with similar meanings. To gain the intuition, we first decompose
the calculation of the hidden rating embedding $\mathbf{h}^{inf}_{b}$ from the
raw rating vector $\mathbf{r}$ with a dense layer into two basic operations,
embedding and element-wise sum, as follows:

$\mathbf{h}^{inf}_{b}=\mathbf{W}^{(1)}_{b}\cdot\mathbf{r}=\sum_{j:r_{j}=1}\mathbf{w}^{(1)}_{b,:j}.$
(2)

If we assume that each element in $\mathbf{W}^{(1)}_{b}$, i.e.,
$w^{(1)}_{b,ij}$ is independent and identically distributed (i.i.d.) Gaussian
variable with zero mean and a small variance $\mathcal{N}(0,\epsilon^{2})$,
the $j$th element in $\mathbf{h}_{b}$, which is denoted as $h_{b,j}$, is the
sum of $N_{int}$ independent Gaussian variables. Therefore, $h_{b,j}$ is also
a Gaussian variable that follows $N(0,N_{int}\epsilon^{2})$, where
$N_{int}=\sum_{j:r_{j}=1}1$ is the number of items this user has interacted
with, i.e., the density of ratings. According to the basic probability and
statistics theory, the squared L2-norm of $\mathbf{h}_{b}$, which we denote as
$\|\mathbf{h}_{b}\|^{2}$, is the sum of squares of $K_{h}$ Gaussian variables
with zero mean and $N_{int}\epsilon^{2}$ variance, which follows the scaled
Chi-square distribution $N_{int}\epsilon^{2}\cdot\chi_{K_{h}}^{2}$ and is
equivalent to Gamma distribution with cumulative distribution function
$\Gamma(K_{h}/2,2N_{int}\epsilon^{2})$ [39]. The expected value of
$\|\mathbf{h}_{b}\|^{2}$, according to the property of Gamma distribution, is
$N_{int}K_{h}\epsilon^{2}$, which is a monotonic increasing function of the
number of interacted item $N_{int}$, i.e., the rating density, a main
indicator for the sufficiency level of collaborative information.

####  3.4.2 Bandwidth and User Collaborative Embedding

This property reveals that $\|\mathbf{h}_{b}\|^{2}$ is positively correlated
with the sufficiency level of the collaborative information. Moreover,
compared to the L2-norm of the sparse rating vector $\mathbf{r}$, the hidden
embedding $\mathbf{h}_{b}$ lies in a latent low dimensional space where user
collaborative representations are more compact, which could better reflect the
similarity relationship of user rating patterns by eliminating redundant
information contained in similar items. Therefore, the L2-norm of
$\mathbf{h}_{b}$ contains important information regarding interaction sparsity
and interaction similarity that are suitable for the inference of the
bandwidth that we have defined in the previous section. In addition, the
L2-normalization of the embeddings eliminates the negative influence of the
difference in the number of items the users have interacted with by scaling
the hidden rating embeddings of users with different activity levels into the
same sphere space, which makes the direction of $\mathbf{h}_{b}$, more
suitable representation than the original $\mathbf{h}_{b}$ to infer the user
collaborative embedding. This is in contrast with the previous auto-encoder-
based approaches such as Mult-vae [10], CondVAE [32], where the information
regarding the sparsity level of the ratings is discarded after the
L2-normalization of the input ratings.

In practice, we calculate the bandwidth by a linear transformation of
$\|\mathbf{h}_{b}\|$ transformed with sigmoid activation function to squash
the value between [0, 1]:

$\alpha=\operatorname{sigmoid}\left(\|\mathbf{h}_{b}\|w^{\alpha}_{b}+b^{\alpha}_{b}\right).$
(3)

Since $\|\mathbf{h}_{b}\|$ is strictly non negative, in backward propagation
the weight $w^{\alpha}_{b}$ can only be updated in one direction for all the
samples in one mini-batch. This is problematic, since the training loop would
make $\alpha$ converge to zero or one where the bandwidth for all users are
identical, which fails to discriminate users with different sufficiency level
of collaborative information. This is referred to as the modality collapsing
problem in variational inference. In this article, this problem is addressed
through the batch normalization [40]. We denote a mini-batch of the length of
user hidden embedding as
$\mathcal{B}=\\{\|\mathbf{h}_{b,0}\|,\|\mathbf{h}_{b,I_{mini}-1}\|,\cdots,\|\mathbf{h}_{b,I_{mini}}\|\\}$.
Before feeding each sample in the mini-batch $\mathcal{B}$ into Eq. (3) to
calculate the bandwidth, we renormalize it by:

$\widehat{\|\mathbf{h}_{b,i}\|}\leftarrow\frac{\|\mathbf{h}_{b,i}\|-\mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^{2}+\epsilon}},$
(4)

where $\mu_{\mathcal{B}}$ and $\sigma_{\mathcal{B}}^{2}$ are the mean and
sample variance of $\|\mathbf{h}_{b}\|$ in the mini-batch, and $\epsilon$ is a
small value to avoid division by zero error. By batch normalization, we
observe that the inferred bandwidth $\alpha$ corresponds more consistently
with the sufficiency level of the user collaborative information, and the
training procedure becomes more stable as well. In the testing phase,
$\mu_{\mathcal{B}}$ and $\sigma_{\mathcal{B}}^{2}$ are fixed to their running
averages estimated from the training data.

####  3.4.3 Neural Network Implementations

The user feature module infers the feature-based latent user embedding from
the extracted user features by another MLP, which serves as the auxiliary
information source to the collaborative information. The detailed inference
process of the latent variables $Z_{b},Z_{t}$, and $D$ through the inference
network is described as follows:

(1) For each layer $l$ of the collaborative and user-feature module of the
inference network:

(a) For each column $n$ of the weight matrices, draw

$\mathbf{W}^{(l)}_{\\{b,t\\},:n}\sim\mathcal{N}\left(\mathbf{0},\lambda_{w}^{-1}\mathbf{I}_{K_{l}}\right);$

(b) Draw the bias vector from
$\mathbf{b}^{(l)}_{\\{b,t\\}}\sim\mathcal{N}\left(\mathbf{0},\lambda_{w}^{-1}\mathbf{I}_{K_{l}}\right)$;

(c) For ${\mathbf{h}}^{(l)}_{\\{b,t\\}}$ of a user $u$, draw

$\mathbf{h}^{(l)}_{\\{b,t\\}}\sim\delta\left(\operatorname{f_{act}}\left(\mathbf{h}^{(l-1)}_{\\{b,t\\}}\mathbf{W}^{(l)}_{\\{b,t\\}}+\mathbf{b}^{(l)}_{\\{b,t\\}}\right)\right).$

(2) For the user-dependent channel variable:

(a) Calculate the bandwidth from its logits, which is inferred by the L2-norm
of the hidden rating embedding $\mathbf{h}^{(L)}_{b}$ as follows:

$\alpha=\operatorname{sigmoid}\left(\|\mathbf{h}^{(L)}_{b}\|w^{\alpha}_{b}+b^{\alpha}_{b}\right);$

(b) For VBAE-hard, draw the Bernoulli channel variable:

$d\sim Bernoulli(\alpha);$

(c) For VBAE-soft, draw the Beta channel variable:

$d\sim Beta(\alpha,\sigma^{2}_{fixed}).$

(3) For the collaborative and user feature latent variable:

(a) Draw the mean and standard deviation:

$\left[\bm{\mu}_{\\{b,t\\}},\log\sigma_{\\{b,t\\}}\right]\sim\delta\left(\mathbf{h}^{(L)}_{\\{b,t\\}}\mathbf{W}^{\mu+\sigma}_{\\{b,t\\}}+\mathbf{b}^{\mu+\sigma}_{\\{b,t\\}}\right);$

(b) Draw the sample of the latent variable:

$\mathbf{z}_{\\{b,t\\}}\sim\mathcal{N}\left(\bm{\mu}_{\\{b,t\\}},\operatorname{diag}\left(\bm{\sigma}_{\\{b,t\\}}^{2}\right)\right).$

It is not trivial to draw samples from the Bernoulli or the Beta channel
variable such that gradients can be back-propagated to the trainable weights
of the inference network, and we defer the discussion of the solution, which
is called the reparameterization trick, to Section 3.7. In VBAE, $D$ is
inferred from the observed ratings via a non-linear neural network, where the
explicit uncertainty caused by sparsity can be directly captured from the
input $R$. In addition, since the collaborative embedding variables are
inferred in an amortized manner, the implicit uncertainty can be captured by
the weights, which are actively learned and shared among all users. The user
latent variable $V$ simultaneously captures the user collaborative similarity
based on user interactions and the user feature similarity from extracted user
features. The neural network implementation of VBAE is schematically
illustrated in Fig. 2.

###  3.5 Training Objective

To jointly learn the parameters of the generative network and the inference
network, we maximize the evidence lower bound (ELBO), which is an
approximation of the marginal log-likelihood of evidence $p_{\theta}(R,X)$:

$\displaystyle\mathcal{L}(q)$ $\displaystyle=\mathbb{E}_{q_{\phi}}[\log
p_{\theta}(R,X,Z_{b},Z_{t},D)-\log q_{\phi}(Z_{b},Z_{t},D|R,X)]$ (5)
$\displaystyle=\mathbb{E}_{q_{\phi}}[\log p(R|Z_{b},Z_{t},D)+\log p(X|Z_{t})]$
$\displaystyle-\mathbb{KL}\left[q(Z_{b},D|R)\|p(Z_{b},D)\right]-\mathbb{KL}\left[q(Z_{t}|X)\|p(Z_{t})\right],$

and the value of $\mathcal{L}(q)$, for a fixed $\theta$, achieves the maximum
if and only if the discrepancy between the variational approximation
$q_{\phi}$ and the true posterior $p_{\theta}$ measured by the Kullback-Leiber
(KL) divergence is zero (i.e., iif. $q_{\phi}=p_{\theta}$).

###  3.6 Maximum A Posteriori Estimation

Although the collaborative and user feature module of VBAE can be jointly
trained as Eq. (5), extra computational and memory consumption is inevitable.
In addition, in a joint training, the VBAE model may converge to an
undesirable sub-optima where it relies solely on one main information source
for the recommendation, which is undesirable since the user ratings and
features contain complementary information, both of which are important for
recommendations. Therefore, we take an EM-like optimization approach, where we
iteratively consider only one of the variational distributions in
${q_{\phi}(Z_{b},D|R)}$ and $q_{\phi}(Z_{t}|X)$ and fix the random variables
(e.g., to their means or their to previous estimates) that concern the other.
Consequently, for the collaborative part ${q_{\phi}(Z_{b},D|R)}$, we fix
$Z_{t}$ to the estimated mean to calculate $\mathbf{v}$, and the objective
becomes:

$\displaystyle\mathcal{L}_{b\\_step}^{MAP}=\mathbb{E}_{q_{\phi}(Z_{b},D|R)}[\log
p(R|V)]-\mathbb{KL}\left[q_{\phi}(D|R)\|p(D)\right]$ (6)
$\displaystyle-\mathbb{KL}\left[q_{\phi}(Z_{b}|R)\|p(Z_{b})\right]-\frac{\lambda_{w}}{2}\cdot\sum_{l}\left(\|\mathbf{W}_{b}^{(l)}\|_{F}^{2}+\|\mathbf{b}_{b}^{(l)}\|_{2}^{2}\right).$

Eq. (6) can be viewed alternatively as paying a cost equals to the KL with
prior value whenever the model tries to access the noisy user feature
information. Moreover, the cost is dynamically decided by the urgency level of
introducing the extra user feature information based on the sufficiency level
of the collaborative information, which prevents the model from depending
unnecessary upon the noisy features. After one-step optimization of
$\mathcal{L}_{b\\_step}^{MAP}$, we then fix $Z_{b}$ and $D$ to their estimated
values to calculate $\mathbf{v}$, and maximize the following objective for the
user feature part:

$\displaystyle\mathcal{L}_{t\\_step}^{MAP}=\mathbb{E}_{q_{\phi}(Z_{t}|X)}[\log
p(R|V)]+\mathbb{E}_{q_{\phi}(Z_{t}|X)}\left[\log p(X|Z_{t})\right]$ (7)
$\displaystyle-\mathbb{KL}\left[q_{\phi}(Z_{t}|X)\|p(Z_{t})\right]-\frac{\lambda_{w}}{2}\cdot\sum_{l}\left(\|\mathbf{W}_{t}^{(l)}\|_{F}^{2}+\|\mathbf{b}_{t}^{(l)}\|_{2}^{2}\right).$

Intuitively, for both $b$_step and $t$_step, the objective consists of two
parts. The first part is the expected log-likelihood term, where the inferred
hidden Gaussian embeddings and the latent channel variable are encouraged to
best explain the extracted user features and the observed historical
interactions; the second part is the KL with prior terms and the L2 weight
decay terms, which act as regularizers to prevent over-fitting and avoid
polarization of embeddings due to the sparsity of interactions [41]. Liang et
al. [10] have shown that the KL regularization in the collaborative part could
be too strong, which over-constrains the representational ability of latent
collaborative embeddings. As a solution, they introduced a scalar $\beta$ to
control the weight of the KL term for the latent collaborative variable in Eq.
(6), which has its theoretical foundation in both beta-VAE [42] and
variational information bottleneck theory [43]. We anneal the $\beta$ from 0
to 0.2 as [10] in our implementation of VBAE, and have confirmed the
effectiveness of KL annealing in our experiments. Under such settings, the
model learns to encode as much information of $\mathbf{r}$ in $\mathbf{z}_{b}$
as it can in the initial training stages while gradually regularizing $z_{b}$
by forcing it close to the prior as the training proceeds [44].

###  3.7 Monte Carlo Gradient Estimator

In this section, we derive the gradients of $\mathcal{L}_{b\\_step}^{MAP}$ and
$\mathcal{L}_{t\\_step}^{MAP}$ w.r.t. the trainable parameters of the
generation and inference networks to make them amenable to SGD optimization.
For Gaussian and Bernoulli distributions, since their KL divergence with prior
can be computed analytically, the minimization of the KL terms in Eqs. (6),
(7) w.r.t. the weights of the inference network $\phi$ can be analytically
calculated. However, since the gradients of the expected log-likelihood terms,
which we note as $\operatorname{ELM}_{\\{b,t\\}\\_step}$, need to be back-
propagated through stochastic nodes $Z_{b}$, $Z_{t}$ and $D$, it precludes us
from calculating an analytic solution. Hence, we introduce Monte Carlo methods
to form unbiased estimators of the gradients. For $\theta$, as the generative
distribution is explicit in expectation form, its gradient can be estimated by
generating samples from the encoder distribution, calculating the gradients,
and taking the average [45].

####  3.7.1 User Embeddings: Vanilla Reparameterization

As for $\phi$ that is associated with the inference of user feature and
collaborative embeddings, we use the reparameterization trick, where we
transform the stochastic nodes into differentiable bivariate functions of
their parameters and random noises to allow gradients to pass through the
distribution parameters [46]. Specifically, for the Gaussian embedding
variables $Z_{\\{b,t\\}}$, with the vanilla reparameterization trick [22],
their samples can be reformulated via:

$\mathbf{z}_{\\{b,t\\}}^{(l)}([\bm{\mu}_{\\{b,t\\}},\bm{\sigma}_{\\{b,t\\}}],\bm{\epsilon}^{(l)})=\bm{\mu}_{\\{b,t\\}}+\bm{\epsilon}^{(l)}\odot\bm{\sigma}_{\\{b,t\\}}.$
(8)

where $\bm{\epsilon}^{(l)}\sim\mathcal{N}(0,\mathbf{I}_{K})$. Eq. (8) could be
viewed alternatively as injecting Gaussian noise to the hidden user
collaborative and user feature variables, which is the main mechanism that
previous auto-encoder-based recommender systems adopt to address the rating
and feature noise problems.

####  3.7.2 Hard Channel: Gumbel-Softmax Reparameterization

Injecting Gaussian noise to user latent collaborative and feature embeddings
can only simulate the generation process of low-level noise. However, as we
have argued in the Introduction, pervasive high-level noise, which is
information that are irrelevant to the recommendation purpose, exists in the
extracted user features. Therefore, we introduce the user-dependent channel to
avoid excessive model reliance on these features for a better generalization.
For the channel in VBAE-hard that follows the Bernoulli distribution, we note
that sampling from which is equivalent to sampling a one-hot vector from a
two-class Categorical distribution with probability mass
$\bm{\alpha}=[\alpha,1-\alpha]$ and discarding the second dimension.
Therefore, we resort to the Gumbel-softmax trick [47] and reformulate samples
of the channel $D$ via the Concrete distribution [48]:

$\displaystyle d^{(l)}(\bm{\alpha},g^{(l)})$
$\displaystyle=\frac{\exp\left(\left(\log\left(\alpha_{1}\right)+g^{(l)}_{1}\right)/\tau\right)}{\sum_{i=1}^{2}\exp\left(\left(\log\left(\alpha_{i}\right)+g^{(l)}_{i})/\tau\right)\right)}$
(9)
$\displaystyle=\operatorname{sigmoid}\left(\log\left(\frac{\alpha}{1-\alpha}\right)+g^{(l)}_{1}-g^{(l)}_{2},\tau\right),$

where $g^{(l)}_{1}\perp g^{(l)}_{2}\sim Gumbel(0,1)$ and $\tau$ is the
temperature of the softmax and the sigmoid. When $\tau$ approaches zero, the
samples $d^{(l)}$ are proved to be equivalent to samples drawn from the
corresponding Bernoulli distribution with the probability $\alpha$. In
practice, $\tau$ is generally annealed as the training proceeds for a more
stable convergence.

####  3.7.3 Soft Channel: Logistic-Normal Reparameterization

Similarly, we draw the Beta channel variable of VBAE-soft by keeping the first
dimension of a sample from the corresponding two-class Dirichlet distribution.
However, unlike Gaussian and Categorical distributions, there is no consensus
regarding how to reparameterize a Dirichlet variable [49, 50, 36]. In this
article, we eschew the commonly used reparameterization strategies that
transform a uniformly distributed vector by the inverse of the Dirichlet
cumulative distribution function as the bivariate transformation, but we
derive its reparameterization with logistic-normal approximation instead [50].
The reason is that, the logistic-normal distribution converts the original
parameters $[\alpha_{1},\alpha_{2}]$ of the Dirichlet (the values that should
be predicted by the inference network) to the mean and standard deviation of a
Gaussian distribution, such that the convergence to a low-entropy area is
smoother. Otherwise, to reach a low variance area of the Dirichlet requires
large values of $[\alpha_{1},\alpha_{2}]$, which is difficult to learn by the
inference network and results in unstable training dynamics [36]. The
relationship between the parameters of logistic-normal and the corresponding
Dirichlet is formulated as follows

$\displaystyle\mu_{1}=-\mu_{2}=\frac{1}{2}\bigg{(}\log\alpha_{1}-\log\alpha_{2}\bigg{)}$
(10)
$\displaystyle\sigma_{1}=\sigma_{2}=\frac{1}{4}\bigg{(}\frac{1}{\alpha_{1}}+\frac{1}{\alpha_{2}}\bigg{)}=\delta^{\prime}_{fixed}.$

Note that we fix the logistic-normal to a small value, as here only the mean
of the Beta channel variable (i.e., the bandwidth) is important, and a small
value of the variance prevents the Dirichlet distribution from stuck into a
low-entropy area. This is also a common trick used in most regression task,
where the outputs are assumed to be Gaussian, for which the variance is
trivial to model. The sample $d^{(l)}$ from the logistic-normal is drawn
according to

$\displaystyle d^{(l)}(\bm{\alpha},\bm{\epsilon}^{(l)})$
$\displaystyle=\frac{\exp\left(\left(\mu_{1}+\sigma_{1}\cdot\epsilon_{1}^{(l)}\right)\right)}{\sum_{i=1}^{2}\exp\left(\left(\mu_{i}+\sigma_{i}\cdot\epsilon_{i}^{(l)}\right)\right)}$
(11)
$\displaystyle=\operatorname{sigmoid}\left(2\cdot\mu_{1}+\sigma_{1}\cdot(\epsilon^{(l)}_{1}-\epsilon^{(l)}_{2})\right),$

where $\epsilon^{(l)}_{1}\perp\epsilon^{(l)}_{2}\sim\mathcal{N}(0,1)$. A close
look at Eq. (11) shows that it bears great similarity to Eq. (9), since we can
view
$2\cdot\mu_{1}=log(\frac{\alpha_{1}}{\alpha_{2}})=log(\frac{\alpha}{1-\alpha})$
as pseudo logits of the bandwidth, and both Eqs. add and subtract two i.i.d.
random variables to the logits of bandwidth before squashing it into (0, 1)
with the sigmoid function. The major difference between these two equations is
that for Eq. (9), a small temperature of the sigmoid pushes the value of
$\hat{d}$ to 0 or 1, i.e., the two extremities, such that for each user the
channel is either open or closed in one iteration. However, in Eq. (9), the
temperature is set to one and therefore $\hat{d}$ can take any value between
[0, 1], which avoids swerve of gradient direction in training and smooths the
convergence.

####  3.7.4 Unbiased Gradient Estimators

With the stochastic user latent embedding variable and the channel variable
reparameterized with the strategies we introduce above, the unbiased gradient
estimator of the objective $\operatorname{ELM}_{\\{b,t\\}\\_step}$ w.r.t.
$\phi$ can be formulated as:

$\displaystyle\nabla_{\phi}\operatorname{E}_{b\\_step}\simeq$
$\displaystyle\frac{1}{L}\sum_{l}\bigg{(}\nabla_{\mathbf{z}_{b}^{(l)},\
d^{(l)}}\log
p(\mathbf{r}|\mathbf{v}^{(l)})\cdot\nabla_{\phi}{[\mathbf{z}_{b}^{(l)},\
d^{(l)}]}\bigg{)};$ (12)
$\displaystyle\nabla_{\phi}\operatorname{E}_{t\\_step}\simeq$
$\displaystyle\frac{1}{L}\sum_{l}\bigg{(}\nabla_{\mathbf{z}_{t}^{(l)}}\log
p(\mathbf{r}|\mathbf{v}^{(l)})\cdot\nabla_{\phi}\mathbf{z}_{t}^{(l)}\bigg{)}+$
$\displaystyle\frac{1}{L}\sum_{l}\bigg{(}\nabla_{\mathbf{z}_{t}^{(l)}}\log
p(\mathbf{x}|\mathbf{z}_{t}^{(l)})\cdot\nabla_{\phi}\mathbf{z}_{t}^{(l)}\bigg{)},$

where $\simeq$ is used to denote that the RHS. is an unbiased estimator of the
LHS. Since the variance of the gradient estimated by reparameterization trick
is low, previous work has demonstrated that as long as the batch size is large
enough, it suffices to take a single sample for each user for the training to
converge [46].

###  3.8 Prediction for New Users

After the weights of the generative and inference networks of the VBAE model
are learned, our discussion shifts towards how to predict new relevant items
for users given their observed ratings $\mathbf{r}_{o}$ and noisy features
$\mathbf{x}$. For a user, we first calculate the mean of the collaborative
embedding $\bm{\mu}_{b}$, the bandwidth $\alpha$ from the ratings via the
collaborative inference network, and the mean of the feature embedding
$\bm{\mu}_{t}$ from the user features via the feature inference network. The
user latent variable $\mathbf{v}$ can then be approximated as:

$\mathbb{E}\left[V|\mathbf{r}_{o},\mathbf{x}\right]\approx\mathbb{E}\left[Z_{b}|\mathbf{r}_{o}\right]+\bar{d}\cdot\mathbb{E}\left[Z_{t}|\mathbf{x}\right]=\mu_{b}+\bar{d}\cdot\mu_{t}.$
(13)

To avoid randomness of the channel in testing, for VBAE-hard, we set $\bar{d}$
to a fixed sample from $Bernoulli(\alpha)$ to determine whether or not
information in user feature embeddings $\mu_{t}$ are necessary to be
introduced to support the recommendation, whereas for VBAE-soft, we use the
mean of the Beta channel variable (approximated by logistic-normal), i.e.,
$\alpha$, as $\bar{d}$. Finally, we calculate the multinomial probabilities of
the remaining items from $\mathbf{v}$ via $\operatorname{gen}_{b}$ as:

$\mathbb{E}\left[R|\mathbf{r}_{o},\mathbf{x}\right]=\mathbb{E}\left[\operatorname{Gen}_{b}(V)|\mathbf{r}_{o},\mathbf{x}\right]\approx\operatorname{Gen}_{b}(\mathbb{E}\left[V|\mathbf{r}_{o},\mathbf{x}\right]),$
(14)

where the $\approx$ is due to the non-linearity of $\operatorname{Gen}_{b}$,
and the estimated logits of probabilities of unobserved items are sorted to
get final ranked list of items for recommendation.

##  4 Empirical Study

Figure 5: The distribution of user rating density for citeulike-a, citeulike-t, and Amazon toys & games datasets. The red curves illustrate the estimated probability density functions and the light blue dashed vertical lines shows the percentiles. (The maximum interaction value is cut off to 100, 80, 60 for the datasets for a better illustration effect) Table 1: Attributes of citeulike-a, citeulike-t and Amazon toys & games datasets after preprocessing. In the table, % interactions refers to the density of the rating matrix, max/min/avg/std #visits refer to the corresponding statistics of the number of item that the users provide implicit feedback. | citeulike-a | citeulike-t | toys & games
---|---|---|---
# users | 5,551 | 5,031 | 14,706
# items | 16,980 | 21,133 | 11,722
% interactions | 0.217% | 0.114% | 0.072%
max/min #visits | 403/10 | 1,932/5 | 546/5
avg$\pm$std #visits | $12\pm 12$ | $24\pm 58$ | $8\pm 8$
# features | 8,000 | 20,000 | 8,000

In this section, we present and analyze the extensive experiments we conducted
on three real-world datasets to demonstrate the effectiveness of the proposed
VBAE model for hybrid recommender systems.

###  4.1 Dataset

We use three real-world datasets to evaluate the model performance. Two of the
datasets, citeulike-a [51] and citeulike-t [52] are from CiteULike, where
scholars can add academic articles they are interested in to their libraries
such that new relevant articles can be automatically recommended. The third
dataset, toys & games, is collected by [53] from
Amazon333https://nijianmo.github.io/amazon/index.html. In preprocessing, we
randomly split the users by the ratio of 8:1:1 for training, validation,
testing. For each user, 80% of the interactions are selected as the observed
interactions to learn the user collaborative embedding and the bandwidth of
the channel, and the remaining 20% are hold-out for testing. The user profiles
are built from the features of their interacted items. We represent each
article in the citeulike datasets by the concatenation of its title and
abstract, and each item in toys & games by combining all of its reviews. We
then select discriminative words according to the tf-idf values and normalize
the word counts of each item over the maximum occurrences of each word in all
items. Finally, we calculate the element-wise maximum of the normalized word
counts of the observed items for each user as the user features. Table 1
summarizes the details of the datasets after preprocessing. Fig. 5 illustrates
the distributions of interaction density for different users. From Fig. 5 we
can find that the interaction density distribution clearly demonstrates a
long-tail characteristic, which reflects the uneven distribution of
sufficiency level of collaborative information for users in all three
datasets.

###  4.2 Evaluation Metrics

Two ranking-based metrics are used to evaluate the recommendation performance:
Recall@$M$ and the truncated normalized discounted cumulative gain (NDCG@$M$).
We do not use the precision metric, since the rating matrices in all three
datasets record implicit feedbacks where a zero entry does not necessarily
imply that the user is not interested in the item, but it may be due to the
fact that the user is not aware of its existence as well [33]. For a user $u$,
we first obtain the rank of the held-out items by sorting their multinomial
probabilities calculated as Eq. (14). If we denote the item at rank $r$ by
$i(r)$ and the set of hold-out items for the user by $\mathcal{I}_{u}$,
Recall@$M$ is calculated as:

$\operatorname{Recall}@M(u)=\frac{\sum_{r=1}^{M}\mathbb{I}\left[i(r)\in\mathcal{I}_{u}\right]}{\min\left(M,\left|\mathcal{I}_{u}\right|\right)},$
(15)

where $\mathbb{I}$ in the numerator is the indicator function, and the
denominator is the minimum of $M$ and the number of hold-out items. Recall@$M$
has a maximum of 1, which is achieved when all relevant items are ranked among
the top $M$ positions. Truncated discounted cumulative gain (DCG@$M$) is
computed as

$\operatorname{DCG}@M(u)=\sum_{r=1}^{M}\frac{2^{\mathbb{I}\left[i(r)\in\mathcal{I}_{u}\right]}-1}{\log(r+1)},$
(16)

which, instead of uniformly weighting all positions, introduces a logarithm
discount function over the ranks where larger weights are applied to
recommended items that appear at higher ranks [54]. NDCG@$M$ is calculated by
normalizing the DCG@$M$ to [0, 1] by the ideal DCG@$M$ where all relevant
items are ranked at the top.

###  4.3 Implementation Details

Since the datasets we consider vary both in their scale and scope, we select
the structure and the hyperparameters of VBAE based on evaluation metrics on
validation users through grid search444Due to space limit, please refer to the
JSON files we release with the codes for the searched optimal hyperparameters
and model architecture for each of the three datasets.. In VBAE, sigmoid is
used as both intermediate and output activations. The weights of the inference
network are tied to the generation network the same way as [16] to more
effectively learn representations of user features. Specifically, to avoid the
component collapsing problem where the inferred bandwidth for all users are
identical, batch normalization [40] is applied to the L2-norm of the latent
feature representations such that they have zero mean and unit variance before
the inference of the bandwidth; in addition, a larger decay rate is applied to
the weights of the dense layer for bandwidth inference for regularization. We
first layerwise pretrain the user feature network as the initial starting
point for VBAE, and then iteratively train the collaborative network
($b$_step) and the user feature network ($t$_step) for 100 epochs. Adam is
used as the optimizer with a batch size of 500 users. We randomly split the
datasets into ten train/val/test splits as described in Section 4.1. For each
split, we keep the model with the best NDCG@100 on the validation users and
report the mean metrics on the test users for all splits.

###  4.4 Comparisons with Baselines

In this section, we compare the proposed VBAE with the following state-of-the-
arts collaborative and hybrid recommendation baselines to demonstrate its
effectiveness:

Table 2: Comparisons between VBAE and various baselines. We report the metrics
averaged on ten splits of the datasets. The best method is highlighted in
bold, where the best method in each part is marked with underlines.

(a) citeulike-a | Recall@20 | Recall@40 | NDCG@100
---|---|---|---
VBAE-soft | 0.299 | 0.376 | 0.296
VBAE-hard | 0.293 | 0.373 | 0.294
FM | 0.231 | 0.312 | 0.238
CTR | 0.169 | 0.250 | 0.190
CDL | 0.209 | 0.295 | 0.226
CVAE | 0.236 | 0.334 | 0.247
Multi-VAE | 0.261 | 0.346 | 0.265
RecVAE | 0.265 | 0.354 | 0.269
CoVAE | 0.247 | 0.338 | 0.260
CondVAE | 0.274 | 0.359 | 0.275
DICER | 0.279 | 0.363 | 0.272

(b) citeulike-t | Recall@20 | Recall@40 | NDCG@100
---|---|---|---
VBAE-soft | 0.227 | 0.306 | 0.190
VBAE-hard | 0.223 | 0.308 | 0.193
FM | 0.154 | 0.224 | 0.135
CTR | 0.132 | 0.189 | 0.118
CDL | 0.200 | 0.271 | 0.168
CVAE | 0.216 | 0.294 | 0.181
Multi-VAE | 0.168 | 0.247 | 0.139
RecVAE | 0.177 | 0.251 | 0.148
CoVAE | 0.194 | 0.257 | 0.167
CondVAE | 0.215 | 0.279 | 0.172
DICER | 0.203 | 0.283 | 0.161

(c) toys & games | Recall@20 | Recall@40 | NDCG@100
---|---|---|---
VBAE-soft | 0.145 | 0.196 | 0.107
VBAE-hard | 0.144 | 0.193 | 0.105
FM | 0.088 | 0.121 | 0.062
CTR | 0.124 | 0.179 | 0.089
CDL | 0.133 | 0.181 | 0.092
CVAE | 0.139 | 0.188 | 0.094
Multi-VAE | 0.114 | 0.157 | 0.082
RecVAE | 0.110 | 0.154 | 0.077
CoVAE | 0.120 | 0.174 | 0.085
CondVAE | 0.132 | 0.180 | 0.094
DICER | 0.127 | 0.172 | 0.092

  * •

FM (Factorization Machine) is a widely employed algorithm for hybrid
recommendation with sparse inputs [55]. We use Bayesian parameter search as
suggested in [56] to find optimal hyperparameters and the loss function on the
validation users.

  * •

CTR [51] learns the topics of item content via latent Dirichlet allocation
(LDA) and couples it with probabilistic matrix factorization (PMF) for
collaborative filtering. We find the optimal hyperparameters $a$, $b$,
$\lambda_{u}$, $\lambda_{v}$ and latent dimension $K$ through grid search.

  * •

CDL [28] replaces the LDA in CTR with a stacked Bayesian denoising auto-
encoder (SDAE) [29] to learn the item content embeddings in an end-to-end
manner. We set the mask rate of SDAE to 0.3 and search its architecture the
same way as VBAE.

  * •

CVAE [16] further improves over the CDL by utilizing a VAE in place of the
Bayesian SDAE, where a self-adaptive Gaussian noise is introduced to corrupt
the latent item embeddings instead of corrupting the input features with zero
masks.

  * •

Multi-VAE [10] breaks the linear collaborative modeling ability of PMF by
using a VAE with multinomial likelihood to capture the user collaborative
information in ratings for recommendations.

  * •

CoVAE [57] utilizes the non-linear Multi-VAE as the collaborative backbone and
incorporates item feature information by treating their co-occurrences as
pseudo training samples to collectively train the Multi-VAE with the user
ratings.

  * •

CondVAE [32] builds a user conditional VAE where the user features are used as
the conditions. We extend the original CondVAE by replacing the categorical
user features with the ones we build from the interacted items, which we find
have a better performance on all the datasets.

  * •

DICER [26] is an item-oriented auto-encoder (IAE)-based recommender system
where the item content information is utilized to learn disentangled item
embeddings from their user ratings to achieve more robust recommendations.

  * •

RecVAE [58] improves over the Multi-VAE by designing a new encoder
architecture with a composite prior for user collaborative latent variables
that leads to a more stable training procedure.

Table 2(c) summarizes the comparison results between VBAE and the selected
baselines. As it can be seen, Table 2(c) comprises of three parts. The middle
part shows four hybrid baselines with linear collaborative filtering module,
i.e., matrix factorization (MF). Generally, the performance improves with the
increase of the representational ability of the utilized item embedding model.
Specifically, CVAE, which uses VAE to encode the item content information into
Gaussian variables, performs consistently better than CDL and CTR on all three
datasets. However, we also observe that simple methods such as FM can
outperform some of the deep learning-based baselines (e.g., CDL on citeulike-a
datasets) when their parameters are systematically searched with a Bayesian
optimizer [56].

The bottom part shows baselines that utilize deep neural networks (DNNs) as
the collaborative module. Multi-VAE, RecVAE can capture non-linear
similarities among users, so they improve almost consistently over the linear
hybrid baselines when the datasets are comparatively dense (e.g. the
citeulike-a dataset), even if they do not use any user or item side
information. When the datasets get sparser, however, they cannot perform on
par with the PMF-based hybrid recommenders that augments with item side
information due to lack of sufficient collaborative information. Moreover, we
find that although augmenting user ratings with item feature concurrences as
extra pseudo training samples, CoVAE does not consistently outperform Multi-
VAE on all three datasets, which could suggest that the item feature
occurrences do not necessarily imply user co-purchases. Treating user feature
embeddings as the condition for the user collaborative embeddings, CondVAE
achieves the best performance among all the non-linear UAE-based baselines on
two of the denser citeulike datasets and performs on par with CVAE on the
sparser Amazon toys & games dataset. DiCER, which is an IAE-based recommender
that we include for comparisons, shows clear merits when the dataset has a
large item-to-user ratio (e.g. the citeulike-a and citeulike-t datasets). The
reason may be that for IAE-based recommenders, the number of training samples
is proportional to the number of items, whereas the number of trainable
weights is proportional to the number of users, so a large item-to-user ratio
ensures sufficient training samples and a reasonable amount of trainable
weights to guarantee a good model generalization ability.

Simultaneously addressing the uncertainty of user ratings and noise in user
features, VBAE-soft and VBAE-hard out-performs all baselines on all three
datasets. Although the Bayesian SDAE in CDL, VAE in CVAE, or CondVAE also have
the denoising ability in that they corrupt the item features, latent item
embeddings, or latent user embeddings via masked noise or self-adaptive
Gaussian noise, the noise they address is not recommendation-oriented and is
therefore inevitably low-level. However, high-level and personalized noise
(information that is not relevant to the recommendation purpose) exists
pervasively in recommendation tasks, which cannot be addressed by these
models. In contrast, through the introduction of a user-dependent channel
variable, VBAE actively decides how much information should be accessed from
the user features based on information already contained in the ratings
through a quantum-inspired collaborative uncertainty measurement mechanism.
This ensures the personalized recommendation quality when the ratings are
sparse by incorporating sufficient user feature information while improving
the model generalization ability by avoiding unnecessary dependence on the
noisy user features when the collaborative information is sufficient.

###  4.5 Ablation Study On the User-dependent Channel

Figure 6: NDCG@100 breakdown for users with increasing levels of activity
measured by the number of items that a user has interacted in the history. The
error bar represents the standard deviation across ten random splits.

In this section, we further demonstrate the effectiveness of the established
information regulation mechanism in VBAE by answering the following two
research questions:

RQ1: How do VBAE-hard and VBAE-soft perform compared to VBAE-like models,
which, instead of explicitly considering the personalized difference of the
collaborative uncertainty and feature noise for different users, treat the
fusion of user feature as a fixed procedure for all the users.

RQ2: How well does the inferred bandwidth correspond to the sparsity of
interactions and, therefore, the scarcity of collaborative information? The
answer to this question shows the effectiveness of the proposed quantum-
inspired collaborative uncertainty measurement to distinguish users with
varied sufficiency levels of collaborative information.

####  4.5.1 Comparisons with Models with Fixed Fusion Strategies

To answer the first research question, we design the following three baseline
models as ablation studies:

  * •

DBAE-pass uses an "all-pass" channel to link the user collaborative and
feature networks, where all the information in user feature embeddings are
losslessly transferred to the corresponding user latent variables irrespective
of the individual difference in sufficiency level of the collaborative
information;

  * •

DBAE-stop uses a "stop" channel where the user feature information is entirely
blocked, and only the collaborative information is exploited to calculate the
user latent variables. The difference between DBAE-stop and Multi-VAE is that
Multi-VAE imposes the L2-normalization on the input ratings, whereas DBAE-stop
imposes it on the hidden rating embeddings to make it comparable with VBAE.

  * •

VAE-concat concatenates the user ratings and features as the inputs to the
Multi-VAE to reconstruct the ratings, instead of viewing their fusion from an
information-theoretic perspective. The fusion can be viewed as learning a
fixed weighted combination between user features and ratings.

Table 3: Comparisons between VBAEs, two VBAE-like models with deterministic
channel, and an early fusion baseline which concatenates the user ratings and
features as the inputs for recommendations.

(a) citeulike-a | R@20 | N@100 | Bandwidth | PCC
---|---|---|---|---
VBAE-soft | 0.299 | 0.296 | 0.543 $\pm$ 0.054 | -0.898
VBAE-hard | 0.293 | 0.294 | 0.812 $\pm$ 0.048 | -0.901
DBAE-stop | 0.263 | 0.269 | 0.000 $\pm$ 0.000 | N/A
DBAE-pass | 0.287 | 0.285 | 1.000 $\pm$ 0.000 | N/A
VAE-concat | 0.274 | 0.280 | N/A | N/A

(b) citeulike-t | R@20 | N@100 | Bandwidth | PCC
---|---|---|---|---
VBAE-soft | 0.227 | 0.190 | 0.546 $\pm$ 0.050 | -0.887
VBAE-hard | 0.223 | 0.193 | 0.805 $\pm$ 0.035 | -0.910
DBAE-stop | 0.170 | 0.142 | 0.000 $\pm$ 0.000 | N/A
DBAE-pass | 0.212 | 0.178 | 1.000 $\pm$ 0.000 | N/A
VAE-concat | 0.215 | 0.172 | N/A | N/A

(c) toys & games | R@20 | N@100 | Bandwidth | PCC
---|---|---|---|---
VBAE-soft | 0.145 | 0.107 | 0.560 $\pm$ 0.057 | -0.803
VBAE-hard | 0.144 | 0.105 | 0.829 $\pm$ 0.031 | -0.825
DBAE-stop | 0.119 | 0.088 | 0.000 $\pm$ 0.000 | N/A
DBAE-pass | 0.135 | 0.094 | 1.000 $\pm$ 0.000 | N/A
VAE-concat | 0.132 | 0.095 | N/A | N/A

The collaborative network structure of DBAE-pass and DBAE-stop is set to be
the same as VBAE for a fair comparison. The comparison results are listed in
Table LABEL:tab:results_channel.

As it can be seen, among the five models that we draw comparisons with, DBAE-
stop performs the worst on all the datasets. Since DBAE-stop can be viewed as
an altered version of Mult-vae [16] where the L2-normalization is applied on
the hidden representations rather than the input ratings and extra L2
penalties are imposed on the network weights, this confirms the previous
finding that hybrid recommendation methods augmented with feature information
usually perform better than collaborative-based methods when the ratings are
sparse [28, 16]. Comparatively, DBAE-pass is much harder to beat than DBAE-
stop, since the deficiency of collaborative information for a large number of
users with sparse interaction makes the auxiliary user features valuable for
personalized recommendation, even if the features are noisy. Still, two VBAE-
based methods achieve better performance on all three datasets, which
demonstrates that constraining the feature information allowed to be accessed
for users with sufficient collaborative information can indeed improve model
generalization. Although VAE-concat uses a dense layer to learn a weighted
combination of user features and ratings, it is over-parameterized and prone
to overfitting when the datasets are sparse. Moreover, VAE-concat the weights
are fixed for all users, which ignores the individual differences in the
sufficiency level of collaborative information. Therefore, we also observe
that two VBAE models outperform VAE-concat on all three datasets. The
superiority of user-dependent bandwidth to the all-pass and VAE-concat models
indicates that for users with more informative interactions (i.e., dense and
overlapped with the interactions of other users), the collaborative
information in the ratings is per se very reliable for recommendations, and
the noise introduced by the fusion of user features may outweigh the useful
information and degenerate the recommendation performance.

The explanation for the superiority of VBAE-soft over VBAE-hard could be that
VBAE-soft uses a Beta channel variable with its variance fixed to a small
value, where the feature embeddings are stably and smoothly discounted based
on the bandwidth inferred from user ratings. In contrast, the Bernoulli
channel in VBAE-hard determines whether or not to access user features with
the inferred bandwidth as the access probability, which may be coarse in
granularity and makes the training process less stable than the Beta channel
in VBAE-soft.

To further investigate the effectiveness of the user-dependent channels for
users with different activity levels, we divide the test users into quartiles
and report the NDCG@100 on each group in Fig. 6. When comparing with DBAE-
stop, we mainly focus on users with low activity levels, since for these
users, DBAE-stop accesses no information from user features while VBAE-hard
and VBAE-soft infer a large bandwidth for the channel that allows more
information to be accessed from their features. The leftmost bar group in Fig.
6 shows that VBAE-hard and VBAE-soft significantly outperform DBAE-stop on all
three datasets. The result confirms that incorporating auxiliary feature
information can alleviate the uncertainty of collaborative embeddings and
improve recommendation performance when the ratings are extremely sparse, even
if the user features are noisy. When comparing with DBAE-pass, on the other
hand, we focus on users with high activity levels. Although for these users,
VBAE-hard and VBAE-soft access less information from user features, the
rightmost bar group of Fig. 6 shows that NDCG@100 improves consistently for
these users. This indicates that for users with dense interactions, the
collaborative information in the ratings is per se very reliable for
recommendations, and the noise introduced by the fusion of user features may
outweigh the useful information and lowers the recommendation performance. The
improvement is more significant on citeulike-t and toys & games datasets.
Table 1 shows that users in these two datasets span a wider spectrum in their
activity levels, and therefore the reliability of collaborative embeddings
varies drastically for these users. In such a case, the channel can better
distinguish these users and allocate for each user a suitable budget for user
feature information when calculating the user latent variables for
recommendations.

####  4.5.2 Statistic Analysis of the Bandwidth

To answer the second research question, we calculate several statistics of the
inferred bandwidth for all the test users: its averaged value, its user
variability, and its Pearson correlation coefficient (PCC) with the rating
density, and report them in Table LABEL:tab:results_channel. Table
LABEL:tab:results_channel shows that the bandwidth inferred through the
proposed quantum-inspired collaborative uncertainty measurement tends to vary
across users with different rating sparsity levels. Moreover, the bandwidth
has an over -0.8 PCC with the density of user interactions on all the
datasets. Such results indicate that the channel in VBAE-hard and VBAE-soft
can distinguish users with different amounts of collaborative information in
their ratings and dynamically control the extra amount of information that
needs to be accessed from the user features based on the inferred bandwidth,
which more convincingly demonstrates the effectiveness of the user-dependent
channel in VBAE-hard and VBAE-soft. In addition, the average bandwidth of
VBAE-hard is significantly larger than that of VBAE-soft on all three
datasets. The reason could be that a large bandwidth for VBAE-hard helps to
maintain the stability of the Bernoulli channel in training.

###  4.6 Discussion of Broader Impacts of VBAE

Although we demonstrate the effectiveness of VBAE by its application in
recommender systems in this article, VBAE is a general framework that is
applicable to any heterogeneous information system where one information
source is comparatively reliable but could be missing, whereas another
information source is abundant but is susceptible to noise. One typical
example of such a system other than recommendations is the "audio-assisted
action recognition in the dark" task [59], which aims to detect actions in
under-illuminated videos. In the task, the visual information is the more
reliable modality for action prediction but could be missing due to bad
illumination, whereas the audio track always accompanies the video but may
contain lots of irrelevant information (e.g., background music) for the action
recognition purpose. To apply VBAE to these new tasks, the only mandatory
change required is to design a suitable per data point uncertainty measurement
of the first information source instead of the quantum-inspired measurement
proposed in this paper tailored for the user ratings, to dynamically decide
the information allowed to be accessed from the second source, so that the
model will not overfit on the noise in the second auxiliary modality.
Therefore, we speculate that VBAE could have a broader potential impact in
areas of data mining and heterogeneous information systems other than
recommendations.

##  5 Conclusions

In this article, we develop an information-driven generative model,
collaborative variational bandwidth auto-encoder (VBAE), to address
uncertainty and noise problems associated with two heterogeneous sources,
i.e., ratings and user features in recommender systems. In VBAE, we establish
an information regulation mechanism to fuse the collaborative and feature
information, where a user-dependent channel variable is introduced to
dynamically control how much information should be accessed from the user
features given the information already contained in the collaborative
embedding. The channel alleviates the uncertainty problem when the ratings are
sparse while improving the model generalization ability with respect to noisy
user features. The effectiveness of VBAE is demonstrated by extensive
experiments conducted on three real-world datasets.

## References

  * Zhang et al. [2019] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay.  Deep learning based recommender system: A survey and new perspectives.  _ACM Computing Surveys_ , 52(1):1–38, 2019.
  * Wang and Li [2014] Hao Wang and Wu-Jun Li.  Relational collaborative topic regression for recommender systems.  _IEEE Trans. Knowl. Data Eng._ , 27(5):1343–1355, 2014.
  * He et al. [2017] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  In _Proc. WWW_ , pages 173–182, 2017.
  * Su et al. [2010] Ja-Hwung Su, Wei-Jyun Huang, S Yu Philip, and Vincent S Tseng.  Efficient relevance feedback for content-based image retrieval by mining user navigation patterns.  _IEEE Trans. Knowl. Data Eng._ , 23(3):360–372, 2010.
  * Xu et al. [2013] Bin Xu, Jiajun Bu, Chun Chen, Can Wang, Deng Cai, and Xiaofei He.  Emr: A scalable graph-based ranking model for content-based image retrieval.  _IEEE Trans. Knowl. Data Eng._ , 27(1):102–114, 2013.
  * Lucas et al. [2013] Joel P Lucas, Nuno Luz, MaríA N Moreno, Ricardo Anacleto, Ana Almeida Figueiredo, and Constantino Martins.  A hybrid recommendation approach for a tourism system.  _Expert Syst. Appl._ , 40(9):3532–3550, 2013\.
  * Xu et al. [2021a] Yang Xu, Lei Zhu, Zhiyong Cheng, Jingjing Li, Zheng Zhang, and Huaxiang Zhang.  Multi-modal discrete collaborative filtering for efficient cold-start recommendation.  _IEEE Trans. Knowl. Data Eng._ , 2021a.
  * Yi et al. [2021a] Jing Yi, Yaochen Zhu, Jiayi Xie, and Zhenzhong Chen.  Cross-modal variational auto-encoder for content-based micro-video background music recommendation.  _ArXiv preprint_ , 2021a.
  * Sedhain et al. [2015] Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie.  AutoRec: Autoencoders meet collaborative filtering.  In _Proc. WWW_ , pages 111–112, 2015.
  * Liang et al. [2018] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara.  Variational autoencoders for collaborative filtering.  In _Proc. WWW_ , pages 689–698, 2018.
  * Li et al. [2015] Sheng Li, Jaya Kawale, and Yun Fu.  Deep collaborative filtering via marginalized denoising auto-encoder.  In _Proc. CIKM_ , pages 811–820, 2015.
  * Wang et al. [2016] Hao Wang, Xingjian Shi, and Dit-Yan Yeung.  Collaborative recurrent autoencoder: Recommend while learning to fill in the blanks.  In _Proc. NeurIPS_ , pages 415–423, 2016.
  * Zhu et al. [2019] Ziwei Zhu, Jianling Wang, and James Caverlee.  Improving top-k recommendation via joint collaborative autoencoders.  In _Proc. WWW_ , pages 3483–3482, 2019.
  * Zhang et al. [2017] Shuai Zhang, Lina Yao, and Xiwei Xu.  AutoSVD++: An efficient hybrid collaborative filtering model via contractive auto-encoders.  In _Proc. SIGIR_ , pages 957–960, 2017.
  * Wu et al. [2016] Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester.  Collaborative denoising auto-encoders for top-N recommender systems.  In _Proc. WSDM_ , pages 153–162, 2016.
  * Li and She [2017] Xiaopeng Li and James She.  Collaborative variational autoencoder for recommender systems.  In _Proc. KDD_ , pages 305–314, 2017.
  * Ma et al. [2019] Jianxin Ma, Chang Zhou, Peng Cui, Hongxia Yang, and Wenwu Zhu.  Learning disentangled representations for recommendation.  In _Proc. NeurIPS_ , pages 5711–5722, 2019.
  * Ma et al. [2020] Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu.  Disentangled self-supervision in sequential recommenders.  In _Proc. KDD_ , pages 483–491, 2020.
  * Hou et al. [2019] Yunfeng Hou, Ning Yang, Yi Wu, and S Yu Philip.  Explainable recommendation with fusion of aspect information.  _World Wide Web_ , 22(1):221–240, 2019.
  * Yi et al. [2021b] Qiaomin Yi, Ning Yang, and Philip Yu.  Dual adversarial variational embedding for robust recommendation.  _IEEE Trans. Knowl. Data Eng._ , 2021b.
  * Slaney [2011] Malcolm Slaney.  Web-scale multimedia analysis: Does content matter?  _IEEE MultiMedia_ , 18(2):12–15, 2011.
  * Kingma and Welling [2014] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In _Proc. ICLR_ , 2014.
  * Baldi [2012] Pierre Baldi.  Autoencoders, unsupervised learning, and deep architectures.  In _Proc. ICML workshop_ , pages 37–49, 2012.
  * Lee et al. [2017] Wonsung Lee, Kyungwoo Song, and Il-Chul Moon.  Augmented variational autoencoders for collaborative filtering with auxiliary information.  In _Proceedings CIKM_ , pages 1139–1148, 2017.
  * Sachdeva et al. [2019] Noveen Sachdeva, Giuseppe Manco, Ettore Ritacco, and Vikram Pudi.  Sequential variational autoencoders for collaborative filtering.  In _Proc. WSDM_ , pages 600–608, 2019.
  * Zhang et al. [2020] Yin Zhang, Ziwei Zhu, Yun He, and James Caverlee.  Content-collaborative disentanglement representation learning for enhanced recommendation.  In _Proc. RecSys_ , pages 43–52, 2020.
  * Mnih and Salakhutdinov [2007] Andriy Mnih and Russ R Salakhutdinov.  Probabilistic matrix factorization.  In _Proc. NeurIPS_ , pages 1257–1264, 2007.
  * Wang et al. [2015] Hao Wang, Naiyan Wang, and Dit-Yan Yeung.  Collaborative deep learning for recommender systems.  In _Proc. KDD_ , pages 1235–1244, 2015.
  * Vincent et al. [2010] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol.  Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.  _J. Mach. Learn. Res._ , 11(Dec):3371–3408, 2010\.
  * Zhou et al. [2020] Jin Peng Zhou, Zhaoyue Cheng, Felipe Pérez, and Maksims Volkovs.  TAFA: two-headed attention fused autoencoder for context-aware recommendations.  In _Proc. RecSys_ , pages 338–347, 2020.
  * Koren [2008] Yehuda Koren.  Factorization meets the neighborhood: a multifaceted collaborative filtering model.  In _Proc. KDD_ , pages 426–434, 2008.
  * Pang et al. [2019] Bo Pang, Min Yang, and Chongjun Wang.  A novel top-N recommendation approach based on conditional variational auto-encoder.  In _Proc. PAKDD_ , pages 357–368, 2019.
  * Hu et al. [2008] Yifan Hu, Yehuda Koren, and Chris Volinsky.  Collaborative filtering for implicit feedback datasets.  In _Proc. ICDM_ , pages 263–272, 2008.
  * Dong et al. [2017] Xin Dong, Lei Yu, Zhonghuo Wu, Yuxia Sun, Lingfeng Yuan, and Fangxi Zhang.  A hybrid collaborative filtering model with deep structure for recommender systems.  In _Proc. AAAI_ , pages 1309–1315, 2017.
  * Goyal et al. [2020] Anirudh Goyal, Yoshua Bengio, Matthew Botvinick, and Sergey Levine.  The variational bandwidth bottleneck: Stochastic evaluation on an information budget.  In _Proc. ICLR_ , 2020.
  * Deng et al. [2018] Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush.  Latent alignment and variational attention.  In _Proc. NeurIPS_ , pages 9712–9724, 2018.
  * Blei et al. [2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American Statistical Association_ , 112(518):859–877, 2017.
  * Li et al. [2019] Qiuchi Li, Benyou Wang, and Massimo Melucci.  CNM: An interpretable complex-valued network for matching.  In _Proc. NAACL_ , pages 4139–4148, 2019.
  * Johnson et al. [2002] Richard Arnold Johnson, Dean W Wichern, et al.  Multivariate linear regression models.  In _Applied Multivariate Statistical Analysis_ , chapter 7, pages 360–417. 2002.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _Proc. ICML_ , pages 448–456, 2015.
  * Park et al. [2020] Dongmin Park, Hwanjun Song, Minseok Kim, and Jae-Gil Lee.  TRAP: Two-level regularized autoencoder-based embedding for power-law distributed data.  In _Proc. WWW_ , pages 1615–1624, 2020.
  * Higgins et al. [2017] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.  Beta-VAE: Learning basic visual concepts with a constrained variational framework.  In _Proc. ICLR_ , 2017.
  * Alemi et al. [2017] Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy.  Deep variational information bottleneck.  In _Proc. ICLR_ , 2017.
  * Bowman et al. [2016] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio.  Generating sentences from a continuous space.  In _Proc. CoNLL_ , pages 10–21, 2016.
  * Rezende et al. [2014] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In _Proc. ICML_ , volume 32, pages 1278–1286, 2014.
  * Gal [2016] Yarin Gal.  _Uncertainty in deep learning_.  PhD thesis, University of Cambridge, 2016.
  * Jang et al. [2017] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In _Proc. ICLR_ , 2017.
  * Maddison et al. [2017] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  In _Proc. ICLR_ , 2017.
  * Joo et al. [2020] Weonyoung Joo, Wonsung Lee, Sungrae Park, and Il-Chul Moon.  Dirichlet variational autoencoder.  _Pattern Recognit._ , 107:107514, 2020.
  * Srivastava and Sutton [2017] Akash Srivastava and Charles Sutton.  Autoencoding variational inference for topic models.  In _Proc. ICLR_ , 2017.
  * Wang and Blei [2011] Chong Wang and David M Blei.  Collaborative topic modeling for recommending scientific articles.  In _Proc. KDD_ , pages 448–456, 2011.
  * Wang et al. [2013a] Hao Wang, Binyi Chen, and Wu-Jun Li.  Collaborative topic regression with social regularization for tag recommendation.  In _Proc. IJCAI_ , pages 2719–2725, 2013a.
  * He and McAuley [2016] Ruining He and Julian McAuley.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _Proc. WWW_ , pages 507–517, 2016.
  * Wang et al. [2013b] Yining Wang, Liwei Wang, Yuanzhi Li, Di He, Wei Chen, and Tie-Yan Liu.  A theoretical analysis of NDCG ranking measures.  In _Proc. CoLT_ , pages 1–30, 2013b.
  * Guo et al. [2017] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He.  DeepFM: A factorization-machine based neural network for ctr prediction.  In _Proc. IJCAI_ , pages 1725–1731, 2017.
  * Dacrema et al. [2019] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach.  Are we really making much progress? a worrying analysis of recent neural recommendation approaches.  In _Proc. RecSys_ , pages 101–109, 2019.
  * Chen and de Rijke [2018] Yifan Chen and Maarten de Rijke.  A collective variational autoencoder for top-n recommendation with side information.  In _Proc. WDLRS_ , pages 3–9, 2018.
  * Shenbin et al. [2020] Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I Nikolenko.  RecVAE: A new variational autoencoder for top-N recommendations with implicit feedback.  In _Proc. WSDM_ , pages 528–536, 2020.
  * Xu et al. [2021b] Yuecong Xu, Jianfei Yang, Haozhi Cao, Jianxiong Yin, and Simon See.  Arid: A new dataset for recognizing action in the dark.  In _IJCAI Workshop_ , volume 1370, page 70. Springer Nature, 2021b.