11institutetext: Faculty of Information Technology, Czech Technical University
in Prague
Prague, Czech Republic
11email: {yorshula<EMAIL_ADDRESS>
# Linear Self-Attention Approximation via Trainable Feedforward Kernel

Uladzislau Yorsh  Alexander Kovalenko  0000-0002-7194-1874

Restrictive limitation of Transformers[18] due to the quadratic complexity of
self-attention mechanism motivated a new research field of efficient
Transformers[17], which approximate the original architecture with
asymptotically faster models.

Despite the fact that Transformers are pervasive, unbiased and able to
virtually handle arbitrarily long dependencies, the quadratic space and time
complexity limit Transformer applications on long sequences. In this
connection, various findings on approximating the attention with
asymptotically faster modules have been made in order to tackle longer
sequences. However, given the absence of a unified and systematic benchmark,
overall evaluating remained uncertain until Tay et al. [16] published the
benchmark for efficient Transformer models called ”Long Range Arena”, that
consists of task of various data types.

In pursue of the faster computation, Efficient Transformers demonstrate an
impressive variety of approaches—models attaining sub-quadratic attention
complexity can utilize a notion of sparsity [15, 3, 2] or a low-rank
approximation of inputs [19, 20] to reduce an amount of attended keys; another
ways to reduce complexity include locality-sensitive hashing[12], key
pooling[21], additional memory to store information in compacted form[7, 14]
or hybridization with another architectures, such as CNNs[8, 1].

Often based on strong mathematical basis, kernelized approaches allow to
approximate an attention with linear complexity while retaining high accuracy.
The work by Katharopoulos et al. [11] describes an approximation consisting of
computing an attention by a dot product of projected queries and keys.
Consequently, the work by Choromanski et al.k[4] demonstrated that such an
approximation can be arbitrarily precise and mathematically robust, while the
work by Chowdhury et al. [5] reported that the projection can be learned.
Therefore, in the present paper we aim to expand the idea of trainable kernel
methods to approximate self-attention mechanism of the Transformer
architecture.

Our contribution: given that feedforward neural network with at least one
hidden layer [6], arbitrary non-linearity, and arbitrary number of neurons is
able to approximate any well-behaved function to any accuracy that gives
feedforward neural network the potential of being universal approximator [9].
Therefore, we propose that trainable kernel function $\phi(\cdot)$ can
approximate traditional softmax attention efficiently. Therefore, we study the
possibility of using the feedforward neural network to represent
$\phi(\cdot)$. We experiment with the architecture of $\phi(\cdot)$ and test
its performance on the three Long Range Arena tasks—text classification,
document matching and ListOps, following the instruction on limitation [16]
the number of trainable parameters in the model to provide comparable metrics.

Kernelized Model. Kernelized models are based on the following factorization
of an attention operation:

$\text{Att}(q_{i},K,V)=\sum_{j=1}^{L}\frac{\kappa(q_{i},k_{j})}{\sum_{j^{\prime}=1}^{L}\kappa(q_{i},k_{j^{\prime}})}v_{j}\approx\frac{\phi(q_{i})^{T}\sum_{j=1}^{L}\phi(k_{j})v_{j}^{T}}{\phi(q_{i})^{T}\sum_{j=1}^{L}\phi(k_{j})}$

where $q_{i}$ is a query token, $K$ and $V$ are key and value matrices,
$\kappa(q,k)$ is a kernel function to model ($\exp{(q^{T}k)}$ for a base
Transformer) and $\phi(\cdot)$ is a projection function we approximate. The
$\phi(\cdot)$ is required to be positive to maintain numeric stability, and
can vary from simple functions like ELU + 1 to stochastic softmax kernel
approximations. In our work, instead of approximation strategies with strong
priors we employ a general function such as feedforward NN.

Feedforward Kernel. We start with a single-layer FFN, defined as:

$\phi(\mathbf{X})=Softplus(\mathbf{X}\mathbf{W})$

where $W\in R^{n\times n}$ is the layer weight matrix. Surprisingly, this
model already shows the notable performance gain over the Performer and comes
close to the leader of the original LRA paper. Following the [10], we can
boost performance by forcing the orthogonality via orthogonal initialization
and regularization.

We also tried to stack more layers, but observed no performance gain – with or
without normalization layers in between. We tried the GELU and logistic
sigmoid non-linearities.

GLU Kernel. Gated Linear Units are defined as:

$GLU(\mathbf{X})=\mathbf{X}\mathbf{W}_{f}\odot\sigma(\mathbf{X}\mathbf{W}_{g})$

where $\sigma(\cdot)$ is a logistic sigmoid and $W_{f}$, $W_{g}$ are weight
matrices. This layer provides and element-wise nonlinearity and may represent
more complex functions, but requires a doubled parametrization compared to a
linear one.

For the purposes of our model, we need to modify the last GLU to force the
positive output:

$GLU_{output}(\mathbf{X})=Softplus(\mathbf{X}\mathbf{W}_{f})\odot\sigma(\mathbf{X}\mathbf{W}_{g})$

We also force the orthogonality of $W_{f}$ in these units in the same way as
in the previous subsection. We refer this regularized units as
O(rthogonal)GLU.

To mitigate the parametrization growth we apply transforms head-wise, and
suggest that gating does not require that amount of information as the input
transform. Thus, we can approximate the $W_{g}$ with, say, two low-rank
matrices of sizes $n\times r$ resp. $r\times n$ where $r<\frac{n}{2}$ is the
$W_{g}$ approximation rank. We refer this unit as A(pproximated)OGLU.

Gating. Compared to the orthogonal single-layer FFN, a single-layer OGLU model
converges even faster and shows significantly less score variance between
runs. These units can also be sequentially stacked with a benefit, up to some
extent. On the other hand, the doubled parametrization will not allow stacking
more than two units without going beyond the 10% of additional parameters.

By approximating the gating weight matrix, we are able to stack more units –
but according to the Table 1, this brings no advantage with the higher
computational costs. We used the matrices of rank $r=n/4$ to approximate the
gate, reducing the layer parametrization by 25%.

Experiments. Following the recommendations from [16], we replicate the
learning schedule and all the hyperparameters that relate to our model, while
keeping additional parametrization below 10% Due to the limitation in
computational power, we restrict ourselves only to the three LRA tasks—BPE
text classification, BPE text matching and ListOps, with input lengths
$4K/4K/2K$ respectively. To provide comparable and reproducible and results,
we used the gradient accumulation in order to simulate larger batch sizes.
Each model was trained five times to observe model behavior and to avoid so-
called black swans—random seeds that give radically different results [13].
Mean and best results are reported in Table 1

Model | Complexity | Classif. | Matching | ListOps
---|---|---|---|---
Transformer | $\mathcal{O}(L^{2})$ | 64.27 | 57.46 | 36.37
Linear kernel† | $\mathcal{O}(CL)$ | 65.77 | 73.51 | 18.54
$1\times$ GLU | $\mathcal{O}(CL)$ | 65.82 | 72.17 | 18.67
$2\times$ GLU | $\mathcal{O}(CL)$ | 65.99 | 73.36 | 18.42
$3\times$ GLU | $\mathcal{O}(CL)$ | 65.87 | 72.60 | 18.68
Orth. linear kernel | $\mathcal{O}(CL)$ | 65.86 | 72.63 | 18.19
$1\times$ OGLU | $\mathcal{O}(CL)$ | 65.95 | 72.50 | 18.45
$2\times$ OGLU | $\mathcal{O}(CL)$ | 66.02 | 72.96 | 18.32
$3\times$ AOGLU | $\mathcal{O}(CL)$ | 66.06 | 72.57 | 18.45
Table 1: Results of our models on the chosen LRA tasks, mean results for five
runs. We denote by † models that show significant variance in results.

## Acknowledgment

This research is supported by the Czech Ministry of Education, Youth and
Sports from the Czech Operational Programme Research, Development, and
Education, under grant agreement No. CZ.02.1.01/0.0/0.0/15003/0000421.

## References

  * [1] Bello, I., Zoph, B., Vaswani, A., Shlens, J., Le, Q.V.: Attention augmented convolutional networks (2020)
  * [2] Beltagy, I., Peters, M.E., Cohan, A.: Longformer: The long-document transformer (2020)
  * [3] Child, R., Gray, S., Radford, A., Sutskever, I.: Generating long sequences with sparse transformers (2019)
  * [4] Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., Belanger, D., Colwell, L., Weller, A.: Rethinking attention with performers (2021)
  * [5] Chowdhury, S.P., Solomou, A., Dubey, A., Sachan, M.: On learning the transformer kernel (2021)
  * [6] Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathematics of control, signals and systems 2(4), 303–314 (1989)
  * [7] Dai, Z., Yang, Z., Yang, Y., Carbonell, J.G., Le, Q.V., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. CoRR abs/1901.02860 (2019), http://arxiv.org/abs/1901.02860
  * [8] Gulati, A., Qin, J., Chiu, C.C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., Pang, R.: Conformer: Convolution-augmented transformer for speech recognition (2020)
  * [9] Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural networks 4(2), 251–257 (1991)
  * [10] Jia, K., Li, S., Wen, Y., Liu, T., Tao, D.: Orthogonal deep neural networks (2019)
  * [11] Katharopoulos, A., Vyas, A., Pappas, N., Fleuret, F.: Transformers are rnns: Fast autoregressive transformers with linear attention (2020)
  * [12] Kitaev, N., Łukasz Kaiser, Levskaya, A.: Reformer: The efficient transformer (2020)
  * [13] Picard, D.: Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision (2021)
  * [14] Rae, J.W., Potapenko, A., Jayakumar, S.M., Lillicrap, T.P.: Compressive transformers for long-range sequence modelling (2019)
  * [15] Roy, A., Saffar, M., Vaswani, A., Grangier, D.: Efficient content-based sparse attention with routing transformers (2020)
  * [16] Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., Metzler, D.: Long range arena: A benchmark for efficient transformers (2020)
  * [17] Tay, Y., Dehghani, M., Bahri, D., Metzler, D.: Efficient transformers: A survey (2020)
  * [18] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need (2017)
  * [19] Wang, S., Li, B.Z., Khabsa, M., Fang, H., Ma, H.: Linformer: Self-attention with linear complexity. CoRR abs/2006.04768 (2020), https://arxiv.org/abs/2006.04768
  * [20] Xiong, Y., Zeng, Z., Chakraborty, R., Tan, M., Fung, G., Li, Y., Singh, V.: Nyströmformer: A nyström-based algorithm for approximating self-attention. CoRR abs/2102.03902 (2021), https://arxiv.org/abs/2102.03902
  * [21] Zhang, H., Gong, Y., Shen, Y., Li, W., Lv, J., Duan, N., Chen, W.: Poolingformer: Long document modeling with pooling attention. CoRR abs/2105.04371 (2021), https://arxiv.org/abs/2105.04371


# An Interpretable Alternative to Neural Representation Learning for Rating
Prediction - Transparent Latent Class Modeling of User Reviews

Giuseppe Serra* *Work done when affiliated with NEC Labs Europe and with the
University of Birmingham. Goethe University Frankfurt
Germany
<EMAIL_ADDRESS>Peter Tiňo  University of Birmingham
United Kingdom
<EMAIL_ADDRESS>Zhao Xu  NEC Labs Europe
Germany
<EMAIL_ADDRESS>Xin Yao  SUSTech
China
<EMAIL_ADDRESS>
###### Abstract

Nowadays, neural network (NN) and deep learning (DL) techniques are widely
adopted in many applications, including recommender systems. Given the sparse
and stochastic nature of collaborative filtering (CF) data, recent works have
critically analyzed the effective improvement of neural-based approaches
compared to simpler and often transparent algorithms for recommendation.
Previous results showed that NN and DL models can be outperformed by
traditional algorithms in many tasks. Moreover, given the largely black-box
nature of neural-based methods, interpretable results are not naturally
obtained. Following on this debate, we first present a transparent
probabilistic model that topologically organizes user and product latent
classes based on the review information. In contrast to popular neural
techniques for representation learning, we readily obtain a statistical,
visualization-friendly tool that can be easily inspected to understand user
and product characteristics from a textual-based perspective. Then, given the
limitations of common embedding techniques, we investigate the possibility of
using the estimated interpretable quantities as model input for a rating
prediction task. To contribute to the recent debates, we evaluate our results
in terms of both capacity for interpretability and predictive performances in
comparison with popular text-based neural approaches. The results demonstrate
that the proposed latent class representations can yield competitive
predictive performances, compared to popular, but difficult-to-interpret
approaches.

##  I Introduction

In recent years, with the advent of more powerful and capable machines,
researchers have started focusing more and more on developing (deep) neural
architectures for a wide range of applications. The success of neural-based
approaches in different domains, as language modeling [1, 2] or computer
vision [3, 4], led these models to also dominate the recommender systems
research area [5, 6, 7]. However, recent works have raised some concerns about
the relative performance improvements of deep learning approaches compared to
simpler algorithms for recommendation tasks [8, 9, 10]. Indeed, as shown in
previous works [11, 12], new recommendation methods do not significantly
outperform existing approaches or even can be outperformed by very simple
methods, e.g. nearest-neighbor-based techniques [13]. Previous investigations
in this direction were mainly focused on pure collaborative filtering (CF)
data, where the only available input is the rating matrix. Nevertheless,
recent studies in this area tend to conclude that numerical rating data are
not informative enough for discovering user preferences. Consequently, given
the availability of large collections of textual data, such as product reviews
or social media posts, many approaches have tried to extend and improve
recommendation models by leveraging such textual information [14, 15, 16, 17,
18].

In fact, corpora of textual documents contain a wealth of information. They
can be used to improve the predictive performances of recommendation systems,
but more importantly, they provide human understandable explanation about user
preference and product properties. To integrate the extra textual information
into recommendation systems, most existing works employ embedding methods,
i.e., representing items, words, and documents as vectors (also known as
embedding) for better flexibility. Although the embedding-based methods often
provide good predictions, the resulting embeddings are usually not
explainable; if singularly evaluated, a 100D or 200D vectors can be hard to
comprehend by humans.

Inspired by recent discussions about the “neural hype” [11], in this paper we
investigate whether using a simpler, more transparent and principled way to
learn user and product latent representations can lead to comparable results
in the recommendation task, i.e. review-based rating prediction. We present a
probabilistic framework for the topographic organization of review data. In
contrast with previous neural-based works, we impose a double two-dimensional
topological organization of user and product latent classes based on the
textual information. As a result, the latent classes of users and products are
organized on two different square grids that reflect the textual input space.
The grid organization makes the investigation and interpretation of the
results fast and intuitive. Additionally, the probabilistic assumptions of our
system enable us to analyze the extracted information in a statistical manner.

With the interpretable topographically organized latent space representations
obtained in our probabilistic framework, one can naturally solve many
downstream learning tasks, such as product rating prediction. Motivated by the
strong correlation between reviews and ratings, we believe that exploiting our
simple, yet meaningful representations of textual review information will lead
to competitive rating prediction results, while maintaining the explainability
of the latent classes. We compare our results with state-of-the-art
approaches, including both neural network (NN) based methods and non-NN ones.
The main goal of this work is not to develop a new approach that outperforms
the state-of-the-art with respect to a single predictive score (e.g. accuracy
or AUC), but rather to propose a principled and interpretable method that
focuses on learning a transparent latent structure of the review data, while
having competitive rating prediction performances. Finally, inspired by the
recent debates on the phantom progress [10] of NN-based methods for
recommendation tasks, we compare our findings in terms of both
interpretability and predictive performances with respect to the most popular
text-based neural approaches for rating prediction.

In summary, the main contributions of this paper are threefold:

  1. 1.

We present a topographic organization of user and product latent classes based
on the latent structure of the review data. Common embedding techniques model
each item by employing dense and complicated representations. In our case, we
model classes of users and products resulting on latent vectors that are
compact, interpretable and rather discrete. Also, different from existing
works where the analysis of the user and product characteristics from the
textual perspective is practically ignored, we propose a more complete
investigation of the available information.

  2. 2.

In contrast with previous works, the probabilistic assumptions of the
framework allow us to explicitly impose through model formulation the
interpretability of the latent codes. Differently from neural-based models
where explanations are typically generated through a modification of a
difficult-to-interpret network (e.g., by integrating an attention mechanism),
in our case, interpretability is at the core of the proposed approach. Rather
than implementing an architecture capable of creating explanations without a
true understanding of the data, we present a probabilistic model grounded on
data understanding which, in turn, is inherently interpretable. In addition,
the topographic organization can help us to understand the relationships among
different latent classes. Indeed, classes close to each other in the
topographic maps should exhibit similar patterns.

  3. 3.

Motivated by the strong correlation between reviews and ratings, we exploit
our representations of the review information as input for a rating prediction
task. We contribute to the debate about the effective performances of neural-
based approaches in comparison with more principled and simpler methods.
Differently from previous investigations, we present a comprehensive
comparison mainly focused on text-based approaches for rating prediction,
considering both interpretability capacity and predictive performances.

The rest of the paper is organized as follows. We start off with a brief
review of related works. Afterwards we describe the proposed model. Before
concluding, we present the experimental results on multiple data sets. In the
experimental section, we perform both quantitative and qualitative evaluations
of the proposed method.

##  II Related Works

There are two lines of research related to the work.

###  II-A Topographic Organization in Latent Models

Kohonen’s seminal work on self-organizing maps (SOM) [19] was introduced in
the 1980s. Given the ability to produce low-dimensional representations of
high-dimensional data while providing a good approximation of the input space,
many extensions and advancements have been proposed in later years. Generative
topographic mapping (GTM) [20] is one of the most popular probabilistic
alternatives to SOM. GTM provides a generative extension of the SOM, assuming
a specific discretised non-Gaussian latent prior. Applications and extensions
of SOM span many different domains. For example, in [21], the author proposed
a data-driven, statistical approach to visualize large collections of text
documents using two-dimensional maps. In CF applications, [22] introduced a
topographic organization of latent classes for rating preferences. Differently
from their work, where the user preferences are organized using the numerical
information, in this paper, we propose to induce a topographic organization of
both user and product latent classes exploiting the associated textual
information. As a result, we obtain two separate grids (one for users and one
for products) reflecting the word patterns of the data. As reported in [21],
the most nuanced and sophisticated medium to express our feelings is our
language. We believe that it is important to understand and organize the
review information in a structured and intuitive way.

###  II-B Text-based Recommendation Models

Despite the statistical foundation and the nice visualization capabilities of
the previous methods, with the advent of more powerful and capable machines,
researchers started focusing more and more on developing deep neural
architectures for recommendation. More generally, for this task, one of the
most popular approaches is matrix factorization (MF) and, specifically,
Singular Value Decomposition (SVD). This method maps users and items into a
latent factor space and computes the rating as a dot product between the user
and the product embeddings. Although such approaches are effective and simple,
the results are poorly interpretable. Indeed, the embeddings of users and
items are not explainable and, not knowing what each feature means, it is
impossible to unveil user preferences and product characteristics [23].

Given the availability of large collections of product reviews, researchers
have recently extended latent factor models to leverage the textual
information for improving rating prediction performances. In fact, recent
studies in this area tend to conclude that the numerical rating information is
not powerful enough for discovering user preferences. One of the first
attempts that demonstrates the usefulness of leveraging features extracted
from reviews to improve the rating prediction accuracy was presented in [24].
Among other popular works in this direction, Hidden Factors as Topics (HFT)
[14] learns topics using a Latent Dirichlet Allocation (LDA)-like model for
each item and an MF for ratings. Ratings Meets Reviews (RMR) [25] uses the
same LDA model for modeling the textual information, but it uses Gaussian
mixtures for the rating prediction part. Similarly, TopicMF [26] learns topics
from each review. In [27], instead of using an approach based on LDA, the
explored methods are neural network-based. Most of the existing works combine
two learning objectives; one (unsupervised) for the textual information, and
one (supervised) for rating prediction. The unsupervised loss acts as a
regularization term for the rating prediction loss while taking advantage of
the review data. Consequently, the vector representations of the reviews are
learned to work well for rating prediction tasks while preventing overfitting.

Another category of models uses deep learning approaches for learning latent
representations of users and items. These methods mainly differ in the neural
architecture they use. In [23], the authors propose an attention-based CNN
(Attn+CNN) to build vector representations of users and products. In [28] they
propose a hierarchical Bayesian model called Collaborative Deep Learning
(CDL), which jointly performs deep representation learning and CF for the
rating matrix. In [18], the model learns vector representations of users,
items, and reviews. The review embedding is learned as a translation in the
vector space between the user and the product embeddings.

Even though many works have been published in this direction, as pointed out
in [8] and [9], it is debatable whether deep-learning-based models are really
making progress in this research area. Additionally, the majority of the
existing works deal with embedding, but the resulting vector representations
usually do not reflect any visualization-driven assumption of the data, making
the output poorly explainable. Indeed, if singularly evaluated, common
embeddings are not informative. Also, since the ultimate goal is to improve
the rating prediction part, the textual information is solely used as an
additional source to achieve this goal. Consequently, the latent information
contained in the textual data is not fully exploited and investigated.
Moreover, there have been related methods in the direction of explainable
recommendations. These works address the problem of generating explanations
through knowledge graph reasoning [29], neural attentive models [30, 31, 32],
attraction modeling [33] or substitute recommendation systems [34]. However,
they do not aim to explicitly generate interpretable vector representations.
Instead, the main objective is to generate user-specific explainable
recommendations using the complete textual information and highlighting words
that are important to explain item-specific ratings. Differently, in our work,
we propose a framework for deeper understanding of the data, presenting a two-
step approach that starts from the interpretable organization of the textual
information to arrive at the rating prediction task.

##  III Proposed Framework

In this section, we present the key ingredients of our framework. Based on our
observation that analysis of the textual latent patterns is often not fully
covered, we propose an interpretable review-based probabilistic model for
rating prediction. First, we describe the estimation of the model parameters.
Then, to make our model able to integrate new user data after training, we
propose an out-of-sample extension allowing us to compute the latent class
assignments of new users that reviewed products available in our data set.
Finally, we present how to exploit the estimated latent space representations
as model inputs for a rating prediction task.

###  III-A Topological Organization of the Latent Model

Consider a collection of users ${\cal U}=\\{u_{1},u_{2},...,u_{N}\\}$,
products ${\cal P}=\\{p_{1},p_{2},...,p_{M}\\}$ and words ${\cal
V}=\\{w_{1},w_{2},...,w_{V}\\}$. The data $\cal D$ is a collection of $R$
triples ${\cal D}=\\{(u^{i},p^{i},r^{i})\\}_{i=1}^{R}$, each triple
identifying the user $u^{i}\in{\cal U}$ writing a review $r^{i}$ on product
$p^{i}\in{\cal P}$. The review $r^{i}$ is a multi-set of words from ${\cal
V}$, $r^{i}=(w^{i}_{1},w^{i}_{2},...,w^{i}_{S_{i}})$, $w^{i}_{j}\in{\cal V}$.
The latent variables $\mathbf{z}_{u}\in\\{1,\dots,K\\}$ and
$\mathbf{z}_{p}\in\\{1,\dots,L\\}$ represent abstract classes of users and
products.

Given a review $i$, the probability of sampling a word $w_{j}\in r^{i}$ is
modeled as:

$\displaystyle P(w_{j}^{i}|u^{i},p^{i})$ $\displaystyle=$
$\displaystyle\sum_{k=1}^{K}\sum_{\ell=1}^{L}\bigg{(}P(w_{j}^{i}|\mathbf{z}_{u}=k,\mathbf{z}_{p}=\ell)\big{.}$
(1)
$\displaystyle\big{.}P(\mathbf{z}_{u}=k|u^{i})P(\mathbf{z}_{p}=\ell|p^{i})\bigg{)}.$

We impose a grid topology on latent classes via the channel noise methodology
[21, 22]. Let’s assume $\mathbf{y}$ and $\mathbf{z}$ are two different latent
classes in our grid. The channel noise for both the product and user latent
class grids is defined using the neighborhood function:

$P(\mathbf{y}|\mathbf{z})=\frac{\text{exp}\left(\frac{-\|\mathbf{z}-\mathbf{y}\|^{2}}{2\sigma^{2}}\right)}{\sum_{\mathbf{y}^{\prime}}\text{exp}\left(\frac{-\|\mathbf{z}-\mathbf{y}^{\prime}\|^{2}}{2\sigma^{2}}\right)}$
(2)

where $\sigma>0$ controls the ’concentration’ of the transition probabilities
among the neighbors of the latent class $\mathbf{y}$.111Since the channel
noise formulation is the same for both the user and product latent grids we do
not use subscripts $u$ or $p$ when referring to the latent class. Overall,
when $\mathbf{y}$ and $\mathbf{z}$ are close to each other on the grid, the
probability of being corrupted one into another is higher than when they are
distant. Additionally, when $\sigma$ is close to 0, then the transition
probabilities are more concentrated around $\mathbf{y}$ than for larger values
of $\sigma$.

For each user $u\in\mathcal{U}$, the generative process is as follows:

  1. 1.

the latent class assignment $\mathbf{z}_{u}=k$ is randomly sampled from the
user-conditional probability distribution $P(\cdot|u)$ on $\mathbf{z}_{u}$;

  2. 2.

the latent class assignment may be corrupted by the channel noise
$P(\mathbf{y}_{u}|\mathbf{z}_{u}=k)$, resulting in a (possibly new) class
assignment $\mathbf{y}_{u}=k^{\prime}$.

For each product $p\in\mathcal{P}$, we can proceed equivalently to get
possibly corrupted assignments $\mathbf{y}_{p}=\ell^{\prime}$ starting from
the sampled assignments $\mathbf{z}_{p}=\ell$. The topographic organization
has a twofold advantage: 1) it enables a model-based visualization tool of the
word patterns that can be easily investigated; 2) it regularises the latent
class model and hence prevents overfitting, while employing a large number of
latent classes [22].

Finally, the model has now the following form:

$P(\mathbf{y}_{u}=k^{\prime}|u^{i})=\sum_{k=1}^{K}P(\mathbf{y}_{u}=k^{\prime}|\mathbf{z}_{u}=k)P(\mathbf{z}_{u}=k|u^{i})$
(3)
$P(\mathbf{y}_{p}=\ell^{\prime}|p^{i})=\sum_{\ell=1}^{L}P(\mathbf{y}_{p}=\ell^{\prime}|\mathbf{z}_{p}=\ell)P(\mathbf{z}_{p}=\ell|p^{i})$
(4) $\displaystyle P(w_{j}^{i}|u^{i},p^{i})=$
$\displaystyle\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\bigg{(}P(w_{j}^{i}|\mathbf{y}_{u}=k^{\prime},\mathbf{y}_{p}=\ell^{\prime})\big{.}$
$\displaystyle\big{.}P(\mathbf{y}_{u}=k^{\prime}|u^{i})P(\mathbf{y}_{p}=\ell^{\prime}|p^{i})\bigg{)}.$
(5)

In our model the class-conditional word distribution
$P(w_{j}^{i}|\mathbf{y}_{u}=k^{\prime},\mathbf{y}_{p}=\ell^{\prime})$ is
formulated as multinomial.

###  III-B Inference and Learning

Assuming independent data items in $\cal D$, the log-likelihood of the model
is:

$\mathcal{L}=\sum_{i=1}^{R}\log P(r^{i}|u^{i},p^{i}).$ (6)

We will consider a simple review model assuming independence of the words
appearing in the review i:

$P(r^{i}|u^{i},p^{i})=\prod_{j=1}^{S_{i}}P(w^{i}_{j}|u^{i},p^{i}).$ (7)

Hence, the log-likelihood reads:

$\mathcal{L}=\sum_{i=1}^{R}\sum_{j=1}^{S_{i}}\log P(w^{i}_{j}|u^{i},p^{i}).$
(8)

Plugging in eqs. (3–III-A), we obtain:

$\displaystyle\mathcal{L}=\sum_{i=1}^{R}\sum_{j=1}^{S_{i}}\log\bigg{[}$
$\displaystyle\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}P(w^{i}_{j}|\mathbf{y}_{u}=k^{\prime},\mathbf{y}_{p}=\ell^{\prime})$
$\displaystyle\sum_{k=1}^{K}P(\mathbf{y}_{u}=k^{\prime}|\mathbf{z}_{u}=k)P(\mathbf{z}_{u}=k|u^{i})$
$\displaystyle\sum_{\ell=1}^{L}P(\mathbf{y}_{p}=\ell^{\prime}|\mathbf{z}_{p}=\ell)P(\mathbf{z}_{p}=\ell|p^{i})\bigg{]}.$
(9)

For training, we use the Expectation-Maximization (EM) algorithm enabling
maximum likelihood estimation (MLE) in latent variable models. It iterates the
Expectation (E) and Maximization (M) steps until convergence. Detailed
derivations of the following equations are presented in Appendix A.

###  III-C E-step

In the E-step, the algorithm evaluates the current estimates of the model
parameters by computing the expected values of the latent variables. We will
denote these quantities using $\hat{P}(\cdot|\cdot)$. Note that we have two
levels of hidden variables. First, given the user and the product they
reviewed, we do not know which latent classes $\mathbf{z}_{u}$ and
$\mathbf{z}_{p}$ represented the (user, product) couple when writing the
review. Second, we know that the underlying latent classes $\mathbf{z}_{u}$
and $\mathbf{z}_{p}$ may have been disrupted to latent classes
$\mathbf{y}_{u}$ and $\mathbf{y}_{p}$ before producing the review, but we do
not know their identity. To simplify mathematical notation, we will denote
$\mathbf{z}_{u}=k$, $\mathbf{z}_{p}=\ell$, $\mathbf{y}_{u}=k^{\prime}$ and
$\mathbf{y}_{p}=\ell^{\prime}$ as $\mathbf{z}_{u}^{k}$,
$\mathbf{z}_{p}^{\ell}$, $\mathbf{y}_{u}^{k^{\prime}}$ and
$\mathbf{y}_{p}^{\ell^{\prime}}$, respectively.
$\hat{P}({\mathbf{z}}_{u}^{k},{\mathbf{z}}_{p}^{\ell}|u,p,w)$ is evaluated as:

$\frac{P(\mathbf{z}_{u}^{k}|u)P(\mathbf{z}_{p}^{\ell}|p)\sum_{k^{\prime}}\sum_{\ell^{\prime}}S(k,\ell)}{\sum\limits_{k^{\prime\prime}}\sum\limits_{\ell^{\prime\prime}}P(\mathbf{z}_{u}^{k}|u)P(\mathbf{z}_{p}^{\ell}|p)\sum\limits_{k^{\prime}}\sum\limits_{\ell^{\prime}}S(k^{\prime\prime},\ell^{\prime\prime})}$
(10)

with

  * -

$S(\alpha,\beta)=P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})P(\mathbf{y}_{u}^{k^{\prime}}|\mathbf{z}_{u}^{\alpha})P(\mathbf{y}_{p}^{\ell^{\prime}}|\mathbf{z}_{p}^{\beta})$.

Instead,
$\hat{P}({\mathbf{y}}_{u}^{k^{\prime}},{\mathbf{y}}_{p}^{\ell^{\prime}}|w,u,p)$
is computed as:

$\frac{P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})\sum_{k}G(k^{\prime})\sum_{\ell}W(\ell^{\prime})}{\sum\limits_{k^{\prime\prime}}\sum\limits_{\ell^{\prime\prime}}P(w|\mathbf{y}_{u}^{k^{\prime\prime}},\mathbf{y}_{p}^{\ell^{\prime\prime}})\sum\limits_{k}G(k^{\prime\prime})\sum\limits_{\ell}W(\ell^{\prime\prime})}$
(11)

where

  * -

$G(\alpha)=P(\mathbf{y}_{u}^{\alpha}|\mathbf{z}_{u}^{k})P(\mathbf{z}_{u}^{k}|u)$

  * -

$W(\beta)=P(\mathbf{y}_{p}^{\beta}|\mathbf{z}_{p}^{\ell})P(\mathbf{z}_{p}^{\ell}|p)$.

###  III-D M-step

In the M-step, the algorithm maximizes the expectation computed in the E-step
by re-estimating the model parameters. To do so, we need to specify functional
forms of the distributions for $P(\mathbf{z}_{u}^{k}|u)$,
$P(\mathbf{z}_{p}^{\ell}|p)$ and
$P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})$. It is
natural to model these distributions as multinomial distributions. Thus, we
assume:

$\displaystyle P(\mathbf{z}_{u}^{k}|u)$ $\displaystyle\sim\text{Multinomial}$
(12) $\displaystyle P(\mathbf{z}_{p}^{\ell}|p)$
$\displaystyle\sim\text{Multinomial}$ (13) $\displaystyle
P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})$
$\displaystyle\sim\text{Multinomial}.$ (14)

The update equation for
$P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})$ is:

$\begin{split}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})&=\frac{\sum\limits_{(u,p)\in{\cal
B}(w)}\hat{P}(\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)}{\sum\limits_{w^{\prime}}\sum\limits_{(u,p)\in{\cal
B}(w^{\prime})}\hat{P}(\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w^{\prime})}\end{split}$
(15)

where ${\cal B}(w)$ is the set of (user, product) tuples associated with the
word $w$.

Denoting the set of words used by user $u$ to review product $p$ by ${\cal
W}(u,p)$, we obtain the update equations for $P(\mathbf{z}_{u}^{k}|u)$ and
$P(\mathbf{z}_{p}^{\ell}|p)$ as:

$\begin{split}P(\mathbf{z}_{u}^{k}|u)&=\frac{\sum_{p}\sum_{w\in{\cal
W}(u,p)}\sum_{\ell}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)}{\sum_{p}|{\cal
W}(u,p)|}\end{split}$ (16)

$\begin{split}P(\mathbf{z}_{p}^{\ell}|p)&=\frac{\sum_{u}\sum_{w\in{\cal
W}(u,p)}\sum_{k}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)}{\sum_{u}|{\cal
W}(u,p)|},\end{split}$ (17)

where $|{\cal W}(u,p)|$ is the size of ${\cal W}(u,p)$.

Figure 1: Input example for our architecture. Each pixel represents the value
of the corresponding latent class probability.

###  III-E Topographic Initialization with SOM

The successful application of the EM algorithm depends on the initial position
in the parameter space since the algorithm can be sensitive to parameter
initialization. We follow an approach similar to the one presented in [22].
Different from their work, where the SOM was run on a dataset of user ratings,
we run two different instances of SOM (one for users and one for products)
using the review information. We set the number of nodes in SOM to be
equivalent to the number of latent classes, i.e. $K$ for users and $L$ for
products. We denote by $\mathcal{V}$ the vocabulary set. For each user $u$,
there is an associated $|\mathcal{V}|$-dimensional vector $\mathbf{v}_{u}$.
Each vector dimension represents a word $w$ in the vocabulary and the
corresponding value is the term frequency, i.e. how many times the user has
written a review using that word. The analogous reasoning is valid for
products. In the latter case, each vector dimension represents how many times
a word $w$ has been used for reviewing the product $p$.

After training the two instances of SOM, the conditional latent priors for
users and products, i.e. $P(\mathbf{z}_{u}^{k}|u)$ and
$P(\mathbf{z}_{p}^{\ell}|p)$ respectively, are computed as follows. If the
user $u$ belongs to the SOM cluster node
$k\in\mathcal{Z}_{u}=\\{1,\dots,K\\}$, then we softened the hard assignment
using the following transformation:

$P(\mathbf{z}_{u}^{k}|u)=\begin{cases}A\phantom{(1-A)/(K-1)}\qquad\text{if
}u\in k\\\ (1-A)/(K-1)\phantom{A}\qquad\text{otherwise.}\end{cases}$ (18)

The parameter $A$ is defined such that if the user $u$ belongs to the class
$k$, $P(\mathbf{z}_{u}^{k}|u)=A$ should be $B>1$ times higher than
$P(\mathbf{z}_{u}^{k^{\prime}}|u)$ for all the other
$k^{\prime}\in\mathcal{Z}_{u}$. Thus, $A=B/(K-1+B)$. The product conditional
latent prior can be generated in an equivalent manner by changing the
parameters accordingly.

The empirical distribution for word patterns
$P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})$ is then
computed via the following procedure: (a) to introduce the topology, we follow
the steps described in Section III-A to get (possibly corrupted) class indices
$\mathbf{y}_{u}^{k^{\prime}}$ and $\mathbf{y}_{p}^{\ell^{\prime}}$ for all
users $u$ and products $p$; (b) we denote with $N(w,k^{\prime},\ell^{\prime})$
the number of times the word $w$ has been used by users belonging to the
latent class $k^{\prime}$ to review products that belong to the latent class
$\ell^{\prime}$. Thus, the empirical distribution is estimated as:

$P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})=\frac{N(w,k^{\prime},\ell^{\prime})+m}{mV+\sum_{w^{\prime}\in\mathcal{V}}N(w^{\prime},k^{\prime},\ell^{\prime})}.$
(19)

Due to sparseness, we apply a Laplace correction for smoothing the empirical
distribution. The parameter $m$ is a positive number and, in this case, $m=1$.
For further details, we refer the reader to [22].

###  III-F Out-of-sample extension

Figure 2: The schematic view of the proposed architecture. After the input
layer, we follow the principles of related CNN-based models for rating
prediction tasks.

Suppose we have an unknown user $u_{n}\notin\mathcal{U}$ reviewing a product
$\bar{p}$ that is available in our dataset. Let’s denote with $\bar{r}$ the
associated review. Using Bayes’ rules, we can extend the model to consider
users that were not included during the training phase. In this way, we can
learn the latent class assignment of unseen users as follows.

Using the independence of words assumed in (7), we can compute
$P(\bar{r}|\mathbf{y}_{u_{n}}^{k^{\prime}},\bar{p})$ as:

$P(\bar{r}|\mathbf{y}_{u_{n}}^{k^{\prime}},\bar{p})=\prod_{w\in\bar{r}}P(w|\mathbf{y}_{u_{n}}^{k^{\prime}},\bar{p})$
(20)

where

$\begin{split}P(w|\mathbf{y}_{u_{n}}^{k^{\prime}},\bar{p})=\sum_{\ell^{\prime}=1}^{L}P(w|\mathbf{y}_{u_{n}}^{k^{\prime}},\mathbf{y}_{\bar{p}}^{\ell^{\prime}})P(\mathbf{y}_{p}^{\ell^{\prime}}|\bar{p}).\end{split}$
(21)

Consequently, by the Bayes’ rule, we have:

$\begin{split}P(\mathbf{y}_{u_{n}}^{k^{\prime}}|\bar{r})=\frac{P(\bar{r}|\mathbf{y}_{u_{n}}^{k^{\prime}},\bar{p})P(\mathbf{y}_{u_{n}}^{k^{\prime}}|u_{n})}{\sum_{k^{\prime\prime}}P(\bar{r}|\mathbf{y}_{u_{n}}^{k^{\prime\prime}},\bar{p})P(\mathbf{y}_{u_{n}}^{k^{\prime\prime}}|u_{n})},\end{split}$
(22)

assuming $P(\mathbf{y}_{u_{n}}^{k^{\cdot}}|{u_{n}})\sim\text{Unif}(0,1)$ as
flat prior.

###  III-G Rating prediction part

After running the EM algorithm, we have the estimates of the conditional
distributions $P(\mathbf{z}_{u}^{k}|u)$ and $P(\mathbf{z}_{p}^{\ell}|p)$ for
all users and products in our dataset. These quantities represent the
probability assignments of the items to their respective latent classes. For
the next phase of our experiment, we will use the encoded review information
as input for the rating prediction task. Intuitively, given the topological
organization of the latent classes on two-dimensional grids, we can think of
these quantities as images where each pixel represents a latent class and the
corresponding value is the latent class probability assignment. An example of
the model input transformation for a sampled review is given in Fig. 1. Given
the large employment of embedding techniques for representation learning in
recommendation systems, we propose to use the learned latent class assignments
as representations of users and products instead. In this way, we can generate
a self-explainable latent representation of the items that, at the same time,
should perform well on a rating prediction task. The schematic view of our
architecture for rating prediction is presented in Fig. 2. After the input
layer, the CNN-block consists of the common layers used in CNN-based models.
We perform two convolution operations with batch normalization and Rectified
Linear Units (ReLU) [35] as activation function. As a result, the training is
faster and more stable. Then, we apply a max-pooling operation to take the
maximum value of each feature map obtained by the convolution operations.
After the max-pooling operation, the convolutional results are reduced to a
fixed size vector for both users and products. We concatenate the results
given by the max pooling operation, and we use the resulting vector as input
for the fully connected network (FCN) block. This block simply represents a
neural network with several dense layers. More details on the structure of
this block are given in Section IV-B. The proposed architecture follows some
principles of CNN-based models for rating prediction, e.g. [16, 23]. We are
not interested in getting better results than state-of-the-art approaches,
instead, our ultimate goal in this part is to investigate whether our proposed
model input is informative enough to get competitive results compared with
more specialized techniques and architectures. In theory, one might
potentially change the architecture as desired.

The objective is the error of the predictions. Following previous works, this
will be validated according to the Mean Squared Error (MSE):

$\text{MSE}=\frac{1}{|\mathcal{T}|}\sum_{(u,p)\in\mathcal{T}}(\widehat{\texttt{score}}_{u,p}-\texttt{score}_{u,p})^{2}$
(23)

where $\mathcal{T}$ represents either the test or the validation set.

##  IV Experiments

In this section, we evaluate the performance of our proposed
framework.222https://github.com/GiuseppeSerra93/TLCM First, we analyze the
probabilistic latent model and the generated organization of user and product
latent classes. Second, we investigate the generative extension of our
approach. Last, we compare the performance of our model with state-of-the-art
approaches for the rating prediction task.

TABLE I: Statistics of the Preprocessed Data Sets. | Reviews | Users | Items
---|---|---|---
Amazon Instant Videos | 36241 | 5127 | 1685
Automotive | 20285 | 2926 | 1835
Baby | 156335 | 19442 | 7050
Beauty | 194130 | 22356 | 12101
Cell Phones and Acc. | 191336 | 27864 | 10429
Digital Music | 64260 | 5539 | 3568
Grocery and Gourmet Food | 148902 | 14675 | 8713
Health and Personal Care | 333274 | 38583 | 18534
Musical Instruments | 10218 | 1427 | 900
Office Products | 52988 | 4902 | 2420
Patio, Lawn and Garden | 13213 | 1684 | 962
Pet Supplies | 152367 | 19848 | 8510
Sports and Outdoors | 289181 | 35588 | 18357
Tools and Home Improv. | 133445 | 16634 | 10217
Toys and Games | 161603 | 19405 | 11924
Videogames | 227859 | 24291 | 10672

###  IV-A Data Sets Description

We evaluate our method with a benchmark dataset commonly used for product
rating prediction: Amazon Product Data.333http://jmcauley.ucsd.edu/data/amazon
The dataset is a collection of product reviews and metadata, divided per
category, retrieved from May 1999 to July 2014. We focus on the 5-core version
of the data sets, where each user and item has at least 5 associated reviews.
The ratings are integer values between 1 and 5. We use data from 16 categories
for our experiments. Following [36], the review text has been normalized by:
(a) setting the maximum length of a raw review to 300; (b) lowering case
letters; (c) removing stopwords, numbers, and special characters; (d) removing
non-existing words using an English vocabulary as a filter; (e) lemmatization;
(f) removing reviews with just one word. Since the textual information shows
strong diversity depending on the product category, we treat each dataset
independently. Hence, we select a vocabulary for each category. The vocabulary
size is $V=2000$ for all product categories. The word selection is based on a
modified version of the tf-idf index. After the vocabulary selection, we
further filter the reviews to remove the ones without any word belonging to
the corresponding vocabulary. Some statistics of the preprocessed data sets
are summarized in Table I. From the table, we can observe that the sample size
of the data may vary consistently depending on the category. For example, the
largest category (Beauty) has around 30 times as many reviews as the smallest
one (Musical instruments). Similarly, the number of users and products
oscillates within a wide range of values among different categories. The data
sets represent different and realistic conditions; this acts as a good
framework for evaluating the quality of the results.

###  IV-B Experimental Settings

We set the number of user latent classes $K=25$ and product classes $L=16$; we
evaluated the robustness of our method to changes in the hyperparameters $K$
and $L$ but did not observe any significant difference in both qualitative and
quantitative analysis. This is in line with the constraint imposed by the
topological organization; using a larger number of latent classes did not lead
to overfitting. The specificity parameters for users and products, defined in
(2), are set respectively to $\sigma_{u}=3$ and $\sigma_{p}=2$. For the SOM
initialization (Section III-E), we used the Python package
Minisom.444https://github.com/JustGlowing/minisom

For the probabilistic model part, following previous works (e.g. [22, 37]), we
apply the so-called all but one protocol. From each user having at least $10$
reviews, we randomly select one review to be assigned to the test set. The
main focus of this part of the experiment is on discovering word patterns of
users and products within large collections of review data, hence we do not
need a generative formulation of the model. We evaluate the training procedure
according to the normalized negative log-likelihood (NLL). Based on (6), we
have:

$NLL_{\mathcal{T}}=-\frac{1}{|\mathcal{T}|}\sum_{(u,p,r)\in\mathcal{T}}\log\hat{P}(r|u,p)$
(24)

where $\mathcal{T}$ represents either the training or test set.

For the rating prediction task, as in previous works, the data are randomly
split by reviews into training (80%), validation (10%), and test (10%) sets.
Additionally, we remove reviews from the validation and test sets if either
the associated user or product does not belong to the training data set. The
FCN block depicted in Fig. 2 is a neural network with four hidden fully-
connected layers $[128,64,32,16]$. We evaluate different configurations of
this block by changing the number of layers and the number of units for each
layer. We select the best configuration based on the validation scores. The
final experiment was run for $200$ learning iterations and validated every
iteration. A single epoch performs Adam optimizer [38] with a learning rate
set to $0.02$ and batch size of $256$. The training metric is the MSE, as
defined in (23). To prevent overfitting, we monitor the validation set score
using early stopping with patience set to 10 epochs.

###  IV-C Topographic Organization of Latent Classes

Given the model parameters estimated in Section III-D, we can analyze the
organization of user and product latent classes by inspecting the associated
word distributions. The word distribution for each user latent class, under
the assumption of uninformative flat prior over latent classes, is computed
as:

$\begin{split}P(w|\mathbf{y}_{u}^{k^{\prime}})&=\sum_{\ell^{\prime}=1}^{L}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})P(\mathbf{y}_{p}^{\ell^{\prime}})\\\
&=\frac{1}{L}\sum_{\ell^{\prime}=1}^{L}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}}).\end{split}$
(25)

Analogically, the word-distribution for each product latent class can be
derived as follows:

$\begin{split}P(w|\mathbf{y}_{p}^{\ell^{\prime}})&=\sum_{k^{\prime}=1}^{K}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})P(\mathbf{y}_{u}^{k^{\prime}})\\\
&=\frac{1}{K}\sum_{k^{\prime}=1}^{K}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}}).\end{split}$
(26)

The visualization of the most probable words associated with each latent class
helps us to evaluate the results. The list of the $10$ most probable words for
product latent classes of the Automotive category is provided in Table II.

TABLE II: Product-class Word Distributions - Automotive Category. light |  light |  blade |  wax
---|---|---|---
bulb |  fit |  wiper |  kit
bright |  lead |  great |  product
stock |  great |  fit |  great
white |  good |  good |  wiper
brighter |  bright |  product |  water
fit |  price |  well |  snow
price |  well |  rain |  blade
lead |  order |  brand |  wipe
replacement |  horn |  jeep |  rain
filter |  great |  product |  product
oil |  good |  hose |  paint
change |  fit |  great |  great
fit |  tool |  good |  pad
price |  price |  tank |  clay
fuel |  product |  fit |  good
socket |  code |  well |  wax
mile |  well |  try |  polish
wrench |  purchase |  say |  try
good |  say |  quality |  water
oil |  good |  product |  product
change |  great |  great |  jack
fluid |  product |  good |  great
pump |  mount |  cover |  wash
drain |  item |  strap |  shine
price |  quality |  lock |  spray
leak |  fit |  trailer |  trailer
transmission |  price |  hitch |  good
good |  door |  door |  lift
great |  hitch |  step |  wax
battery |  plug |  wheel |  towel
charge |  tender |  leather |  wash
power |  battery |  brush |  water
charger |  great |  cleaner |  gauge
plug |  product |  seat |  pressure
device |  cable |  great |  cloth
motorcycle |  motorcycle |  level |  cleaning
phone |  good |  product |  accurate
compressor |  spark |  trailer |  valve
cord |  winter |  good |  great

From the results, we can notice clear patterns of the most probable words
associated with each latent class. Additionally, we can observe the
topological organization of the latent classes: words patterns are similar for
adjacent classes in the grid. For example, adjacent classes at the top left of
the grid refer to lighting accessories, while going down to the bottom left we
can deduce that the latent classes refer to electrical and oil system tools.
On the right part of the grid, instead, we have latent classes referring to
cleaning accessories. Knowing the probabilistic assignments for a sampled
item, we can easily identify the most probable latent classes and evaluate the
corresponding word distributions. This visualization tool lends itself well to
other types of result investigation. For example, if we are interested in
exploring a specific user latent class, one may analyze how the product latent
classes change when conditioned on the user class of interest. Theoretically,
the product latent classes should change according to the main topic of the
user latent class, i.e. most of the product latent classes should now be
associated with the user class of interest. The same reasoning is valid
starting from a specific product latent class. Given these considerations, we
provide a flexible and straightforward tool to investigate the latent
characteristics of the items, making the results clear and understandable.

###  IV-D Out-of-sample Extension Results

After evaluating the quality of the latent class organizations, we investigate
whether our model is able to correctly assign unseen users to the
corresponding latent classes. Given a product category, we randomly select a
review $\bar{r}$ written by an unknown user $u_{n}$ from the out-of-sample
set. Then, given the review $\bar{r}$, we compute
$P(\mathbf{y}_{u_{n}}^{k^{\prime}}|\bar{r})$ as described in (22).

Figure 3: Results of the out-of-sample extension from the category Pet
Supplies. The words in the bottom box are the ones used by the unknown user
$u_{n}$ to review a product $p$ in our dataset. The considered words are
included in the corresponding vocabulary $\mathcal{V}_{\text{pet}}$. Note that
the darker the color, the higher is the probability assignment to the
corresponding latent class.

Fig. 3 shows the results taken from the Pet Supplies category. To make the
visualization easier and intuitive, we visualize the learned user latent
probability assignments over the user-class word distributions. First, by
examining the most probable words associated with each user latent class, we
can observe how the latent classes are topologically well organized. Then, by
analyzing the words used by the unknown user listed in the bottom box, we can
assume that the reviewed product is related to the aquariums subcategory. The
results show the ability to assign the unknown user to the correct user latent
classes. In fact, the most probable latent classes associated with the new
user are the ones related to aquariums, as easily perceivable by inspecting
the corresponding word lists in the grid. It is important to mention that we
are able to get meaningful latent class assignments when the words used by the
unknown users are, to some extent, informative. For example, if the words were
only adjectives (e.g. good, nice) or verbs (e.g. look, work) it would be
difficult to classify unknown users correctly. This highlights the importance
of selecting a good vocabulary set as a starting point for the evaluation.

TABLE III: MSE for TLCM and State-of-the-art Approaches. Category | NMF | SVD | TNET | MPCN | TransRev | DeepCoNN | TLCM-CNN | TLCM-LR
---|---|---|---|---|---|---|---|---
Amazon Instant Videos | 1.628 | 1.206 | 1.007 | 0.997 | 0.884 | 0.957 | 0.961 | 1.004
Automotive | 1.171 | 0.818 | 0.946 | 0.861 | 0.855 | 0.792 | 0.879 | 0.883
Baby | 2.066 | 1.445 | 1.338 | 1.304 | 1.100 | 1.440 | 1.094 | 1.309
Beauty | 2.163 | 1.521 | 1.404 | 1.387 | 1.158 | 1.214 | 1.194 | 1.352
Cell Phones and Acc. | 2.438 | 1.664 | 1.431 | 1.413 | 1.279 | 1.378 | 1.294 | 1.447
Digital Music | 1.787 | 1.381 | 1.004 | 0.970 | 0.782 | 0.808 | 0.836 | 1.070
Grocery, Gourmet Food | 1.879 | 1.370 | 1.129 | 1.125 | 0.957 | 1.542 | 1.064 | 1.117
Health | 2.045 | 1.452 | 1.249 | 1.238 | 1.011 | 1.093 | 1.091 | 1.232
Musical Instruments | 1.287 | 0.703 | 1.100 | 0.923 | 0.690 | 0.896 | 0.670 | 0.697
Office Products | 0.988 | 0.719 | 0.840 | 0.779 | 0.724 | 0.723 | 0.735 | 0.834
Patio, Lawn and Garden | 1.429 | 0.967 | 1.123 | 1.011 | 0.941 | 1.020 | 1.045 | 1.100
Pet Supplies | 2.071 | 1.438 | 1.346 | 1.328 | 1.191 | 1.447 | 1.244 | 1.381
Sports | 1.490 | 1.011 | 0.994 | 0.980 | 0.823 | 0.898 | 0.898 | 0.958
Tools and Home Improv. | 1.793 | 1.222 | 1.122 | 1.096 | 0.879 | 1.208 | 0.954 | 1.091
Toys and Games | 1.542 | 1.111 | 0.974 | 0.973 | 0.784 | 0.806 | 0.859 | 0.928
Videogames | 2.188 | 1.629 | 1.276 | 1.257 | 1.082 | 1.135 | 1.111 | 1.370
| 1.748* | 1.229* | 1.143* | 1.103* | 0.946* | 1.085* | 0.996* | 1.111*

###  IV-E Rating Prediction Task

We compare our method with several baselines considering both factorization-
based approaches and methods that take advantage of the textual information
for improving the rating prediction performance.

Non-negative Matrix Factorization (NMF) is a standard baseline for CF. The
rating prediction is set as the dot product between item and user factors,
i.e. $q_{i}^{\top}p_{u}$. The latent factors are kept positive.
Singular Value Decomposition (SVD) is very similar to NMF. The rating is
computed as a dot product between the user and the product latent factors. In
both cases, the results are poorly explainable. Indeed, it is not possible to
understand user and product characteristics from the evaluation of the latent
factors.
Multi-Point Co-Attention Networks (MPCN) [17] proposes a pointer-based model
to extract important reviews from user and item reviews, based on the
intuition that not all the reviews are equally informative. Once the important
reviews are selected, a co-attentive layer learns the most important words
associated. In this way, the model can learn the most informative user and
product reviews for each user-item pair.
Deep Co-Operative Neural Networks (DeepCoNN) [16] learns convolutional user
and item representations from the textual information. Even though is a text-
based model, the analysis is focused only on the rating prediction
performances. Additionally, given the deep nature of the representations, the
results are poorly interpretable.
TransNets (TNET) [15] is an extension of the DeepCoNN model. It introduces an
additional transform layer that learns an approximation of the review
corresponding to the target user-item pair and, during the training phase,
enforces it to be similar to the embedding of the actual target review. The
model can be used to suggest reviews that are more similar to the one
potentially written by the target user. The qualitative evaluation just rely
on the visualization of some sampled reviews by highlighting the most similar
sentences between the original review and the predicted most helpful one.
However, if singularly evaluated, the learned embeddings do not provide any
interpretable information.
TransRev [18] is similar to TransNets in that it learns review embeddings as a
translation of the reviewer representation to the reviewed product embedding.
Ratings are predicted based on an approximation of the review embedding at
test time based on the difference between the embedding of the user and the
item. As in TransNets, the approximation is also used to suggest a tentative
review to users for further elaborations. The evaluation of the learned word
embedding is performed using t-SNE. Items and users are related to reviews by
means of non-interpretable latent representations.
TLCM-CNN stands for Transparent Latent Class Model with CNN and represents the
rating prediction model described in Section III-G. This comparison helps us
to investigate if the latent representations of our probabilistic model, even
though not learned ad-hoc, represent an informative input for the rating
prediction task. For the reasons explained before, our input representations,
if able to get competitive results, would be better in terms of
interpretability capacity compared to the embedding techniques presented in
the previous works.
TLCM-LR performs a simple linear regression (LR) that, for each review, takes
as input the concatenation of the estimates of the corresponding user and
product learned by our probabilistic model. As stated in section III-G, the
architecture for rating prediction can be changed as desired. In this case, we
decided to use a linear regression model since, among other methods, it is
known to be a transparent model. Following the direction of previous
investigations on the neural hype, this experiment allows us to understand
whether the use of complicated architectures is helping to make significant
progress for recommendation tasks.

As anticipated, the ultimate goal of this work is to build a review-based
model suitable for both visualizing and learning user and product
characteristics. Then, driven by the limitations of existing approaches to
represent users and products through explainable latent vectors, we propose to
exploit our estimated quantities as input for rating prediction. This idea is
fully motivated by the strong correlation between ratings and reviews. In
addition, we also propose to use the same information as input for an
interpretable model, i.e. a linear regression, to understand its predictive
performances compared to neural approaches. This experimental part completes
our analysis about the “phantom progress” of neural-based approaches for
representation learning and recommendation tasks, considering also the
interpretability capacity of the models taken into consideration.

In terms of interpretability, as already mentioned, all the baselines can
provide meaningful representations in the latent space, but neither the single
numbers contained in the vectors nor their dimensions have an interpretable
meaning [39]. The main limitation of these techniques laid in the fact that
they can capture relations among items by using vectors that are only
meaningful to each other. For instance, if we try to evaluate user vectors
without employing the review vectors, we would not be able to comprehend the
latent textual information associated with them. Differently, our model can
directly encode the textual information within the estimated quantities,
providing vectors that are self-explainable. Additionally, our latent
representations are more compact and sparse, making them easier to evaluate.
Finally, the intuitive visualization properties provided by our square grids
create a simple tool to investigate the results, further enhancing their
interpretability.

In terms of rating prediction performance, we independently repeat each
experiment on five different random splits closely following the experimental
setup specified in the relevant studies, including reporting the averaged Mean
Squared Error (MSE) as the performance measure. For a fair comparison, we
conduct and compare the baselines on the same data splits, when possible,
using the default configurations provided by the authors. If not, whether for
difficulty in reusing the source code or its unavailability, we directly copy
the results from the original papers. In detail, for SVD and NMF we used the
Python package Surprise555http://surpriselib.com/ and we selected the best
learning parameters through grid search. For DeepCoNN, since the original
authors’ source code has not been released, we used a third-party
implementation.666https://github.com/chenchongthu/DeepCoNN In this case, we
applied the default hyperparameter setting. For the remaining baselines, we
copied the results from the corresponding original papers. Results in terms of
MSE are reported in Table III. The asterisk indicates the macro MSE across all
the product categories. From the results, in line with the conclusions of
existing works, review-based methods outperform the ones based only on the
numerical information. We can observe that our model is able to outperform
most of the state-of-the-art approaches in the analyzed categories and has
comparable performances with the best one, i.e. TransRev. Additionally, it is
worth to mention that our simplest version of the architecture, i.e. TLCM-LR,
is also able to get comparable results against most of the state-of-the-art
approaches. This demonstrates that: 1) by carefully encoding the textual
information into the latent representations, it is possible to get similar
results in comparison with more complicated techniques that provide dense and
high-dimensional vector representations of the items; b) in line with the
conclusions of previous investigations on CF data, also in case the textual
information is used, we do not need to employ complicated neural-based models
to get competitive results. The latent representations proposed in our work,
even though not learned to specifically perform well on a rating prediction
task, are able to maintain competitiveness in comparison with more specialized
architectures. This clearly suggests that one may potentially use the proposed
model input with nice interpretable properties, without worsening the rating
prediction task considerably.

##  V Conclusion

Recently, several approaches for rating predictions of textual reviews in the
framework of deep neural networks have appeared in the literature [14, 27, 15,
16, 17, 18]. Given the highly stochastic nature of the data and relative data
sparsity, one can legitimately ask to what extent can the full predictive
power of deep networks be utilized in this context. The question is even more
relevant when one realises that clear interpretability of the deep network
functionality is still an open problem [40]. To answer this question, we
present an approach for product rating prediction using a relatively simple
and interpretable latent class probabilistic model utilizing topographic
organization of user and product latent classes based on the review
information. In existing works, the review information is usually exploited to
enhance the rating prediction performances, but is not fully inspected to
understand user and product features. In contrast, we propose a deeper
understanding of the data, presenting a two-step approach that starts from the
interpretable organization of textual information to arrive at the rating
prediction task. The organization of the latent classes on 2-dimensional grids
provides a visualization tool that can be used to statistically investigate
user and product features from a review-based perspective. Through this
organization, we can arrange complicated and unstructured textual data in a
simple way. The thorough analysis in the experimental section demonstrates the
ease of analyzing the latent review patterns using tools from probabilistic
theory. The visualization of the results, presented in sections IV-C and IV-D,
shows that the lower-dimensional latent representations of users and products
are a good approximation of the textual input space. Consequently, driven by
the assumption that ratings and reviews are strongly correlated, we propose to
use the resulting latent features as input for a rating prediction task. In
this part, we contribute to the debate about the “phantom progress” of deep
learning approaches for recommendation tasks. Our investigation, differently
from the previous ones, is mainly focused on methods that take advantage of
the textual information for rating prediction tasks, and includes an
evaluation that considers the capacity of such models to represent users,
products, and reviews utilizing interpretable latent vectors. The results
suggest that, also in the textual-based case, the use of dense and complicated
representations is not fully motivated. Indeed, even though our
representations are not learned for a rating prediction task specifically, the
results are comparable to models that learn ad-hoc representations.
Nonetheless, being highly interpretable, the proposed latent representations
overcome the limitations of the common embedding techniques used in most of
the considered previous works. Finally, the prediction results of our linear
regression model suggest that we do not need to implement deep architectures
either. This simple model is able to outperform some of the baselines and get
comparable results with the remaining ones, while being fully transparent.
Naturally, there is always a trade-off between model capabilities and
interpretability. The nice and explainable visualization properties of our
constrained model may affect the modeling capabilities for rating prediction.
On the other hand, better modeling capabilities through common embedding
techniques and deep architectures provide representations that are created for
performing well on a specific task, but at the price of losing the human
interpretation of the results.

## Acknowledgment

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No 766186.

## References

  * [1] G. Lample, M. Ott, A. Conneau, L. Denoyer, and M. Ranzato, “Phrase-based & neural unsupervised machine translation,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , 2018, pp. 5039–5049.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 2019, pp. 4171–4186.
  * [3] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, “Image transformer,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4055–4064.
  * [4] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2020.
  * [5] J. Li, P. Ren, Z. Chen, Z. Ren, T. Lian, and J. Ma, “Neural attentive session-based recommendation,” in _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , 2017, pp. 1419–1428.
  * [6] R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, and J. Leskovec, “Graph convolutional neural networks for web-scale recommender systems,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 974–983.
  * [7] S. Wu, Y. Tang, Y. Zhu, L. Wang, X. Xie, and T. Tan, “Session-based recommendation with graph neural networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2019, pp. 346–353.
  * [8] Y. Zhang and X. Chen, “Explainable recommendation: A survey and new perspectives,” _Foundations and Trends in Information Retrieval_ , pp. 1–101, 2020.
  * [9] M. F. Dacrema, P. Cremonesi, and D. Jannach, “Are we really making much progress? a worrying analysis of recent neural recommendation approaches,” in _Proceedings of the 13th ACM Conference on Recommender Systems_ , 2019, pp. 101–109.
  * [10] M. F. Dacrema, S. Boglio, P. Cremonesi, and D. Jannach, “A troubling analysis of reproducibility and progress in recommender systems research,” _ACM Transactions on Information Systems (TOIS)_ , pp. 1–49, 2021.
  * [11] J. Lin, “The neural hype and comparisons against weak baselines,” in _ACM SIGIR Forum_. ACM New York, NY, USA, 2019, pp. 40–51.
  * [12] M. Ludewig and D. Jannach, “Evaluation of session-based recommendation algorithms,” _User Modeling and User-Adapted Interaction_ , pp. 331–390, 2018.
  * [13] D. Jannach and M. Ludewig, “When recurrent neural networks meet the neighborhood for session-based recommendation,” in _Proceedings of the Eleventh ACM Conference on Recommender Systems_ , 2017, pp. 306–310.
  * [14] J. McAuley and J. Leskovec, “Hidden factors and hidden topics: understanding rating dimensions with review text,” in _Proceedings of the 7th ACM conference on Recommender systems_. ACM, 2013, pp. 165–172.
  * [15] C. Tu, Z. Zhang, Z. Liu, and M. Sun, “Transnet: Translation-based network representation learning for social relation extraction.” in _IJCAI_ , 2017, pp. 2864–2870.
  * [16] L. Zheng, V. Noroozi, and P. S. Yu, “Joint deep modeling of users and items using reviews for recommendation,” in _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_ , 2017, pp. 425–434.
  * [17] Y. Tay, A. T. Luu, and S. C. Hui, “Multi-pointer co-attention networks for recommendation,” in _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2018, pp. 2309–2318.
  * [18] A. Garcia-Duran, R. Gonzalez, D. Onoro-Rubio, M. Niepert, and H. Li, “Transrev: Modeling reviews as translations from users to items,” in _European Conference on Information Retrieval_. Springer, 2020, pp. 234–248.
  * [19] T. Kohonen, “Self-organized formation of topologically correct feature maps,” _Biological cybernetics_ , pp. 59–69, 1982.
  * [20] C. M. Bishop, M. Svensén, and C. K. Williams, “Gtm: The generative topographic mapping,” _Neural computation_ , pp. 215–234, 1998.
  * [21] T. Hofmann, “Probmap–a probabilistic approach for mapping large document collections,” _Intelligent Data Analysis_ , pp. 149–164, 2000.
  * [22] G. Polčicová and P. Tiňo, “Making sense of sparse rating data in collaborative filtering via topographic organization of user preference patterns,” _Neural Networks_ , pp. 1183–1199, 2004.
  * [23] S. Seo, J. Huang, H. Yang, and Y. Liu, “Interpretable convolutional neural networks with dual local and global attention for review rating prediction,” in _Proceedings of the eleventh ACM conference on recommender systems_ , 2017, pp. 297–305.
  * [24] N. Jakob, S. H. Weber, M. C. Müller, and I. Gurevych, “Beyond the stars: exploiting free-text user reviews to improve the accuracy of movie recommendations,” in _Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion_ , 2009, pp. 57–64.
  * [25] G. Ling, M. R. Lyu, and I. King, “Ratings meet reviews, a combined approach to recommend,” in _Proceedings of the 8th ACM Conference on Recommender systems_ , 2014, pp. 105–112.
  * [26] Y. Bao, H. Fang, and J. Zhang, “Topicmf: Simultaneously exploiting ratings and reviews for recommendation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2014.
  * [27] A. Almahairi, K. Kastner, K. Cho, and A. Courville, “Learning distributed representations from reviews for collaborative filtering,” in _Proceedings of the 9th ACM Conference on Recommender Systems_ , 2015, pp. 147–154.
  * [28] H. Wang, N. Wang, and D.-Y. Yeung, “Collaborative deep learning for recommender systems,” in _Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , 2015, pp. 1235–1244.
  * [29] Y. Xian, Z. Fu, H. Zhao, Y. Ge, X. Chen, Q. Huang, S. Geng, Z. Qin, G. De Melo, S. Muthukrishnan _et al._ , “Cafe: Coarse-to-fine neural symbolic reasoning for explainable recommendation,” in _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, 2020, pp. 1645–1654.
  * [30] C. Chen, M. Zhang, Y. Liu, and S. Ma, “Neural attentional rating regression with review-level explanations,” in _Proceedings of the 2018 World Wide Web Conference_ , 2018, pp. 1583–1592.
  * [31] S. Yu, Y. Wang, M. Yang, B. Li, Q. Qu, and J. Shen, “Nairs: A neural attentive interpretable recommendation system,” in _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_ , 2019, pp. 790–793.
  * [32] X. Dong, J. Ni, W. Cheng, Z. Chen, B. Zong, D. Song, Y. Liu, H. Chen, and G. De Melo, “Asymmetrical hierarchical networks with attentive interactions for interpretable review-based recommendation,” in _Proceedings of the AAAI conference on artificial intelligence_ , 2020, pp. 7667–7674.
  * [33] L. Hu, S. Jian, L. Cao, and Q. Chen, “Interpretable recommendation via attraction modeling: Learning multilevel attractiveness over multimodal movie contents,” in _IJCAI International Joint Conference on Artificial Intelligence_ , 2018.
  * [34] T. Chen, H. Yin, G. Ye, Z. Huang, Y. Wang, and M. Wang, “Try this instead: Personalized and interpretable substitute recommendation,” in _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 2020.
  * [35] A. F. Agarap, “Deep learning using rectified linear units (relu),” _arXiv preprint arXiv:1803.08375_ , 2018.
  * [36] G. Serra, Z. Xu, M. Niepert, C. Lawrence, P. Tiňo, and X. Yao, “Interpreting node embedding with text-labeled graphs,” in _2021 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2021, pp. 1–8.
  * [37] T. Hofmann, “Learning what people (don’t) want,” in _European Conference on Machine Learning_. Springer, 2001, pp. 214–225.
  * [38] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [39] L. K. Şenel, I. Utlu, F. Şahinuç, H. M. Ozaktas, and A. Koç, “Imparting interpretability to word embeddings while preserving semantic structure,” _Natural Language Engineering_ , 2021.
  * [40] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. manuscript based on c. rudin please stop explaining black box machine learning models for high stakes decisions,” in _Proceedings of NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Learning_ , 2018.

##  Appendix A Derivations of EM Equations

###  A-A E-step

First, we estimate
$\hat{P}({\mathbf{z}}_{u}^{k},{\mathbf{z}}_{p}^{\ell}|w,u,p)$ as:

$\frac{P(w|u,p,\mathbf{z}_{u}^{k},\mathbf{z}_{p}^{\ell})P(\mathbf{z}_{u}^{k}|u,p)P(\mathbf{z}_{p}^{\ell}|u,p)}{\sum\limits_{k^{\prime\prime}}\sum\limits_{\ell^{\prime\prime}}P(w|u,p,\mathbf{z}_{u}^{k^{\prime\prime}},\mathbf{z}_{p}^{\ell^{\prime\prime}})P(\mathbf{z}_{u}^{k^{\prime\prime}}|u,p)P(\mathbf{z}_{p}^{\ell^{\prime\prime}}|u,p)}.$
(27)

By model assumptions, $\mathbf{z}_{u}$ is independent from products and
$\mathbf{z}_{p}$ is independent from users, so we have
$P(\mathbf{z}_{u}^{k}|u,p)=P(\mathbf{z}_{u}^{k}|u)$ and
$P(\mathbf{z}_{p}^{\ell}|u,p)=P(\mathbf{z}_{p}^{\ell}|p)$. Consequently:

$\begin{split}P(w|&u,p,\mathbf{z}_{u}^{k},\mathbf{z}_{p}^{\ell})=\sum_{k^{\prime}}\sum_{\ell^{\prime}}P(w,\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}}|u,p,\mathbf{z}_{u}^{k},\mathbf{z}_{p}^{\ell})\\\
&=\sum_{k^{\prime}}\sum_{\ell^{\prime}}\bigg{[}P(w|u,p,\mathbf{z}_{u}^{k},\mathbf{z}_{p}^{\ell},\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})\\\
&\qquad
P(\mathbf{y}_{u}^{k^{\prime}}|u,p,\mathbf{z}_{u}^{k})P(\mathbf{y}_{p}^{\ell^{\prime}}|u,p,\mathbf{z}_{p}^{\ell})\bigg{]}\\\
&=\sum_{k^{\prime}}\sum_{\ell^{\prime}}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})P(\mathbf{y}_{u}^{k^{\prime}}|\mathbf{z}_{u}^{k})P(\mathbf{y}_{p}^{\ell^{\prime}}|\mathbf{z}_{p}^{\ell}),\end{split}$
(28)

and so $\hat{P}({\mathbf{z}}_{u}^{k},{\mathbf{z}}_{p}^{\ell}|w,u,p)$ is equal
to

$\frac{P(\mathbf{z}_{u}^{k}|u)P(\mathbf{z}_{p}^{\ell}|p)\sum_{k^{\prime}}\sum_{\ell^{\prime}}S(k,\ell)}{\sum\limits_{k^{\prime\prime}}\sum\limits_{\ell^{\prime\prime}}P(\mathbf{z}_{u}^{k}|u)P(\mathbf{z}_{p}^{\ell}|p)\sum\limits_{k^{\prime}}\sum\limits_{\ell^{\prime}}S(k^{\prime\prime},\ell^{\prime\prime})},$
(29)

with

  * -

$S(\alpha,\beta)=P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})P(\mathbf{y}_{u}^{k^{\prime}}|\mathbf{z}_{u}^{\alpha})P(\mathbf{y}_{p}^{\ell^{\prime}}|\mathbf{z}_{p}^{\beta})$.

Analogously, to compute
$\hat{P}({\mathbf{y}}_{u}^{k^{\prime}},{\mathbf{y}}_{p}^{\ell^{\prime}}|w,u,p)$,
we have:

$\frac{P(w|u,p,\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})P(\mathbf{y}_{u}^{k^{\prime}}|u,p)P(\mathbf{y}_{p}^{\ell^{\prime}}|u,p)}{\sum_{k^{\prime\prime}}\sum_{\ell^{\prime\prime}}P(w|u,p,\mathbf{y}_{u}^{k^{\prime\prime}},\mathbf{y}_{p}^{\ell^{\prime\prime}})P(\mathbf{y}_{u}^{k^{\prime\prime}}|u,p)P(\mathbf{y}_{p}^{\ell^{\prime\prime}}|u,p)}$
(30)

with

$\begin{split}P(\mathbf{y}_{u}^{k^{\prime}}|u,p)&=\sum_{k}P(\mathbf{y}_{u}^{k^{\prime}},\mathbf{z}_{u}^{k}|u,p)\\\
&=\sum_{k}P(\mathbf{y}_{u}^{k^{\prime}}|u,p,\mathbf{z}_{u}^{k})P(\mathbf{z}_{u}^{k}|u,p)\\\
&=\sum_{k}P(\mathbf{y}_{u}^{k^{\prime}}|\mathbf{z}_{u}^{k})P(\mathbf{z}_{u}^{k}|u).\end{split}$
(31)

Following the same reasoning:

$\begin{split}P(\mathbf{y}_{p}^{\ell^{\prime}}|u,p)&=\sum_{\ell}P(\mathbf{y}_{p}^{\ell^{\prime}},\mathbf{z}_{p}^{\ell}|u,p)\\\
&=\sum_{\ell}P(\mathbf{y}_{p}^{\ell^{\prime}}|u,p,\mathbf{z}_{p}^{\ell})P(\mathbf{z}_{p}^{\ell}|u,p)\\\
&=\sum_{\ell}P(\mathbf{y}_{p}^{\ell^{\prime}}|\mathbf{z}_{p}^{\ell})P(\mathbf{z}_{p}^{\ell}|p).\end{split}$
(32)

Hence,
$\hat{P}({\mathbf{y}}_{u}^{k^{\prime}},{\mathbf{y}}_{p}^{\ell^{\prime}}|w,u,p)$
becomes:

$\frac{P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})\sum_{k}G(k^{\prime})\sum_{\ell}W(\ell^{\prime})}{\sum\limits_{k^{\prime\prime}}\sum\limits_{\ell^{\prime\prime}}P(w|\mathbf{y}_{u}^{k^{\prime\prime}},\mathbf{y}_{p}^{\ell^{\prime\prime}})\sum\limits_{k}G(k^{\prime\prime})\sum\limits_{\ell}W(\ell^{\prime\prime})},$
(33)

where

  * -

$G(\alpha)=P(\mathbf{y}_{u}^{\alpha}|\mathbf{z}_{u}^{k})P(\mathbf{z}_{u}^{k}|u)$

  * -

$W(\beta)=P(\mathbf{y}_{p}^{\beta}|\mathbf{z}_{p}^{\ell})P(\mathbf{z}_{p}^{\ell}|p)$

###  A-B M-step

For the M-step, we assume the functional forms of the distributions for
$P(\mathbf{z}_{u}^{k}|u)$, $P(\mathbf{z}_{p}^{\ell}|p)$ and
$P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})$ as:

$\displaystyle P(\mathbf{z}_{u}|u)$ $\displaystyle\sim\text{Multinomial}$ (34)
$\displaystyle P(\mathbf{z}_{p}|p)$ $\displaystyle\sim\text{Multinomial}$ (35)
$\displaystyle P(w|u,p)$ $\displaystyle\sim\text{Multinomial}.$ (36)

We introduce latent indicator variables
$\delta^{i,j}_{k,k^{\prime},\ell,\ell^{\prime}}$ such that if the user $u^{i}$
belonging to latent user class $k$ (that was corrupted through channel noise
to latent user class $k^{\prime}$) used the word $w^{i}_{j}$ when reviewing
the product $p^{i}$ belonging to latent product class $\ell$ (corrupted by
channel noise to latent product class $\ell^{\prime}$), then
$\delta^{i,j}_{k,k^{\prime},\ell,\ell^{\prime}}=1$, otherwise
$\delta^{i,j}_{k,k^{\prime},\ell,\ell^{\prime}}=0$.

The complete-data log likelihood is then:

$\begin{split}\mathcal{L}_{C}=&\sum_{i=1}^{R}\sum_{j=1}^{S_{i}}\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\sum_{k=1}^{K}\sum_{\ell=1}^{L}\delta^{i,j}_{k,k^{\prime},\ell,\ell^{\prime}}\\\
&\bigg{[}\log\bigg{(}P(w^{i}_{j}|\mathbf{y}_{u^{i}}=k^{\prime},\mathbf{y}_{p^{i}}=\ell^{\prime})\big{.}\\\
&P(\mathbf{y}_{u^{i}}=k^{\prime}|\mathbf{z}_{u^{i}}=k)P(\mathbf{z}_{u^{i}}=k|u^{i})\\\
&P(\mathbf{y}_{p^{i}}=\ell^{\prime}|\mathbf{z}_{p^{i}}=\ell)P(\mathbf{z}_{p^{i}}=\ell|p^{i})\bigg{)}\bigg{]}.\end{split}$
(37)

The expected complete data log-likelihood is:

$\begin{split}\mathcal{L}_{C}=&\sum^{R}_{i=1}\sum_{j=1}^{S_{i}}\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\sum_{k=1}^{K}\sum_{\ell=1}^{L}E[\delta^{i,j}_{k,k^{\prime},\ell,\ell^{\prime}}]\\\
&\left[\log
P(w_{j}^{i}|\mathbf{y}_{u^{i}}^{k^{\prime}},\mathbf{y}_{p^{i}}^{\ell^{\prime}})+\log
P(\mathbf{y}_{u^{i}}^{k^{\prime}}|\mathbf{z}_{u^{i}}^{k})\right.\\\
&+\left.\log
P(\mathbf{y}_{p^{i}}^{\ell^{\prime}}|\mathbf{z}_{p^{i}}^{\ell})+\log
P(\mathbf{z}_{u^{i}}^{k}|u^{i})+\log
P(\mathbf{z}_{p^{i}}^{\ell}|p^{i})\right].\end{split}$ (38)

For convenience, we will write $\mathcal{L}_{C}$ as:

$\begin{split}\mathcal{L}_{C}=&\sum^{R}_{i=1}\sum_{j=1}^{S_{i}}\Delta^{u^{i},p^{i},w^{i}_{j}}_{u,p,w}\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\sum_{k=1}^{K}\sum_{\ell=1}^{L}E[\delta^{i,j}_{k,k^{\prime},\ell,\ell^{\prime}}]\\\
&\left[\log
P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})+\log
P(\mathbf{y}_{u}^{k^{\prime}}|\mathbf{z}_{u}^{k})\right.\\\ &+\left.\log
P(\mathbf{y}_{p}^{\ell^{\prime}}|\mathbf{z}_{p}^{\ell})+\log
P(\mathbf{z}_{u}^{k}|u)+\log P(\mathbf{z}_{p}^{\ell}|p)\right],\end{split}$
(39)

where $\Delta^{\alpha}_{a}=1$ if and only if $a=\alpha$, otherwise
$\Delta^{\alpha}_{a}=0$. We maximize the expected complete data log-likelihood
extended with Lagrange multiplier terms:

$\begin{split}<\mathcal{L}_{C}>\quad=&\sum^{R}_{i=1}\sum_{j=1}^{S_{i}}\Delta^{u^{i},p^{i},w^{i}_{j}}_{u,p,w}\\\
&\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\sum_{k=1}^{K}\sum_{\ell=1}^{L}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)\\\
&\left[\log
P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})+\log
P(\mathbf{y}_{u}^{k^{\prime}}|\mathbf{z}_{u}^{k})\right.\\\ &+\left.\log
P(\mathbf{y}_{p}^{\ell^{\prime}}|\mathbf{z}_{p}^{\ell})+\log
P(\mathbf{z}_{u}^{k}|u)+\log P(\mathbf{z}_{p}^{\ell}|p)\right]\\\
&+\sum_{u\in\cal{U}}\lambda_{u}\left(\sum_{k}P(\mathbf{z}_{u}^{k}|u)-1\right)\\\
&+\sum_{p\in\cal{P}}\lambda_{p}\left(\sum_{\ell}P(\mathbf{z}_{p}^{\ell}|p)-1\right)\\\
&+\sum_{k^{\prime},\ell^{\prime}}\lambda_{k^{\prime},\ell^{\prime}}\left(\sum_{w\in\cal{V}}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})-1\right).\end{split}$
(40)

By setting $\frac{\partial<\mathcal{L}_{C}>}{\partial
P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})}=0$, we have:

$\begin{split}&\frac{\partial<\mathcal{L}_{C}>}{\partial
P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})}=0\Longleftrightarrow\\\
&\sum_{k=1}^{K}\sum_{\ell=1}^{L}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)\frac{1}{P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})}\\\
&+\lambda_{k^{\prime},\ell^{\prime}}=0.\end{split}$ (41)

Then

$\begin{split}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})&=-\frac{\sum\limits_{(u,p)\in\mathcal{B}_{w}}\sum\limits_{k}\sum\limits_{\ell}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)}{\lambda_{k^{\prime},\ell^{\prime}}}\\\
&=-\frac{\sum\limits_{(u,p)\in\mathcal{B}_{w}}\hat{P}(\mathbf{y}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)}{\lambda_{k^{\prime},\ell^{\prime}}},\end{split}$
(42)

where ${\cal B}(w)$ is the set of (user, product) tuples associated with the
word $w$. Substituting
$P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})$ back to the
constraint we have:

$\begin{split}&\sum_{w}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})=1\Longleftrightarrow\\\
&\lambda_{k^{\prime},\ell^{\prime}}=-\sum_{w}\sum_{(u,p)\in{\cal
B}(w)}\hat{P}(\mathbf{y}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w),\end{split}$
(43)

and so

$\begin{split}P(w|\mathbf{y}_{u}^{k^{\prime}},\mathbf{y}_{p}^{\ell^{\prime}})&=\frac{\sum\limits_{(u,p)\in{\cal
B}(w)}\hat{P}(\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)}{\sum\limits_{w^{\prime}}\sum\limits_{(u,p)\in{\cal
B}(w^{\prime})}\hat{P}(\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w^{\prime})}.\end{split}$
(44)

By setting $\frac{\partial<\mathcal{L}_{C}>}{\partial
P(\mathbf{z}_{u}^{k}|u)}=0$, we have:

$\begin{split}&\frac{\partial<\mathcal{L}_{C}>}{\partial
P(\mathbf{z}_{u}^{k}|u)}=0\Longleftrightarrow\\\
&\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\sum_{\ell=1}^{L}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)\frac{1}{P(\mathbf{z}_{u}^{k}|u)}\\\
&+\lambda_{u}=0.\end{split}$ (45)

Denoting the set of words used by user $u$ to review product $p$ by ${\cal
W}(u,p)$, we obtain:

$\begin{split}P(\mathbf{z}_{u}^{k}|u)&=-\frac{\sum\limits_{p}\sum\limits_{w\in{\cal
W}(u,p)}\sum\limits_{k^{\prime}}\sum\limits_{\ell^{\prime}}\sum\limits_{\ell}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)}{\lambda_{u}}\\\
&=-\frac{\sum\limits_{p}\sum\limits_{w\in{\cal
W}(u,p)}\sum\limits_{\ell}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)}{\lambda_{u}}.\end{split}$
(46)

Substituting $P(\mathbf{z}_{u}^{k}|u)$ back to the constraint we have:

$\begin{split}&\sum_{k=1}^{K}P(\mathbf{z}_{u}^{k}|u)=1\\\
&\Longleftrightarrow\lambda_{u}=-\sum_{k}\sum_{\ell}\sum_{p}\sum_{w\in{\cal
W}(u,p)}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)\\\
&\Longleftrightarrow\lambda_{u}=-\sum_{p}|{\cal W}(u,p)|\end{split}$ (47)

and so

$\begin{split}P(\mathbf{z}_{u}^{k}|u)&=\frac{\sum_{p}\sum_{w\in{\cal
W}(u,p)}\sum_{\ell}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)}{\sum_{p}|{\cal
W}(u,p)|}.\end{split}$ (48)

By setting $\frac{\partial<\mathcal{L}_{C}>}{\partial
P(\mathbf{z}_{p}^{\ell}|p)}=0$, we have:

$\begin{split}&\frac{\partial<\mathcal{L}_{C}>}{\partial
P(\mathbf{z}_{p}^{\ell}|p)}=0\Longleftrightarrow\\\
&\sum_{k^{\prime}=1}^{K}\sum_{\ell^{\prime}=1}^{L}\sum_{k=1}^{K}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)\frac{1}{P(\mathbf{z}_{p}^{\ell}|p)}+\lambda_{p}=0.\end{split}$
(49)

Then,

$\begin{split}P(\mathbf{z}_{p}^{\ell}|p)&=-\frac{\sum\limits_{u}\sum\limits_{w\in{\cal
W}(u,p)}\sum\limits_{k^{\prime}}\sum\limits_{\ell^{\prime}}\sum\limits_{k}\hat{P}({\mathbf{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell},\mathbf{{y}}_{u}^{k^{\prime}},\mathbf{{y}}_{p}^{\ell^{\prime}}|u,p,w)}{\lambda_{p}}\\\
&=-\frac{\sum\limits_{u}\sum\limits_{w\in{\cal
W}(u,p)}\sum\limits_{k}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)}{\lambda_{p}}.\end{split}$
(50)

Substituting $P(\mathbf{z}_{p}^{\ell}|p)$ back to the constraint we have:

$\begin{split}&\sum_{\ell}P(\mathbf{z}_{p}^{\ell}|p)=1\\\
&\Longleftrightarrow\lambda_{p}=-\sum_{k}\sum_{\ell}\sum_{u}\sum_{w\in{\cal
W}(u,p)}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)\\\
&\Longleftrightarrow\lambda_{p}=-\sum_{u}|{\cal W}(u,p)|\end{split}$ (51)

and finally:

$\begin{split}P(\mathbf{z}_{p}^{\ell}|p)&=\frac{\sum_{u}\sum_{w\in{\cal
W}(u,p)}\sum_{k}\hat{P}(\mathbf{{z}}_{u}^{k},\mathbf{{z}}_{p}^{\ell}|u,p,w)}{\sum_{u}|{\cal
W}(u,p)|}.\end{split}$ (52)