# Unsupervised Feature Learning Architecture with Multi-clustering Integration
RBM

Jielei Chu, , Hongjun Wang, Jing Liu,
Zhiguo Gong, , Tianrui Li Jielei Chu, Hongjun Wang, Tianrui Li (the
corresponding author) are with the Institute of Artificial Intelligence,
School of Information Science and Technology, Southwest Jiaotong University,
Chengdu 611756, China. Tianrui Li is also with National Engineering Laboratory
of Integrated Transportation Big Data Application Technology, Southwest
Jiaotong University, Chengdu 611756, China. e-mails: {jieleichu, wanghongjun,
<EMAIL_ADDRESS>Liu is with the School of Business, Sichuan
University, Sichuan, 610065, Chengdu, China. e-mail<EMAIL_ADDRESS>Gong is with the State Key Laboratory of Internet of Things for Smart City,
Department of Computer and Information Science, University of Macau, Macau,
China. Email<EMAIL_ADDRESS>
###### Abstract

In this paper, we present a novel unsupervised feature learning architecture,
which consists of a multi-clustering integration module and a variant of RBM
termed multi-clustering integration RBM (MIRBM). In the multi-clustering
integration module, we apply three unsupervised K-means, affinity propagation
and spectral clustering algorithms to obtain three different clustering
partitions (CPs) without any background knowledge or label. Then, an unanimous
voting strategy is used to generate a local clustering partition (LCP). The
novel MIRBM model is a core feature encoding part of the proposed unsupervised
feature learning architecture. The novelty of it is that the LCP as an
unsupervised guidance is integrated into one step contrastive divergence
($\textbf{{{CD}}}_{1}$) learning to guide the distribution of the hidden layer
features. For the instance in the same LCP cluster, the hidden and
reconstructed hidden layer features of the MIRBM model in the proposed
architecture tend to constrict together in the training process. Meanwhile,
each LCP center tends to disperse from each other as much as possible in the
hidden and reconstructed hidden layer during training. The experiments
demonstrate that the proposed unsupervised feature learning architecture has
more powerful feature representation and generalization capability than the
state-of-the-art graph regularized RBM (GraphRBM) for clustering tasks in the
Microsoft Research Asia Multimedia (MSRA-MM)2.0 dataset.

###### Index Terms:

multi-clustering integration RBM; unsupervised feature learning;
$\textbf{{{CD}}}_{1}$ learning; image clustering.

##  I Introduction

Feature learning is a crucial phase in many applications (e.g. visual
recognition [1], scene analysis [2], object recognition [3], multimodal
learning [4], [5], speech recognition [6], image classification [7]).
Supervised feature learning has achieved great success in machine learning
[8]. However, the labeled data is scarce in many applications. In recent
years, some works focus on the unsupervised feature learning method [9, 10,
11, 12, 13, 14]. But, how to obtain appropriate features distribution without
any background is still a hard problem in machine learning. To address this, a
novel unsupervised feature learning architecture with multi-clustering
integration RBM is designed in this paper.
Many modeling paradigms such as autoencoders and energy-based models have been
applied to feature learning. The restricted Boltzmann machine (RBM) [15] is a
popular energy-based model for unsupervised feature learning and aims to
explore appropriate hidden features. The structure of a RBM is a bipartite
graph consisting of a binary visible layer and a binary hidden layer. There
are no connections between the visible layer units and the hidden layer units.
The most popular learning algorithms of RBM such as stochastic maximum
likelihood [16] and contrastive divergence (CD) [17] base on the efficient
Gibbs sampling. There are a large number of successful applications based on
the RBMs, e.g., speaker recognition [6], feature fusion [5], clustering [9],
classification [18], [19], [20], computer vision [21] and speech recognition
[22]. Meanwhile, various variants of the RBMs have been proposed by the
researchers, e.g., pairwise constraints RBM with Gaussian visible units
(pcGRBM) [23], classification RBM [24], fuzzy RBM (FRBM) [25] and spike-and-
slab RBM (ssRBM) [26]. For real-valued data, the RBM with Gaussian visible
units [4], [23] as the canonical energy model has usually been applied to
extract the hidden features from image data. Unlike standard RBM, the visible
layer units of the model have Gaussian noise and the hidden layer still
maintains binary units. The CD learning can also be used to train the RBM with
Gaussian visible units [27]. The hidden representations of traditional RBMs do
not have explicit instance-level constraints. So, Chu et al. presented semi-
supervised pcGRBM in which pairwise constraints are fused into the
reconstructed visible layer [23]. However, the labeled data is lacking in many
applications and it is expensive to obtain more labels. So, it is certainly
worth exploring unsupervised feature learning method of RBMs by fusing
external interventions.
In this paper, we present a novel unsupervised feature learning architecture,
which consists of a multi-clustering integration module and a variant of RBM
termed multi-clustering integration RBM (MIRBM). In the multi-clustering
integration module, we choose three unsupervised K-means, affinity propagation
(AP) and spectral clustering (SC) algorithms to obtain three different global
clustering partitions (CPs). Then, an unanimous voting strategy is used to
generate the local clustering partition (LCP) of visible layer data. Hence,
the LCP only has partial visible layer data. The visual example is shown in
Fig. 2. The novel MIRBM model is a core feature encoding part of the proposed
unsupervised feature learning architecture. The novelty of it is that the LCP
as an unsupervised guidance is integrated into the $\textbf{{{CD}}}_{1}$
learning to guide the distribution of the hidden layer features. For the
instance in the same LCP cluster, the hidden and reconstructed hidden layer
features of the MIRBM model in the proposed architecture tend to constrict
together in the training process. Meanwhile, each LCP center tends to disperse
from each other as much as possible in the hidden and reconstructed hidden
layer during training. As far as we know, this is the first work to use the
LCP as as an unsupervised guidance to guide the distribution of the hidden
layer features of the proposed MIRBM model in the unsupervised feature
learning architecture. The contributions of our work are summarized below.

  * •

A novel unsupervised feature learning architecture is proposed, which consists
of a multi-clustering integration module and an MIRBM model. The multi-
clustering integration module is used to obtain unsupervised guidance
information. The MIRBM model is a core feature encoding part to extract hidden
layer features.

  * •

In the multi-clustering integration module of the proposed architecture, three
unsupervised algorithms are employed to obtain three different global CPs
without any background knowledge or label. The LCP is generated by the
unanimous voting strategy from three different CPs.

  * •

The MIRBM model in the proposed architecture uses the LCP as an unsupervised
guidance to guide the distribution of the hidden layer features by integrating
the LCP into the $\textbf{{{CD}}}_{1}$ learning. For the instance in the same
LCP cluster, the hidden and reconstructed hidden layer features of the MIRBM
modle in the proposed architecture tend to constrict together in the training
process. Meanwhile, each LCP center tends to disperse from each other as much
as possible in the hidden and reconstructed hidden layer during training.

  * •

It is demonstrated that the proposed architecture has more powerful feature
representation and generalization capability than the state-of-the-art
GraphRBM in the Microsoft Research Asia Multimedia (MSRA-MM)2.0 dataset for
image clustering task.

The remaining of the paper is organized as follows. The literature review is
provided in Section II. In Section III, the theoretical background is
described. An unsupervised feature learning architecture together with the
MIRBM model are proposed in Section IV. The experimental results are shown in
Section V. Finally, we conclude in Section VI and discuss a few correlative
future work.

##  II Literature Review

In this section, we review literature on supervised, semi-supervised,
unsupervised feature learning based on RBMs and other models, together with
the voting strategy in supervised learning.
Supervised feature learning has proved to be an effective method in machine
learning [7], [28], [8], [29], [30]. Amer et al. [28] proposed a Multimodal
Discriminative CRBMs (MMDCRBMs) model based on a Conditional RBMs (an
extension of the RBM). Its training process is composed of training each
modality using labeled data and training a fusion layer. For multi-modality
deep learning, Bu et al. [31] developed a supervised 3D feature learning
framework in which a RBM is used to mine the deep correlations of different
modalities. Cheng et al. [32] presented a novel duplex metric learning (DML)
framework for feature learning and image classification. The main task of DML
is to learn an effective hidden layer feature of a discriminative stacked
autoencoder (DSAE). In the feature space of the DSAE, similar and dissimilar
samples are mapped close to each other and further apart, respectively. This
framework is the most related work to our study, but it belongs to supervised
feature learning with a DSAE by layer-wisely imposing metric learning method
and it is applied to image classification tasks.
However, supervision information, e.g., labels, is scarce and it is expensive
to obtain more labels in many applications. So, some works [33], [34], [23]
explored semi-supervised feature learning which only needs a small number of
labels. Chu et al. [23] presented a pcGRBM model by fusing pairwise
constraints into the reconstructed visible layer for clustering tasks. To
mitigate the burden of annotation, Yesilbek and Sezgin [35] applied self-
learning methods to build a system that can learn from large amounts of
unlabeled data and few labeled examples for sketch recognition. The systems
perform self-learning by extending a small labeled set with new examples which
are extracted from unlabeled sketches. Chen et al. [34] developed a deep
sparase auto-encoder network with supervised fine-tuning and unsupervised
layer-wise self-learning for fault identification.
The class RBMs have powerful unsupervised feature learning capability. To
exploit more powerful feature leaning ability, many unsupervised feature
learning approaches based on the RBMs have been proposed by previous
researches [36], [37], [38], [39], [40], [9], [41]. Chopra and Yadav [39]
presented a unique technique to extract fault feature from the noisy acoustic
signal by an unsupervised RBM. Zhang et al. [40] proposed unsupervised feature
learning based on recursive autoencoders network (RAE) for image
classfication. They used the spectral and spatial information from original
data to produce high-level features. Xie et al. [41] showed a novel approach
to optimize RBM pre-training by capturing principal component directions of
the input with principal component analysis. Al-Dmour and Al-Ani [42] proposed
a fully-automatic segmentation algorithm in which a neural network (NN) model
is used to extract the features of the brain tissue image and is trained using
clustering labels produced by three clustering algorithms. The obtained
classes are combined by majority voting. The study is closely related to our
work, but our encoding framework based on RBMs is guided by self-learning
local supervisions which stem from unsupervised clustering algorithms and
unanimous voting strategy. More specifically, these self-learning local
supervisions from visible layers are integrated into the CD learning of RBMs
to constrict and disperse the distribution of the hidden layer features and
reconstructed hidden layer features. Stewart and Ermon [43] presented a new
technique to supervise NN by prior domain knowledge for computer vision tasks.
It is a related work to our study. However, their work faces to a
convolutional neural network (CNN) and requires large amounts of prior domain
knowledge and how to encode prior knowledge into loss functions of a CNN is a
new challenge.
Two existing voting strategies are often used to supervised learning in
previous researches. One is the max-voting scheme. For example, Azimi et al.
[44] developed a deep learning method for low carbon steel microstructural
classification via fully CNN (FCNN) accompanied by a max-voting scheme. The
other is the majority voting scheme. For example, Seera et al. [45] applied a
recurrent NN (RNN) to extract features from the Transcranial Doppler (TCD)
signals for classification tasks. This work proposed an ensemble RNN model in
which the majority voting scheme is used to combine the single RNN
predictions. Recently various voting classifiers using majority voting have
been proposed to enhance the performance of the classification [46], [47],
[48], [49], [50], [51].
Chen et al. [9] illustrated a new graph regularized RBM (GraphRBM) to extract
hidden layer representations for unsupervised clustering and classification
problem. Meanwhile, they have considered the manifold structure of the image
data. The GraphRBM is a state-of-the-art unsupervised feature learning model
and it is also the most relevant work to our method. Hence, we compare our
unsupervised feature learning architecture with the GraphRBM in the
experiments.

##  III Theoretical Background

###  III-A Restricted Boltzmann Machine

A RBM [15] consists of two-layer structure: a visible layer and a hidden layer
with stochastic binary units via symmetrically weighted connections. It has no
interior-layer connections both between the visible layer units and between
the hidden layer units. An energy function of a joint distribution of the
visible layer and hidden layer units takes the form:

$E(\textbf{v},\textbf{h})=-\sum\limits_{i\in
visibles}a_{i}v_{i}-\sum\limits_{j\in
hiddens}b_{j}h_{j}-\sum\limits_{i,j}v_{i}h_{j}w_{ij},$ (1)

where v is the visible layer vector and $v_{i}$ is the binary states of
visible unit $i$, h is the hidden layer vector and $h_{j}$ is the binary
states of hidden unit $j$, $a_{i}$ and $b_{j}$ are the biases of visible layer
and hidden layer respectively, $w_{ij}$ is the symmetric connection weight
between $v_{i}$ and $h_{j}$.
The probability distribution over a vector $\mathbf{v}$ and with the
parameters $\theta=\\{\textbf{a},\textbf{b},\textbf{W}\\}$ takes the form:

$p(\textbf{v},\theta)=\frac{e^{-E(\textbf{v},\theta)}}{Z(\theta)}$ (2)

where $Z(\theta)=\sum\limits_{\textbf{v}}e^{-E(\textbf{v},\theta)}$ is a
normalisation constant, a is a vector of the visible layer biases, b is a
vector of the hidden layer biases, W is connection matrix. The conditional
probability distributions of hidden layer and visible layer units of the RBM
are given by:

$p(h_{j}=1|\textbf{v})=\sigma(b_{j}+\sum\limits_{i}v_{i}w_{ij})$ (3)

and

$p(v_{i}=1|\textbf{h})=\sigma(a_{i}+\sum\limits_{j}h_{j}w_{ij}),$ (4)

where $\sigma$ is the sigmoid function.

###  III-B Gaussian Linear Visible Units

The classical RBM was designed with binary units for both the hidden and
visible layers [17]. For training real-valued data, the visible layer of RBM
consists of Gaussian linear units and the hidden layer of RBM is still binary
units. The energy function of RBM with Gaussian linear visible units takes the
form:

$E(\textbf{v},\textbf{h})=-\sum\limits_{i\in
visibles}\frac{{(v_{i}-a_{i})}^{2}}{2\sigma_{i}^{2}}-\sum\limits_{j\in
hiddens}b_{j}h_{j}-\sum\limits_{i,j}\frac{v_{i}}{\sigma_{i}}h_{j}w_{ij},$ (5)

where $\sigma_{i}$ is the standard deviation of visible unit $i$ with Gaussian
noise. In the visible layer, the conditional probability is defined by:

$\displaystyle
p(\textbf{v}|\textbf{h})=\mathcal{N}(\sum\textbf{h}\textbf{W}^{T}+,\sigma^{2}),$
(6)

where $\mathcal{N}(\cdot)$ represents gaussian density
($\mu=\sum\textbf{h}\textbf{W}^{T}+\textbf{a}$). The update rules of the
parameters become simple when the linear visible units have unit variance of
Gaussian noise. Then, the reconstructed values of Gaussian linear visible
units are equal to their top-down input values from the binary hidden units
plus their bias.

###  III-C Contrastive Divergence Learning

To learn the parameters of symmetric connection weight of the RBM by Maximum-
likelihood (ML) learning [17], the update rule is given by:

$\Delta w_{ij}=\varepsilon(<v_{i}h_{j}>_{0}-<v_{i}h_{j}>_{\infty}),$ (7)

where $\varepsilon$ is a learning rate, the angle brackets denote the
expectations of the distribution, $<v_{i}h_{j}>_{\infty}$ represents the
expectations under the distribution of the RBM model. But it is very hard to
obtain unbiased sample of $<v_{i}h_{j}>_{\infty}$.
So, a faster learning algorithm [17] was proposed by applying approximation of
the gradient of CD. Karakida et al. [27] demonstrated that
$\textbf{{{CD}}}_{1}$ learning is simpler than Maximum-likelihood (ML)
learning in RBMs. The $\textbf{{CD}}_{1}$ learning follows the gradient of the
difference of two divergences approximately as follows:

$\displaystyle\texttt{CD}_{1}=\texttt{KL}(p_{0}||p_{\infty})-\texttt{KL}(p_{1}||p_{\infty}),$
(8)

where
$\texttt{KL}(p_{0}||p_{\infty})=\sum\limits_{\textbf{v}}p_{0}(\textbf{v})\texttt{log}\frac{p_{0}(\textbf{v})}{p(\textbf{v};\theta})$
and
$\texttt{KL}(p_{1}||p_{\infty})=\sum\limits_{\textbf{v}}p_{1}(\textbf{v})\texttt{log}\frac{p_{1}(\textbf{v})}{p(\textbf{v};\theta)}$
are Kullback-Leibler divergences, $p_{0}$ is the distribution of the data,
$p_{1}$ is the first step distribution of the Markov chain and $p_{\infty}$ is
the distribution of the model. Then the change of symmetric connection weight
with $\textbf{{{CD}}}_{1}$ learning is given by:

$\Delta w_{ij}=\varepsilon(<v_{i}h_{j}>_{0}-<v_{i}h_{j}>_{1}),$ (9)

where the hidden layer units are driven by visible data, $<v_{i}h_{j}>_{0}$
denotes the expectations under the distribution of visible and hidden layer
units, $<v_{i}h_{j}>_{1}$ represents the expectations under the distribution
of reconstructed visible and hidden layer units. Similarly, the changes of
biases $a_{i}$ and $b_{j}$ with $\textbf{{{CD}}}_{1}$ learning are given by:

$\Delta a_{i}=\varepsilon(<v_{i}>_{0}-<v_{i}>_{1})$ (10)

and

$\Delta b_{j}=\varepsilon(<h_{j}>_{0}-<h_{j}>_{1}).$ (11)

So, the update rules of all parameters take the form

$w_{ij}^{(\tau+1)}=w_{ij}^{(\tau)}+\varepsilon(<v_{i}h_{j}>_{0}-<v_{i}h_{j}>_{1}),$
(12) $a_{i}^{(\tau+1)}=a_{i}^{(\tau)}+\varepsilon(<v_{i}>_{0}-<v_{i}>_{1})$
(13)

and

$b_{j}^{(\tau+1)}=b_{j}^{(\tau)}+\varepsilon(<h_{j}>_{0}-<h_{j}>_{1}).$ (14)

The learning efficiency can be obviously improved by the $\textbf{{{CD}}}_{1}$
learning.

##  IV Unsupervised Feature Learning Architecture with MIRBM

In this section, we present a novel unsupervised feature learning
architecture. The MIRBM model as a core part of the architecture is used to
extract hidden layer features. Then, the detailed inferences for the update
rules of the MIRBM model parameters are listed. We design a learning algorithm
of the MIRBM model and analyse its convergence and complexity.

###  IV-A Architecture

To explore more effective unsupervised feature learning method, we construct a
novel feature learning architecture, which consists of a multi-clustering
integration module and an MIRBM model. It is shown in Fig. 1. The multi-
clustering integration module is used to generate an unsupervised guidance LCP
for the next MIRBM model. Three unsupervised K-means, AP and SC algorithms are
applied to obtain three CPs, then an unanimous voting strategy is used to
generate the LCP. An example of multi-clustering integration module is shown
in Fig. 2.
The MIRBM model is a most important part of the architecture. The unsupervised
LCP information is integrated into the $\textbf{{{CD}}}_{1}$ learning to guide
the train process of it. With the LCP assistance, the hidden and reconstructed
hidden layer features of MIRBM model tend to constrict together in the
training process for the instance in the same LCP cluster. Meanwhile, each LCP
center tends to disperse from each other as much as possible in the hidden and
reconstructed hidden layer during training.

Figure 1: Unsupervised feature learning architecture with MIRBM model. The
multi-clustering integration module is used to obtain unsupervised guidance
information LCP by three unsupervised clustering algorithms and unanimous
voting strategy. The MIRBM model as a core feature encoding part uses
unsupervised LCP to guide the distribution of the hidden layer features by
integrating the LCP into its training process of $\textbf{{{CD}}}_{1}$
learning.  Figure 2: An example of multi-clustering integration module. Left:
three kinds of different CPs which contain all instances are generated by
K-means, AP and SC algorithms. Right: an unanimous voting strategy is used to
produce the LCP with three kinds of CPs. The LCP only contains a portion of
the instance and three dots represent three local centers of it.

###  IV-B The MIRBM model

TABLE I: List of symbols.

Notation | Definition
---|---
$\texttt{V}_{data}$ | Visible layer data set
$\texttt{H}_{data}$ | Hidden layer feature set
$\texttt{V}_{recon}$ | Reconstructed visible layer set
$\texttt{H}_{recon}$ | Hidden layer feature of reconstructed visible layer set
$\mathbf{\textbf{v}}_{s},\mathbf{\textbf{v}}_{t}$ | Visible layer row vector
$\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}$ | Hidden layer feature row vector
$\mathbf{\widetilde{\textbf{v}}}_{s},\mathbf{\widetilde{\textbf{v}}}_{t}$ | Reconstructed visible layer row vector
$\mathbf{\widetilde{\textbf{h}}}_{s},\mathbf{\widetilde{\textbf{h}}}_{t}$ | Hidden layer feature row vector of reconstructed data
$\texttt{V}_{k}$ | All vectors of $\texttt{V}_{k}\subset\texttt{V}_{data}$ belonging to the same cluster.
$\widetilde{\texttt{V}}_{k}$ | All vectors of $\widetilde{\texttt{V}}_{k}\subset\texttt{V}_{recon}$ belonging to the same cluster.
$\texttt{H}_{k}$ | All vectors of $\texttt{H}_{k}\subset\texttt{H}_{data}$ belonging to the same cluster.
$\widetilde{\texttt{H}}_{k}$ | All vectors of $\widetilde{\texttt{H}}_{k}\subset\texttt{H}_{recon}$ belonging to the same cluster.
$\textbf{C}_{k}$ | The center of cluster $\texttt{H}_{k}$
$\widetilde{\textbf{C}}_{k}$ | The center of cluster $\widetilde{\texttt{H}}_{k}$
$\textbf{O}_{k}$ | The center of cluster $\texttt{V}_{k}$
$\widetilde{\textbf{O}}_{k}$ | The center of cluster $\widetilde{\texttt{V}}_{k}$

Suppose that
$\texttt{V}_{data}=\\{\mathbf{\textbf{v}_{1}},\mathbf{\textbf{v}_{2}},\cdots,\mathbf{\textbf{v}_{N}}\\}$
is the original data set.
$\texttt{H}_{data}=\\{\mathbf{\textbf{h}_{1}},\mathbf{\textbf{h}_{2}},\cdots,\mathbf{\textbf{h}_{N}}\\}$
is the hidden layer feature set.
$\texttt{V}_{recon}=\\{\mathbf{\widetilde{\textbf{v}}_{1}},\mathbf{\widetilde{\textbf{v}}_{2}},\cdots,\mathbf{\widetilde{\textbf{v}}_{N}}\\}$
is the reconstructed visible layer data set.
$\texttt{H}_{recon}=\\{\mathbf{\widetilde{\textbf{h}}_{1}},\mathbf{\widetilde{\textbf{h}}_{2}},\cdots,\mathbf{\widetilde{\textbf{h}}_{N}}\\}$
is the hidden features set of reconstructed data. Let
$\texttt{V}_{1},\texttt{V}_{2},\cdots\texttt{V}_{K}$ be $K$ local clusters of
visible layer set $\texttt{V}_{data}$, $\texttt{H}_{i}(i=1,2,\cdots,K)$ are
local clusters mapped of $\texttt{V}_{i}(i=1,2,\cdots,K)$, respectively.
Similarly,
$\widetilde{\texttt{V}}_{1},\widetilde{\texttt{V}}_{2},\cdots,\widetilde{\texttt{V}}_{K}$
are $K$ local clusters of reconstructed visible layer set
$\texttt{V}_{recon}$, $\widetilde{\texttt{H}}_{i}(i=1,2,\cdots,K)$ are $K$
local clusters mapped of $\widetilde{\texttt{V}}_{i}(i=1,2,\cdots,K)$,
respectively. We use the gradient descent method to obtain approximate optimal
parameters of the MIRBM model. In the encoding process, we expect that the
hidden features and the reconstructed hidden features of the instance in the
same LCP cluster become more concentrated together. Meanwhile, each LCP center
tends to disperse from each other as much as possible in the hidden and
reconstructed hidden layer during training of $\textbf{{CD}}_{1}$ learning
method. Therefore, the objective function takes form:

$\displaystyle
F(\theta,\texttt{V}_{data})=-\eta\left[\texttt{KL}(p_{0}||p_{\infty})-\texttt{KL}(p_{1}||p_{\infty})\right]+$
(15)
$\displaystyle\Big{(}\frac{1-\eta}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}\Arrowvert\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t}\Arrowvert^{2}-\frac{1-\eta}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}\Arrowvert\mathbf{\textbf{C}}_{p}-$
$\displaystyle\mathbf{\textbf{C}}_{q}\Arrowvert^{2}\Big{)}+\Big{(}\frac{1-\eta}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\widetilde{\mathbf{\textbf{h}}}_{s},\widetilde{\mathbf{\textbf{h}}}_{t}\in\widetilde{\texttt{H}}_{k}}\Arrowvert\widetilde{\mathbf{\textbf{h}}}_{s}-\widetilde{\mathbf{\textbf{h}}}_{t}\Arrowvert^{2}-\frac{1-\eta}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}\Arrowvert\widetilde{\mathbf{\textbf{C}}}_{p}$
$\displaystyle-\widetilde{\mathbf{\textbf{C}}}_{q}\Arrowvert^{2}\Big{)},$

where $\eta\in(0,1)$ is a scale coefficient, $N_{h}$ is the cardinality of
$\texttt{H}_{k}(k=1,2,\cdots,K)$, $N_{C}=\frac{K(K-1)}{2}$ is the number of
pairwise cluster center.
To simplify the expression of the objective function, $L_{data}(\theta)$ and
$L_{recon}(\theta)$ functions take form:

$\displaystyle L_{data}(\theta)=$ (16)
$\displaystyle\frac{1}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}\Arrowvert\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t}\Arrowvert^{2}-\frac{1}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}\Arrowvert\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q}\Arrowvert^{2}$

and

$\displaystyle L_{recon}(\theta)=$ (17)
$\displaystyle\frac{1}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\widetilde{\mathbf{\textbf{h}}}_{s},\widetilde{\mathbf{\textbf{h}}}_{t}\in\widetilde{\texttt{H}}_{k}}\Arrowvert\widetilde{\mathbf{\textbf{h}}}_{s}-\widetilde{\mathbf{\textbf{h}}}_{t}\Arrowvert^{2}-\frac{1}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}\Arrowvert\widetilde{\mathbf{\textbf{C}}}_{p}-\widetilde{\mathbf{\textbf{C}}}_{q}\Arrowvert^{2}.$

So the objective function has another form:

$\displaystyle F(\theta,\texttt{V}_{data})=$
$\displaystyle-\eta\left[\texttt{KL}(p_{0}||p_{\infty})-\texttt{KL}(p_{1}||p_{\infty})\right]$
(18) $\displaystyle+(1-\eta)\left[L_{data}(\theta)+L_{recon}(\theta)\right].$

###  IV-C The Inference

The next hardest problem is how to optimize the objective function:

$\displaystyle\texttt{arg}\mathop{\min}\limits_{\theta}F(\theta,\texttt{V}_{data})$
(19)

The approximate derivative of the
$\texttt{KL}(p_{0}||p_{\infty})-\texttt{KL}(p_{1}||p_{\infty})$ can be
obtained by $\textbf{{{CD}}}_{1}$ learning method, so the next problems are
how to get the gradients of $L_{data}(\theta)$ and $L_{recon}(\theta)$.
Firstly, we compute the gradients of $L_{data}$ as follows. Because
$L_{data}(\theta)$ has another equivalent form:

$\displaystyle L_{data}(\theta)$
$\displaystyle=\frac{1}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}$
(20)
$\displaystyle-\frac{1}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})^{T}.$

Then we can obtain:

$\displaystyle\frac{\partial L_{data}(\theta)}{\partial w_{ij}}$
$\displaystyle=\frac{1}{N_{h}}\bigg{[}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}}{\partial
w_{ij}}+$ (21)
$\displaystyle\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})}{\partial
w_{ij}}(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}\bigg{]}-$
$\displaystyle\frac{1}{N_{C}}\bigg{[}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})\frac{\partial(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})^{T}}{\partial
w_{ij}}+$
$\displaystyle\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}\frac{\partial(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})}{\partial
w_{ij}}(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})^{T}\bigg{]}.$

From above result, we can see that the following task is how to compute
$\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}}{\partial
w_{ij}}$,
$\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})}{\partial
w_{ij}}$,
$\frac{\partial(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})^{T}}{\partial
w_{ij}}$ and
$\frac{\partial(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})}{\partial
w_{ij}}$. Next, all of them are solved separately.

$\displaystyle\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}}{\partial
w_{ij}}$ $\displaystyle=(\frac{\partial\mathbf{\textbf{h}}_{s}}{\partial
w_{ij}}-\frac{\partial\mathbf{\textbf{h}}_{t}}{\partial w_{ij}})^{T}$ (22)
$\displaystyle=\bigg{[}\frac{\partial(\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W}))}{\partial
w_{ij}}-\frac{\partial(\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{t}\mathbf{W}))}{\partial
w_{ij}}\bigg{]}^{T}.$

Obviously, $\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})$ is a row
vector, all components of which are independent of $w_{ij}$ except $j$
component. So,

$\displaystyle\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})}{\partial
w_{ij}}=$ (23)
$\displaystyle(\underbrace{0,\cdots,0}_{j-1},\frac{\partial\sigma(b_{j}+\sum\limits_{i=1}^{n}v_{si}w_{ij})}{\partial
w_{ij}},\underbrace{0,\cdots,0}_{n-j}).$

Because
$\frac{\partial\sigma(b_{j}+\sum\limits_{i=1}^{n}v_{si}w_{ij})}{\partial
w_{ij}}=h_{sj}(1-h_{sj})v_{si}$, the final result of
$\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})}{\partial
w_{ij}}$ has an expression as follows:

$\displaystyle\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})}{\partial
w_{ij}}=(\underbrace{0,\cdots,0}_{j-1},h_{sj}(1-h_{sj})v_{si},\underbrace{0,\cdots,0}_{n-j}).$
(24)

Similarly, the expression of final result of
$\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{t}\mathbf{W})}{\partial
w_{ij}}$ is as follows:

$\displaystyle\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{t}\mathbf{W})}{\partial
w_{ij}}=(\underbrace{0,\cdots,0}_{j-1},h_{tj}(1-h_{tj})v_{ti},\underbrace{0,\cdots,0}_{n-j}).$
(25)

Then, the final result of
$\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}}{\partial
w_{ij}}$ is a column vector:

$\displaystyle\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})^{T}}{\partial
w_{ij}}=$ (26)
$\displaystyle(\underbrace{0,\cdots,0}_{j-1},h_{sj}(1-h_{sj})v_{si}-h_{tj}(1-h_{tj})v_{ti},\underbrace{0,\cdots,0}_{n-j})^{T}.$

Similarly,

$\displaystyle\frac{\partial(\mathbf{\textbf{h}}_{s}-\mathbf{\textbf{h}}_{t})}{\partial
w_{ij}}=$ (27)
$\displaystyle(\underbrace{0,\cdots,0}_{j-1},h_{sj}(1-h_{sj})v_{si}-h_{tj}(1-h_{tj})v_{ti},\underbrace{0,\cdots,0}_{n-j}),$
$\displaystyle\frac{\partial(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})^{T}}{\partial
w_{ij}}=$ (28)
$\displaystyle(\underbrace{0,\cdots,0}_{j-1},C_{pj}(1-C_{pj})O_{pi}-C_{qj}(1-C_{qj})O_{qi},\underbrace{0,\cdots,0}_{n-j})^{T}$

and

$\displaystyle\frac{\partial(\mathbf{\textbf{C}}_{p}-\mathbf{\textbf{C}}_{q})}{\partial
w_{ij}}=$ (29)
$\displaystyle(\underbrace{0,\cdots,0}_{j-1},C_{pj}(1-C_{pj})O_{pi}-C_{qj}(1-C_{qj})O_{qi},\underbrace{0,\cdots,0}_{n-j}).$

Eqs. (26), (27), (28) and (29) are substituted in Eq. (21). Then

$\displaystyle\frac{\partial L_{data}(\theta)}{\partial w_{ij}}=$ (30)
$\displaystyle\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}(h_{sj}-h_{tj})\bigg{[}h_{sj}(1-h_{sj})v_{si}-h_{tj}(1-h_{tj})v_{ti}\bigg{]}$
$\displaystyle-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(C_{pj}-C_{qj})\bigg{[}C_{pj}(1-C_{pj})O_{pi}-$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad
C_{qj}(1-C_{qj})O_{qi}\bigg{]}.$

Using above same solution, we can obtain:

$\displaystyle\frac{\partial L_{recon}(\theta)}{\partial w_{ij}}=$ (31)
$\displaystyle\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\widetilde{\mathbf{\textbf{h}}}_{s},\widetilde{\mathbf{\textbf{h}}}_{t}\in\widetilde{\texttt{H}_{k}}}(\widetilde{h}_{sj}-\widetilde{h}_{tj})\bigg{[}\widetilde{h}_{sj}(1-\widetilde{h}_{sj})\widetilde{v}_{si}-\widetilde{h}_{tj}(1-\widetilde{h}_{tj})\widetilde{v}_{ti}\bigg{]}$
$\displaystyle-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(\widetilde{C}_{pj}-\widetilde{C}_{qj})\bigg{[}\widetilde{C}_{pj}(1-\widetilde{C}_{pj})\widetilde{o}_{pi}-$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\widetilde{C}_{qj}(1-\widetilde{C}_{qj})\widetilde{o}_{qi}\bigg{]}.$

The following task is how to obtain $\frac{\partial
L_{recon}(\theta)}{\partial b_{j}}$ and $\frac{\partial
L_{recon}(\theta)}{\partial b_{j}}$. Because
$\frac{\partial\sigma(b_{j}+\sum\limits_{i=1}^{n}v_{si}w_{ij})}{\partial
b_{j}}=h_{sj}(1-h_{sj})$ and
$\frac{\partial\sigma(b_{j}+\sum\limits_{i=1}^{n}v_{ti}w_{ij})}{\partial
b_{j}}=h_{tj}(1-h_{tj})$, the final result of
$\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})}{\partial
b_{j}}$ and
$\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{t}\mathbf{W})}{\partial
b_{j}}$ have expressions as follows:

$\displaystyle\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})}{\partial
b_{j}}=(\underbrace{0,\cdots,0}_{j-1},h_{sj}(1-h_{sj}),\underbrace{0,\cdots,0}_{n-j})$
(32)

and

$\displaystyle\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{t}\mathbf{W})}{\partial
b_{j}}=(\underbrace{0,\cdots,0}_{j-1},h_{tj}(1-h_{tj}),\underbrace{0,\cdots,0}_{n-j}).$
(33)

So, the final result of $\frac{\partial L_{data}(\theta)}{\partial b_{j}}$ is
as follows.

$\displaystyle\frac{\partial L_{data}(\theta)}{\partial b_{j}}=$ (34)
$\displaystyle\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}(h_{sj}-h_{tj})\bigg{[}h_{sj}(1-h_{sj})-h_{tj}(1-h_{tj})\bigg{]}$
$\displaystyle-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(C_{pj}-C_{qj})\bigg{[}C_{pj}(1-C_{pj})-$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad
C_{qj}(1-C_{qj})\bigg{]}.$

Similarly, the expression of $\frac{\partial L_{recon}(\theta)}{\partial
b_{j}}$ is as follows.

$\displaystyle\frac{\partial L_{recon}(\theta)}{\partial b_{j}}=$ (35)
$\displaystyle\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\widetilde{\mathbf{\textbf{h}}}_{s},\widetilde{\mathbf{\textbf{h}}}_{t}\in\widetilde{\texttt{H}}_{k}}(\widetilde{h}_{sj}-\widetilde{h}_{tj})\bigg{[}\widetilde{h}_{sj}(1-\widetilde{h}_{sj})-\widetilde{h}_{tj}(1-\widetilde{h}_{tj})\bigg{]}$
$\displaystyle-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(\widetilde{C}_{pj}-\widetilde{C}_{qj})\bigg{[}\widetilde{C}_{pj}(1-\widetilde{C}_{pj})-$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\widetilde{C}_{qj}(1-\widetilde{C}_{qj})\bigg{]}.$

Because
$\frac{\partial\sigma(b_{j}+\sum\limits_{i=1}^{n}v_{si}w_{ij})}{\partial
a_{i}}=0$ and
$\frac{\partial\sigma(b_{j}+\sum\limits_{i=1}^{n}v_{ti}w_{ij})}{\partial
a_{i}}=0$, the final results of
$\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{s}\mathbf{W})}{\partial
a_{i}}$ and
$\frac{\partial\sigma(\mathbf{b}+\mathbf{\textbf{v}}_{t}\mathbf{W})}{\partial
a_{i}}$ are zero vectors. Then we can obtain: $\frac{\partial
L_{data}(\theta)}{\partial a_{i}}=0,\frac{\partial L_{recon}(\theta)}{\partial
a_{i}}=0$.
Finally, following the $\textbf{{CD}}_{1}$ learning and the gradient of the
$L_{data}$ and $L_{recon}$, the update rules of the symmetric connection
weight take the form:

$\displaystyle
w_{ij}^{(\tau+1)}=w_{ij}^{(\tau)}+\eta\varepsilon(<v_{i}h_{j}>_{0}-<v_{i}h_{j}>_{1})+(1-\eta)$
(36)
$\displaystyle\Bigg{\\{}\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}(h_{sj}-h_{tj})\bigg{[}h_{sj}(1-h_{sj})v_{si}-h_{tj}(1-$
$\displaystyle
h_{tj})v_{ti}\bigg{]}-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(C_{pj}-C_{qj})\bigg{[}C_{pj}(1-C_{pj})O_{pi}$
$\displaystyle-
C_{qj}(1-C_{qj})O_{qi}\bigg{]}\Bigg{\\}}+(1-\eta)\Bigg{\\{}\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\widetilde{\mathbf{\textbf{h}}}_{s},\widetilde{\mathbf{\textbf{h}}}_{t}\in\widetilde{\texttt{H}_{k}}}(\widetilde{h}_{sj}$
$\displaystyle-\widetilde{h}_{tj})\bigg{[}\widetilde{h}_{sj}(1-\widetilde{h}_{sj})\widetilde{v}_{si}-\widetilde{h}_{tj}(1-\widetilde{h}_{tj})\widetilde{v}_{ti}\bigg{]}-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}$
$\displaystyle(\widetilde{C}_{pj}-\widetilde{C}_{qj})\bigg{[}\widetilde{C}_{pj}(1-\widetilde{C}_{pj})\widetilde{o}_{pi}-\widetilde{C}_{qj}(1-\widetilde{C}_{qj})\widetilde{o}_{qi}\bigg{]}\Bigg{\\}}.$

The update rules of biases $b_{j}$ take the form:

$\displaystyle
b_{j}^{(\tau+1)}=b_{j}^{(\tau)}+\eta\varepsilon(<h_{j}>_{0}-<h_{j}>_{1})+(1-\eta)$
(37)
$\displaystyle\Bigg{\\{}\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\mathbf{\textbf{h}}_{s},\mathbf{\textbf{h}}_{t}\in\texttt{H}_{k}}(h_{sj}-h_{tj})\bigg{[}h_{sj}(1-h_{sj})-h_{tj}(1-h_{tj})\bigg{]}$
$\displaystyle-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(C_{pj}-C_{qj})\bigg{[}C_{pj}(1-C_{pj})-$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad
C_{qj}(1-C_{qj})\bigg{]}\Bigg{\\}}+(1-\eta)$
$\displaystyle\Bigg{\\{}\frac{2}{N_{h}}\sum\limits_{k=1}^{K}\sum\limits_{\widetilde{\mathbf{\textbf{h}}}_{s},\widetilde{\mathbf{\textbf{h}}}_{t}\in\widetilde{\texttt{H}_{k}}}(\widetilde{h}_{sj}-\widetilde{h}_{tj})\bigg{[}\widetilde{h}_{sj}(1-\widetilde{h}_{sj})-\widetilde{h}_{tj}(1-\widetilde{h}_{tj})\bigg{]}$
$\displaystyle-\frac{2}{N_{C}}\sum\limits_{p=1}^{K-1}\sum\limits_{q=p+1}^{K}(\widetilde{C}_{pj}-\widetilde{C}_{qj})\bigg{[}\widetilde{C}_{pj}(1-\widetilde{C}_{pj})-$
$\displaystyle\qquad\qquad\qquad\qquad\qquad\qquad\qquad\widetilde{C}_{qj}(1-\widetilde{C}_{qj})\bigg{]}\Bigg{\\}}$

and the update rules of biases $a_{i}$ take the form:

$\displaystyle
a_{i}^{(\tau+1)}=a_{i}^{(\tau)}+\eta\varepsilon(<v_{i}>_{0}-<v_{i}>_{1}).$
(38)

Next, for iterative updating rules Eqs. (36), (37) and (38), the convergence
needs to be analyzed theoretically.

###  IV-D Convergence Analysis

In this subsection, we use the convergence analysis method [52], [53] to prove
that the objective function decreases monotonically by the iterative updating
rules Eqs. (36), (37) and (38).
Theorem 1: In each iteration, the objective function Eq. (15) will
monotonically decrease.
Proof: Assume that $\textbf{a}^{\tau}$, $\textbf{b}^{\tau}$ and
$\textbf{W}^{\tau}$ have been derived in the $\tau$th iteration. We fix
$\textbf{a}^{\tau}$ and $\textbf{b}^{\tau}$ to optimize $\textbf{W}^{\tau}$.
Obviously, according to stochastic gradient descent method, there is the
following inequality:

$\displaystyle
F(\textbf{a}^{\tau},\textbf{b}^{\tau},\textbf{W}^{(\tau+1)},\texttt{V}_{data})\leq
F(\textbf{a}^{\tau},\textbf{b}^{\tau},\textbf{W}^{\tau},\texttt{V}_{data})$
(39)

Then, the parameters W and a are fixed as $\textbf{W}^{(\tau+1)}$ and
$\textbf{a}^{\tau}$ to optimize b. We have another inequality:

$\displaystyle
F(\textbf{a}^{\tau},\textbf{b}^{(\tau+1)},\textbf{W}^{(\tau+1)},\texttt{V}_{data})\leq
F(\textbf{a}^{\tau},\textbf{b}^{\tau},\textbf{W}^{(\tau+1)},\texttt{V}_{data})$
(40)

Similarily, the parameters W and b are fixed as $\textbf{W}^{(\tau+1)}$ and
$\textbf{b}^{(\tau+1)}$ to optimize a. We get the third inequality:

$\displaystyle
F(\textbf{a}^{(\tau+1)},\textbf{b}^{(\tau+1)},\textbf{W}^{(\tau+1)},\texttt{V}_{data})\leq
F(\textbf{a}^{\tau},\textbf{b}^{\tau+1},\textbf{W}^{\tau+1},\texttt{V}_{data})$
(41)

So, from Eqs. (39), (40) and (41), we have

$\displaystyle
F(\textbf{a}^{\tau+1},\textbf{b}^{\tau+1},\textbf{W}^{\tau+1},\texttt{V}_{data})\leq
F(\textbf{a}^{\tau},\textbf{b}^{\tau},\textbf{W}^{\tau},\texttt{V}_{data})$
(42)

Thus, the objective function decreases monotonically by the iterative updating
rules Eqs. (36), (37) and (38). The learning algorithm of the MIRBM model is
summarized as follows.

Learning Algorithm of MIRBM

Input: $\texttt{V}_{data}$ is original data set;
$\eta$ is a scale coefficient;
LCP is the local cluster partition of the $\texttt{V}_{data}$.

Output: The model parameters of a,b and W.

Foreach iteration do
For all visible layer vectors do
Sample hidden layer distribution by Eq. (3);
End For
For all hidden layer vectors do
If visible layer is binary unit do
Sample reconstructed visible layer distribution by
Eq. (4);
Else
Sample reconstructed visible layer distribution by
linear transformation;
End For
Obtain the gradients of Eqs. (30), (31), (34) and (35);
Use Eqs. (36), (37) and (38) to update W, b and
a, respectively;
End For
Return$\mathbf{W}$, $\mathbf{b}$ and $\mathbf{a}$.

###  IV-E Complexity Analysis

Let the iteration times of the MIRBM model learning algorithm be $T_{s}$ and
$\texttt{V}_{data}$ have $N$ vectors. Then, the time complexities of hidden
layer feature sampling and reconstructed visible layer distribution sampling
are both $O(N)$. Suppose that the auxiliary LCP guiding has $K$ clusters and
the $k(k=1,2,\cdots,K)$ cluster has $M_{k}$ instances. To obtain the gradients
of Eqs. (30), (31), (34) and (35), the time complexities are both
$O(K\frac{M_{k}(M_{k}-1)}{2})$. Other step of MIRBM model learning algorithm
need $O(1)$ time complex. Eventually, the time complexity of MIRBM model
learning algorithm is $O(T_{s}(N+KM_{k}(M_{k}-1)))$.

##  V Experiment Results

In this section, we compare the proposed unsupervised feature learning
architecture with the state-of-the-art GraphRBM using image clustering task.
The experimental data include fifteen real-valued image datasets. Three groups
nine algorithms are used to prove the hidden feature learning capability and
generalization ability of our architecture. We choose four external evaluation
metrics to evaluate the clustering performance.

###  V-A Datasets

To compare with the state-of-the-art GraphRBM, we choose fifteen real-valued
image datasets from from the Microsoft Research Asia Multimedia (MSRA-MM)2.0
[54]. The summary of the experiment datasets is shown in Table II.

TABLE II: Summary of the experiment datasets.

No. | Dataset | classes | Instances | feature
---|---|---|---|---
1 | alphabet | 3 | 814 | 892
2 | americanflag | 3 | 873 | 892
3 | aquarium | 3 | 922 | 892
4 | baobab | 3 | 900 | 892
5 | bathroom | 3 | 924 | 892
6 | beret | 3 | 876 | 892
7 | birthdaycake | 3 | 932 | 892
8 | blog | 3 | 943 | 892
9 | blood | 3 | 866 | 892
10 | boat | 3 | 857 | 892
11 | boomerang | 3 | 910 | 892
12 | building | 3 | 911 | 892
13 | wallpaper | 3 | 919 | 899
14 | weapon | 3 | 858 | 899
15 | weddingdress | 3 | 883 | 899

###  V-B Experimental Settings

To compare our architecture with the state-of-the-art GraphRBM, we choose
fifteen real-valued image datasets in the experiments. In the MIRBM model of
the proposed architecture, the visible layer units are set to Gaussian linear
visible units for real-valued data sets. A linear transformation is adopted in
the process of visible layer reconstruction. To compare fairness, the learning
rate of our MIRBM model and the GraphRBM is both set to $10^{-2}$. One of the
key parameters $\eta$ is set to $0.1,0.2,\cdots,0.9$ to analysis the
sensitivity.
We choose three groups nine clustering algorithms to evaluate the hidden layer
feature learning ability and generalization ability of our feature learning
architecture. The first group is K-means [55], affinity propagation (AP) [56]
and spectral clustering (SC)[57] algorithms using the original data as their
input. The second group is K-means+GraphRBM, AP+GraphRBM and SC+GraphRBM
algorithms based on the GraphRBM using the hidden layer features of the
GraphRBM as their input. The last group is K-means+MIRBM, AP+MIRBM and
SC+MIRBM algorithms based on our architecture using the hidden layer features
of the MIRBM model as their input.

###  V-C Evaluation Metrics

To prove the hidden layer feature learning capability of the proposed
architecture, we choose four external evaluation metrics accuracy [58], purity
[59], Fowlkes and Mallows Index [60] and Jaccard Index[61] [62]) to evaluate
the clustering performance.

####  V-C1 Accuracy

Given an instance $x_{i}$, let $h_{i}$ and $g_{i}$ be the cluster label and
the true label, respectively. The clustering accuracy is given by:

$\displaystyle AC=\frac{\sum\limits_{i=1}\delta(g_{i},map(h_{i}))}{n},$ (43)

where $AC\in[0,1]$, $n$ is the total number of instances, $map(h_{i})$ maps
each cluster label $h_{i}$ to the equivalent label from the data set, and
$\delta(x,y)$ equals to one if $x=y$ and equals to zero otherwise.

####  V-C2 Purity

The purity is used to measure the extent of each cluster contained data points
from primarily one class. It is an external transparent evaluation measure for
cluster quality. The purity is given by:

$purity=\sum_{i=1}^{K}\frac{n_{i}}{n}P(S_{i}),P(S_{i})=\frac{1}{n_{i}}\max_{j}(n_{i}^{j}),$
(44)

where $n_{i}^{j}$ is the number of the $i$-th input class that is assigned to
the $j$-th cluster and $S_{i}$ is a particular cluster size of $n_{i}$.

####  V-C3 Fowlkes and Mallows Index

The Fowlkes and Mallows index is an external evaluation method for two
clusterings that can be defined as:

$\displaystyle\texttt{{FMI}}=\sqrt{\frac{TP}{TP+FP}\times\frac{TP}{TP+FN}},$
(45)

where $TP$, $FP$ and $FN$ are the number of true positives, false positives
and false negatives, respectively.

####  V-C4 Jaccard Index

The Jaccard Index as an external evaluation method is used to gauge the
diversity and similarity of sample sets. Given two sets $A$ and $B$, each of
them has $n$ attributes. The total number of each combination of attributes is
defined by:

  * •

$M_{a}$ represents the total number of attributes, if $A$ and $B$ have a value
of 1.

  * •

$M_{b}$ represents the total number of attributes, if the attribute of $A$ has
a value 0 and $B$ has a value of 1.

  * •

$M_{c}$ represents the total number of attributes, if the attribute of $A$ has
a value 1 and $B$ has a value of 0.

The Jaccard Index, $J$, is given as:

$\displaystyle J=\frac{M_{a}}{M_{b}+M_{c}+M_{a}}.$ (46)

###  V-D Performance Comparison

In this subsection, we use three groups of nine clustering algorithms to
compare the state-of-the-art GraphRBM and our unsupervised feature learning
architecture.

####  V-D1 Accuracy Comparison

In Table III, we list the results of the accuracy of three groups contrastive
algorithms. The first group is K-means, AP and SC algorithm using original
data as input. The second group is K-means+GraphRBM, AP+GraphRBM and
SC+GraphRBM algorithm using the hidden layer features of the GraphRBM as
input. The final group is K-means+MIRBM, AP+MIRBM and SC+MIRBM algorithm using
the hidden layer features of our MIRBM model as input. From Table III, we can
see that the best performances distribute over the algorithms based on our
architecture. For some datasets (e.g. blog, boomerang, wallpaper and
weddingdress), K-means+MIRBM algorithm shows the best performance. And the
SC+MIRBM algorithm shows the best performance in the datasets baobab and
birthdaycake. For other datasets, the AP+MIRBM algorithm shows the best
performance. For all datasets, the average accuracies of K-means, AP and SC
algorithms are 0.4309, 0.4408 and 0.4194, respectively. The K-means+GraphRBM,
AP+GraphRBM and SC+GraphRBM algorithms based on the GraphRBM raise the average
accuracies to 0.4584, 0.4623 and 0.4472, respectively. However, the average
accuracies of K-means+MIRBM, AP+MIRBM and SC+MIRBM algorithms based on our
architecture raise greatly to 0.5783, 0.6031 and 0.5273, respectively. Hence,
the hidden layer features of MIRBM model as the input of K-means, AP and SC
algorithms can enhance the average accuracies by 14.71%, 16.23% and 10.79%,
respectively. In terms of feature representation ability of our architecture
and the GraphRBM, the average accuracies of the K-means+MIRBM, AP+MIRBM,
SC+MIRBM algorithms are 11.99%, 14.08% and 8.01% higher than K-means+GraphRBM,
AP+GraphRBM, SC+GraphRBM algorithms, respectively.
For each dataset, more intuitive performance comparisons are shown in Fig. 3.
There are twelve datasets (alphabet, americanflag, aquarium, baobab, bathroom,
beret, birthdaycake, blog, boat, boomerang, building and wallpaper) show
outstanding performances about three kinds of algorithms based on our
architecture simultaneously.

####  V-D2 Purity Comparison

We report the results of the purity of K-means+GraphRBM, AP+GraphRBM,
SC+GraphRBM algorithms based on the GraphRBM and K-means+MIRBM, AP+MIRBM,
SC+MIRBM algorithms based on our architecture in Table IV. From Table IV, we
can see that the best performances distribute mainly over the algorithms based
on our architecture expect for the baobab and birthdaycake datasets. The
results of average purity of K-means, AP and SC algorithms using original data
as input are 0.7483, 0.7509 and 0.7465, respectively. But, the results of
average purity of K-means+GraphRBM, AP+GraphRBM and SC+GraphRBM algorithms
using the hidden features of GraphRBM as input are raised to 0.7578, 0.7625
and 0.7568, respectively. For our architecture, the results of average Jac of
K-means+MIRBM, AP+MIRBM and SC+MIRBM algorithms using the hidden features of
it as input are significantly raised to 0.7590, 0.7653 and 0.7607,
respectively. As a whole, the proposed architecture has better feature
learning ability than the GraphRBM.
Among three groups contrastive algorithms, the intuitive comparisons of the
average purity results are shown in Fig. 4. All the algorithms based on our
MIRBM model show better performances.

####  V-D3 FMI Comparison

The FMI results of all contrastive algorithms are listed in Table V. From
Table V, the results of average FMI of K-means, AP and SC algorithms using
original data as input are 0.4382, 0.4359 and 0.4238, respectively. They are
raised to 0.4619, 0.4653 and 0.4529, respectively, by the GraphRBM using
hidden layer features of it as input. However, our architecture further
improves the results of average FMI to 0.5977, 0.6322 and 0.5482,
respectively. Clearly, the GraphRBM and our architecture both have powerful
feature learning capabilities. But, our architecture model shows more powerful
hidden layer feature extraction capabilities than the GraphRBM. From Table V,
the best performances of all datasets are distributed over the algorithms
based on the proposed architecture, especially with regard to AP+MIRBM
algorithm.
The intuitive comparisons of the average FMI results among three groups
contrastive algorithms are shown in Fig. 4. The K-means+MIRBM, AP+MIRBM and
SC+MIRBM algorithms based on the proposed architecture show absolute
advantages.

####  V-D4 Jac Comparison

In Table VI, we report the results of the Jac of K-means+GraphRBM,
AP+GraphRBM, SC+GraphRBM algorithms based on the GraphRBM and K-means+MIRBM,
AP+MIRBM, SC+MIRBM algorithms based on the proposed architecture. Meanwhile,
we report the results of the Jac of original K-means, AP and SC algorithms.
From Table VI, we can see that the best performances of all datasets
distribute over the algorithms based on our architecture. In fact, they
distribute mainly over the algorithms AP+MIRBM. For all datasets, the results
of average Jac of K-means, AP and SC algorithms using original data as input
are 0.2748, 0.2749 and 0.2618, respectively. But, the results of average Jac
of K-means+GraphRBM, AP+GraphRBM and SC+GraphRBM algorithms using the hidden
features of GraphRBM as input are raised to 0.2989, 0.3018 and 0.2905,
respectively. Hence, the GraphRBM has certain capability of feature
representation. For the proposed architecture, the results of average Jac of
K-means+MIRBM, AP+MIRBM and SC+MIRBM algorithms using the hidden features of
it as input are significantly raised to 0.4098, 0.4463 and 0.3702,
respectively. Therefore, the proposed architecture shows excellent performance
for Jac evaluation metric.
The intuitive comparisons of the average Jac results among three groups
contrastive algorithms are shown in Fig. 4. All the algorithms based on the
proposed architecture show absolute advantages.

TABLE III: The performances of the accuracies and variance ($\eta=0.1$).

Dataset (No.) | K-means | AP | SC | K-means+GraphRBM | AP+GraphRBM | SC+GraphRBM | K-means+MIRBM | AP+MIRBM | SC+MIRBM
---|---|---|---|---|---|---|---|---|---
alphabet | 0.4066$\pm 0.0002$ | 0.4042$\pm 0.0005$ | 0.4386$\pm 0.0001$ | 0.3808$\pm 0.0003$ | 0.3796$\pm 0.0001$ | 0.3980$\pm 0.0002$ | 0.4530$\pm 0.0003$ | 0.4607$\pm 0.0002$ | 0.4447$\pm 0.0001$
americanflag | 0.4513$\pm 0.0003$ | 0.4376$\pm 0.0001$ | 0.4204$\pm 0.0002$ | 0.4284$\pm 0.0001$ | 0.4318$\pm 0.0002$ | 0.4124$\pm 0.0003$ | 0.5470$\pm 0.0004$ |  0.5670$\pm 0.0001$ | 0.5384$\pm 0.0005$
aquarium | 0.3720$\pm 0.0001$ | 0.3709$\pm 0.0002$ | 0.3970$\pm 0.0002$ | 0.5293$\pm 0.0001$ | 0.5315$\pm 0.0003$ | 0.5119$\pm 0.0001$ | 0.6817$\pm 0.0001$ | 0.7017$\pm 0.0003$ | 0.6941$\pm 0.0002$
baobab | 0.4989$\pm 0.0002$ | 0.5511$\pm 0.0004$ | 0.4822$\pm 0.0001$ | 0.4722$\pm 0.0006$ | 0.4778$\pm 0.0003$ | 0.4456$\pm 0.0002$ | 0.5900$\pm 0.0004$ | 0.5478$\pm 0.0002$ | 0.6078$\pm 0.0005$
bathroom | 0.3723$\pm 0.0003$ | 0.4221$\pm 0.0001$ | 0.3820$\pm 0.0004$ | 0.4383$\pm 0.0002$ | 0.4372$\pm 0.0002$ | 0.4058$\pm 0.0003$ | 0.7787$\pm 0.0004$ | 0.7987$\pm 0.0001$ | 0.4286$\pm 0.0002$
beret | 0.4635$\pm 0.0001$ | 0.3995$\pm 0.0004$ | 0.4110$\pm 0.0003$ | 0.4532$\pm 0.0002$ | 0.4543$\pm 0.0003$ | 0.4338$\pm 0.0003$ | 0.5097$\pm 0.0005$ | 0.5559$\pm 0.0003$ | 0.4532$\pm 0.0002$
birthdaycake | 0.4957$\pm 0.0003$ | 0.4753$\pm 0.0001$ | 0.4818$\pm 0.0001$ | 0.5697$\pm 0.0004$ | 0.5504$\pm 0.0003$ | 0.5322$\pm 0.0001$ | 0.6506$\pm 0.0002$ | 0.6620$\pm 0.0002$ | 0.6620$\pm 0.0004$
blog | 0.3998$\pm 0.0001$ | 0.3860$\pm 0.0003$ | 0.3796$\pm 0.0003$ | 0.4973$\pm 0.0002$ | 0.4719$\pm 0.0001$ | 0.4677$\pm 0.0003$ | 0.7149$\pm 0.0004$ | 0.6257$\pm 0.0002$ | 0.5440$\pm 0.0005$
blood | 0.4885$\pm 0.0001$ | 0.5589$\pm 0.0003$ | 0.3868$\pm 0.0004$ | 0.6085$\pm 0.0002$ | 0.6282$\pm 0.0002$ | 0.5878$\pm 0.0004$ | 0.4708$\pm 0.0003$ | 0.8072$\pm 0.0005$ | 0.4908$\pm 0.0001$
boat | 0.3991$\pm 0.0002$ | 0.4236$\pm 0.0002$ | 0.4014$\pm 0.0001$ | 0.3711$\pm 0.0003$ | 0.4072$\pm 0.0001$ | 0.3967$\pm 0.0002$ | 0.6124$\pm 0.0004$ | 0.6289$\pm 0.0003$ | 0.6056$\pm 0.0003$
boomerang | 0.3890$\pm 0.0003$ | 0.4923$\pm 0.0002$ | 0.4473$\pm 0.0004$ | 0.3989$\pm 0.0005$ | 0.4143$\pm 0.0003$ | 0.3945$\pm 0.0002$ | 0.5262$\pm 0.0004$ | 0.5253$\pm 0.0003$ | 0.4681$\pm 0.0003$
building | 0.5148$\pm 0.0002$ | 0.4874$\pm 0.0001$ | 0.4742$\pm 0.0003$ | 0.4632$\pm 0.0002$ | 0.4841$\pm 0.0002$ | 0.4479$\pm 0.0001$ | 0.6781$\pm 0.0003$ | 0.7146$\pm 0.0001$ | 0.6981$\pm 0.0004$
wallpaper | 0.4255$\pm 0.0003$ | 0.4342$\pm 0.0002$ | 0.3874$\pm 0.0001$ | 0.4363$\pm 0.0002$ | 0.4407$\pm 0.0002$ | 0.4385$\pm 0.0001$ | 0.5426$\pm 0.0003$ | 0.5256$\pm 0.0003$ | 0.4755$\pm 0.0002$
weapon | 0.4196$\pm 0.0003$ | 0.3753$\pm 0.0002$ | 0.4254$\pm 0.0002$ | 0.4033$\pm 0.0004$ | 0.4079$\pm 0.0006$ | 0.3939$\pm 0.0002$ | 0.4660$\pm 0.0005$ |  0.4860$\pm 0.0006$ | 0.3951$\pm 0.0004$
weddingdress | 0.3669$\pm 0.0002$ | 0.3930$\pm 0.0002$ | 0.3760$\pm 0.0001$ | 0.4258$\pm 0.0004$ | 0.4179$\pm 0.0003$ | 0.4417$\pm 0.0002$ | 0.4534$\pm 0.0002$ | 0.4394$\pm 0.0004$ | 0.4032$\pm 0.0001$
Average | 0.4309 | 0.4408 | 0.4194 | 0.4584 | 0.4623 | 0.4472 | 0.5783 | 0.6031 | 0.5273

Figure 3: Accuracy comparison between the proposed algorithms (K-means+MIRBM,
AP+MIRBM and SC+MIRBM) and contrastive algorithms ($\eta=0.1$).  TABLE IV: The
performances of the purity ($\eta=0.1$).

Dataset (No.) | K-means | AP | SC | K-means+GraphRBM | AP+GraphRBM | SC+GraphRBM | K-means+MIRBM | AP+MIRBM | SC+MIRBM
---|---|---|---|---|---|---|---|---|---
alphabet | 0.8395 | 0.8453 | 0.8447 | 0.8425 | 0.8474 | 0.8442 | 0.8748 | 0.8605 | 0.8439
americanflag | 0.8375 | 0.8212 | 0.8320 | 0.8717 | 0.8738 | 0.8705 | 0.8877 | 0.8947 | 0.8829
aquarium | 0.6961 | 0.6983 | 0.6952 | 0.7074 | 0.7125 | 0.7050 | 0.7148 | 0.7218 | 0.7193
baobab | 0.7676 | 0.7528 | 0.7659 | 0.7955 | 0.8003 | 0.7955 | 0.7842 | 0.7644 | 0.7909
bathroom | 0.5782 | 0.5804 | 0.5775 | 0.5586 | 0.5643 | 0.5595 | 0.5747 | 0.5817 | 0.5801
beret | 0.6942 | 0.7095 | 0.7032 | 0.7132 | 0.7182 | 0.7111 | 0.7135 | 0.7212 | 0.7232
birthdaycake | 0.7342 | 0.7694 | 0.7420 | 0.7931 | 0.7977 | 0.7901 | 0.7093 | 0.7314 | 0.7546
blog | 0.6723 | 0.6684 | 0.6749 | 0.6697 | 0.6740 | 0.6704 | 0.6754 | 0.6789 | 0.6811
blood | 0.4615 | 0.4608 | 0.4606 | 0.4611 | 0.4662 | 0.4604 | 0.4426 | 0.4682 | 0.4496
boat | 0.8102 | 0.8260 | 0.8018 | 0.8109 | 0.8212 | 0.8109 | 0.8231 | 0.8299 | 0.8296
boomerang | 0.9027 | 0.8760 | 0.8825 | 0.9038 | 0.9056 | 0.8989 | 0.8954 | 0.9041 | 0.8911
building | 0.6317 | 0.6637 | 0.6452 | 0.6872 | 0.6924 | 0.6871 | 0.6956 | 0.7024 | 0.7020
wallpaper | 0.7797 | 0.7829 | 0.7770 | 0.7752 | 0.7800 | 0.7755 | 0.7811 | 0.7873 | 0.7836
weapon | 0.9428 | 0.9364 | 0.9165 | 0.9415 | 0.9465 | 0.9397 | 0.9478 | 0.9548 | 0.9345
weddingdress | 0.8921 | 0.8845 | 0.8868 | 0.8738 | 0.8760 | 0.8715 | 0.8848 | 0.8935 | 0.8889
Average | 0.7483 | 0.7509 | 0.7465 | 0.7578 | 0.7625 | 0.7568 | 0.7590 | 0.7653 | 0.7607

TABLE V: The performances of the FMI ($\eta=0.1$).

Dataset (No.) | K-means | AP | SC | K-means+GraphRBM | AP+GraphRBM | SC+GraphRBM | K-means+MIRBM | AP+MIRBM | SC+MIRBM
---|---|---|---|---|---|---|---|---|---
alphabet | 0.3791 | 0.3808 | 0.3817 | 0.3746 | 0.3753 | 0.3738 | 0.5769 | 0.5855 | 0.4463
americanflag | 0.4164 | 0.4101 | 0.4046 | 0.4306 | 0.4431 | 0.4181 | 0.5843 | 0.6443 | 0.6101
aquarium | 0.4412 | 0.4323 | 0.4323 | 0.5133 | 0.5147 | 0.4920 | 0.6770 | 0.7370 | 0.7282
baobab | 0.4659 | 0.4931 | 0.4580 | 0.4844 | 0.4890 | 0.4638 | 0.5451 | 0.5055 | 0.6030
bathroom | 0.4729 | 0.4849 | 0.4726 | 0.4922 | 0.4910 | 0.4904 | 0.7514 | 0.8114 | 0.4996
beret | 0.4746 | 0.4337 | 0.4353 | 0.4631 | 0.4643 | 0.4575 | 0.4747 | 0.5654 | 0.4783
birthdaycake | 0.4612 | 0.4288 | 0.4513 | 0.5434 | 0.5183 | 0.5137 | 0.6069 | 0.6618 | 0.6594
blog | 0.4502 | 0.4477 | 0.4453 | 0.5031 | 0.4842 | 0.4839 | 0.7003 | 0.6256 | 0.5472
blood | 0.5290 | 0.5769 | 0.4923 | 0.6117 | 0.6282 | 0.5946 | 0.5258 | 0.8166 | 0.5858
boat | 0.4212 | 0.4212 | 0.4181 | 0.4053 | 0.4204 | 0.4104 | 0.6234 | 0.6772 | 0.6398
boomerang | 0.3719 | 0.4143 | 0.3884 | 0.3926 | 0.4026 | 0.3911 | 0.5089 | 0.5270 | 0.4307
building | 0.4876 | 0.5060 | 0.4669 | 0.4711 | 0.4861 | 0.4696 | 0.6657 | 0.7456 | 0.7255
wallpaper | 0.4153 | 0.4154 | 0.4044 | 0.4737 | 0.4845 | 0.4671 | 0.6182 | 0.5430 | 0.4921
weapon | 0.4366 | 0.3665 | 0.3695 | 0.3863 | 0.4058 | 0.3818 | 0.5463 | 0.6063 | 0.4728
weddingdress | 0.3702 | 0.3732 | 0.3721 | 0.3914 | 0.3802 | 0.3890 | 0.4555 | 0.5026 | 0.4036
Average | 0.4382 | 0.4359 | 0.4238 | 0.4619 | 0.4653 | 0.4529 | 0.5977 | 0.6322 | 0.5482

TABLE VI: The performances of the Jac ($\eta=0.1$).

Dataset (No.) | K-means | AP | SC | K-means+GraphRBM | AP+GraphRBM | SC+GraphRBM | K-means+MIRBM | AP+MIRBM | SC+MIRBM
---|---|---|---|---|---|---|---|---|---
alphabet | 0.2330 | 0.2341 | 0.2345 | 0.2297 | 0.2302 | 0.2287 | 0.3557 | 0.3852 | 0.2871
americanflag | 0.2612 | 0.2578 | 0.2521 | 0.2739 | 0.2833 | 0.2643 | 0.3652 | 0.4152 | 0.3923
aquarium | 0.2767 | 0.2687 | 0.2665 | 0.3437 | 0.3451 | 0.3234 | 0.4932 | 0.5432 | 0.5350
baobab | 0.3008 | 0.3252 | 0.2932 | 0.3195 | 0.3235 | 0.3018 | 0.3752 | 0.3363 | 0.4240
bathroom | 0.2885 | 0.3024 | 0.2878 | 0.3051 | 0.3041 | 0.3036 | 0.6083 | 0.6583 | 0.3181
beret | 0.3031 | 0.2695 | 0.2686 | 0.2968 | 0.2979 | 0.2908 | 0.3148 | 0.3926 | 0.3119
birthdaycake | 0.2957 | 0.2691 | 0.2873 | 0.3713 | 0.3496 | 0.3454 | 0.4332 | 0.4760 | 0.4660
blog | 0.2773 | 0.2767 | 0.2732 | 0.3299 | 0.3111 | 0.3114 | 0.5280 | 0.4522 | 0.3761
blood | 0.3403 | 0.3940 | 0.2997 | 0.4319 | 0.4516 | 0.4118 | 0.3544 | 0.6787 | 0.4044
boat | 0.2627 | 0.2650 | 0.2598 | 0.2511 | 0.2645 | 0.2552 | 0.4171 | 0.4631 | 0.4393
boomerang | 0.2276 | 0.2607 | 0.2399 | 0.2442 | 0.2520 | 0.2431 | 0.3227 | 0.3460 | 0.2744
building | 0.3126 | 0.3303 | 0.2924 | 0.3019 | 0.3168 | 0.3007 | 0.4891 | 0.5567 | 0.5390
wallpaper | 0.2595 | 0.2599 | 0.2504 | 0.3101 | 0.3190 | 0.3046 | 0.4100 | 0.3661 | 0.3251
weapon | 0.2747 | 0.2243 | 0.2263 | 0.2391 | 0.2532 | 0.2358 | 0.3176 | 0.3676 | 0.2955
weddingdress | 0.2263 | 0.2285 | 0.2276 | 0.2430 | 0.2338 | 0.2410 | 0.2838 | 0.3228 | 0.2528
Average | 0.2748 | 0.2749 | 0.2618 | 0.2989 | 0.3018 | 0.2905 | 0.4098 | 0.4463 | 0.3702

Figure 4: The performances of average purity and FMI and Jac ($\eta=0.1$).

###  V-E Generalization Ability

From Tables III, IV, V and VI, the K-means+MIRBM, AP+MIRBM and SC+MIRBM
algorithms based on the proposed architecture show the best performance in
their own groups with four evaluation metrics. With regard to the accuracy,
FMI and Jac evaluation metrics, the K-means+MIRBM, AP+MIRBM and SC+MIRBM
algorithms show the best performance than other groups algorithms. These mean
that the hidden layer features of the proposed architecture are available for
three different clustering algorithms. Hence, our architecture shows excellent
generalization ability for unsupervised feature learning.

###  V-F Sensitivity Analysis of the Key Parameter $\eta$

The parameter $\eta$ is an important scale coefficient in the proposed
architecture. It has a direct impact on the role of auxiliary guidance LCP.
Hence, we need analysis its sensitivity. The results of K-means+MIRBM,
AP+MIRBM and SC+MIRBM algorithms based on our architecture are shown in Fig.
5. The accuracy, purity, FMI and Jac evaluation metrics show the best
performance when the parameter $\eta$ is set to 0.1. The performance of
accuracy decrease rapidly with the increase of parameter $\eta$ from 0.1 to
0.9. The performance of FMI and Jac decrease rapidly with the increase of
parameter $\eta$ except for K-means+MIRBM algorithm. But, the purity is
insensitive to other evaluating metrics with the increase of parameter $\eta$.

Figure 5: Sensitivity analysis of the key parameters $\eta$ for the
K-means+MIRBM, AP+MIRBM and SC+MIRBM algorithms based on the proposed
architecture.

##  VI Conclusions

We presented a novel unsupervised feature learning architecture for image
clustering. The main motivation behind the proposed architecture was the need
to guide feature distribution for clustering by unsupervised LCP in the
training process. We compared the proposed architecture with state-of-the-art
GraphRBM to reveal the feature representation capability. In our experimental
datasets, the proposed architecture showed better performance than the
GraphRBM. For an important scale coefficient $\eta$, we studied its
sensitivity. When the parameter $\eta=0.1$, the accuracy, purity, FMI and Jac
evaluation metrics showed the best performance. The performance of accuracy
decreased rapidly with the increase of parameter $\eta$. The performance of
FMI and Jac decreased rapidly with the increase of parameter $\eta$ except for
K-means+MIRBM algorithm. But, the purity was insensitive to other evaluating
metrics with the increase of parameter $\eta$ for K-means+MIRBM, AP+MIRBM and
SC+MIRBM algorithms. Furthermore, we demonstrated that the proposed
architecture has excellent generalization ability for different clustering
algorithms.
For future work, we would like to exploit more efficient unsupervised feature
learning architecture for large-scale datasets. Deep architecture is also an
interesting work in the future.

##  VII Acknowledgement

This work were partially supported by the National Science Foundation of China
(Nos. 61773324, 61573292 and 61976247).

## References

  * [1] B. Yuan, J. Tu, R. W. Zhao, Y. Zheng, and Y. G. Jiang, “Learning part-based mid-level representation for visual recognition,” _Neurocomputing_ , 2017\.
  * [2] J. Gao, J. Yang, G. Wang, and M. Li, “A novel feature extraction method for scene recognition based on centered convolutional restricted boltzmann machines,” _Neurocomputing_ , vol. 11, no. 2, pp. p14–19, 2016.
  * [3] W. Diao, X. Sun, F. Dou, M. Yan, H. Wang, and K. Fu, “Object recognition in remote sensing images using sparse deep belief networks,” _Remote Sensing Letters_ , vol. 6, no. 10, pp. 745–754, 2015.
  * [4] S. Choo and H. Lee, “Learning framework of multimodal gaussian–bernoulli rbm handling real-value input data,” _Neurocomputing_ , vol. 275, pp. 1813–1822, 2018.
  * [5] K. Li, C. Zou, S. Bu, Y. Liang, J. Zhang, and M. Gong, “Multi-modal feature fusion for geographic image annotation,” _Pattern Recognition_ , vol. 73, 2017.
  * [6] O. Ghahabi and J. Hernando, “Restricted boltzmann machines for vector representation of speech in speaker recognition,” _Computer Speech and Language_ , vol. 47, pp. 16–29, 2018.
  * [7] C. L. P. Chen and S. Feng, “Generative and discriminative fuzzy restricted boltzmann machine learning for text and image classification,” _IEEE Transactions on Cybernetics_ , pp. 1–12, 2018.
  * [8] Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” _Nature_ , vol. 521, no. 7553, p. 436, 2015.
  * [9] D. Chen, J. Lv, and Z. Yi, “Graph regularized restricted boltzmann machine.” _IEEE Transactions on Neural Networks and Learning Systems_ , vol. PP, no. 99, pp. 1–9, 2017.
  * [10] J. Tang and H. Liu, “An unsupervised feature selection framework for social media data,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 26, no. 12, pp. 2914–2927, Dec 2014.
  * [11] M. Chen, I. W. Tsang, M. Tan, and T. J. Cham, “A unified feature selection framework for graph embedding on high dimensional data,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 27, no. 6, pp. 1465–1477, June 2015.
  * [12] D. Wang, F. Nie, and H. Huang, “Feature selection via global redundancy minimization,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 27, no. 10, pp. 2743–2755, Oct 2015.
  * [13] M. Banerjee and N. R. Pal, “Unsupervised feature selection with controlled redundancy (ufescor),” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 27, no. 12, pp. 3390–3403, Dec 2015.
  * [14] X. Zhu, S. Zhang, R. Hu, Y. Zhu, and j. song, “Local and global structure preservation for robust unsupervised spectral feature selection,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 30, no. 3, pp. 517–529, March 2018.
  * [15] G. Hinton and T. Sejnowski, “Learning and releaming in boltzmann machines,” _Parallel distributed processing: Explorations in the microstructure of cognition_ , vol. 1, pp. 282–317, 1986.
  * [16] T. Tieleman, “Training restricted boltzmann machines using approximations to the likelihood gradient,” in _International Conference on Machine Learning_ , 2008, pp. 1064–1071.
  * [17] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” _Neural Computation_ , vol. 14, no. 8, pp. 1771–1800, 2002\.
  * [18] N. Lu, T. Li, X. Ren, and H. Miao, “A deep learning scheme for motor imagery classification based on restricted boltzmann machines,” _IEEE Transactions on Neural Systems and Rehabilitation Engineering_ , vol. PP, no. 99, pp. 1–1, 2016.
  * [19] O. Fink, E. Zio, and U. Weidmann, “Fuzzy classification with restricted boltzman machines and echo-state networks for predicting potential railway door system failures,” _IEEE Transactions on Reliability_ , vol. 64, no. 3, pp. 861–868, 2015.
  * [20] Y. Chen, X. Zhao, and X. Jia, “Spectral–spatial classification of hyperspectral data based on deep belief network,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 8, no. 6, pp. 2381–2392, 2015.
  * [21] S. Nie, Z. Wang, and Q. Ji, “A generative restricted boltzmann machine based method for high-dimensional motion data modeling,” _Computer Vision and Image Understanding_ , vol. 136(C), pp. 14–22, 2015.
  * [22] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2013, pp. 6645–6649.
  * [23] J. Chu, H. Wang, H. Meng, P. Jin, and T. Li, “Restricted boltzmann machines with gaussian visible units guided by pairwise constraints,” _IEEE Transactions on Cybernetics_ , pp. 1–14, 2018.
  * [24] Q. Yu, Y. Hou, X. Zhao, and G. Cheng, “Rényi divergence based generalization for learning of classification restricted boltzmann machines,” in _2014 IEEE International Conference on Data Mining Workshop (ICDMW)_. IEEE, 2014, pp. 692–697.
  * [25] C. Chen, C. Y. Zhang, L. Chen, and M. Gan, “Fuzzy restricted boltzmann machine for the enhancement of deep learning,” _IEEE Transactions on Fuzzy Systems_ , vol. 23, no. 6, pp. 2163–2173, 2015.
  * [26] A. Courville, G. Desjardins, J. Bergstra, and Y. Bengio, “The spike-and-slab rbm and extensions to discrete and sparse data distributions,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 36, no. 9, pp. 1874–1887, 2014.
  * [27] R. Karakida, M. Okada, and S.-i. Amari, “Dynamical analysis of contrastive divergence learning: Restricted boltzmann machines with gaussian visible units,” _Neural Networks_ , vol. 79, pp. 78–87, 2016.
  * [28] M. R. Amer, T. Shields, B. Siddiquie, A. Tamrakar, A. Divakaran, and S. Chai, “Deep multimodal fusion: A hybrid approach,” _International Journal of Computer Vision_ , pp. 1–17, 2017.
  * [29] H. Luo, P. Jia, S. Qiao, and S. Duan, “Enhancing electronic nose performance based on a novel qpso-rbm technique,” _Sensors and Actuators B: Chemical_ , vol. 259, pp. 241–249, 2018.
  * [30] A. A. Alani, “Arabic handwritten digit recognition based on restricted boltzmann machine and convolutional neural networks,” _Information_ , vol. 8, no. 4, p. 142, 2017.
  * [31] S. Bu, L. Wang, P. Han, Z. Liu, and K. Li, “3d shape recognition and retrieval based on multi-modality deep learning,” _Neurocomputing_ , vol. 259, pp. 183–193, 2017.
  * [32] G. Cheng, P. Zhou, and J. Han, “Duplex metric learning for image set classification.” _IEEE Transactions on Image Processing_ , vol. 27, no. 1, pp. 281–292, 2017.
  * [33] G. Chen, “Deep transductive semi-supervised maximum margin clustering,” _arXiv preprint arXiv:1501.06237_ , 2015.
  * [34] R. Chen, S. Chen, M. He, D. He, and B. Tang, “Rolling bearing fault severity identification using deep sparse auto-encoder network with noise added sample expansion,” _Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability_ , vol. 231, no. 6, pp. 666–679, 2017\.
  * [35] K. T. Yesilbek and T. M. Sezgin, “Sketch recognition with few examples,” _Computers and Graphics_ , vol. 69, pp. 80–91, 2017.
  * [36] C. T. Sari and C. Gunduz-Demir, “Unsupervised feature extraction via deep learning for histopathological classification of colon tissue images,” _IEEE Transactions on Medical Imaging_ , pp. 1–1, 2018.
  * [37] M. A. Keyvanrad and M. M. Homayounpour, “Effective sparsity control in deep belief networks using normal regularization term,” _Knowledge and Information Systems_ , vol. 53, no. 2, pp. 533–550, 2017.
  * [38] X. Tang, N. Zhang, J. Zhou, and Q. Liu, “Hidden-layer visible deep stacking network optimized by pso for motor imagery eeg recognition,” _Neurocomputing_ , vol. 234, no. C, pp. 1–10, 2016.
  * [39] P. Chopra and S. K. Yadav, “Restricted boltzmann machine and softmax regression for fault detection and classification,” _Complex and Intelligent Systems_ , vol. 4, no. 1, pp. 67–77, 2018.
  * [40] X. Zhang, Y. Liang, C. Li, H. Ning, L. Jiao, and H. Zhou, “Recursive autoencoders-based unsupervised feature learning for hyperspectral image classification,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, no. 11, pp. 1928–1932, 2017.
  * [41] C. Xie, J. Lv, and X. Li, “Finding a good initial configuration of parameters for restricted boltzmann machine pre-training,” _Soft Computing_ , pp. 1–9, 2016.
  * [42] H. Al-Dmour and A. Al-Ani, “A clustering fusion technique for mr brain tissue segmentation,” _Neurocomputing_ , vol. 275, pp. 546–559, 2018.
  * [43] R. Stewart and S. Ermon, “Label-free supervision of neural networks with physics and domain knowledge,” in _AAAI_ , 2017, pp. 2576–2582.
  * [44] S. M. Azimi, D. Britz, M. Engstler, M. Fritz, and F. M cklich, “Advanced steel microstructural classification by deep learning methods,” _Scientific Reports_ , vol. 8, no. 1, 2017.
  * [45] M. Seera, C. P. Lim, K. S. Tan, and S. L. Wei, “Classification of transcranial doppler signals using individual and ensemble recurrent neural networks,” _Neurocomputing_ , vol. 249, no. C, pp. 337–344, 2017.
  * [46] K. Verma and R. K. Sharma, “Comparison of hmm- and svm-based stroke classifiers for gurmukhi script,” _Neural Computing and Applications_ , pp. 1–13, 2016.
  * [47] K. A. Abdalmalak and A. Gallardo-Antol n, “Enhancement of a text-independent speaker verification system by using feature combination and parallel structure classifiers,” _Neural Computing and Applications_ , pp. 1–15, 2016\.
  * [48] H. Guan, T. Liu, J. Jiang, D. Tao, J. Zhang, H. Niu, W. Zhu, Y. Wang, J. Cheng, and N. A. Kochan, “Classifying mci subtypes in community-dwelling elderly using cross-sectional and longitudinal mri-based biomarkers:,” _Frontiers in Aging Neuroscience_ , vol. 9, pp. 309–322, 2017.
  * [49] M. Tahir, B. Jan, M. Hayat, S. U. Shah, and M. Amin, “Efficient computational model for classification of protein localization images using extended threshold adjacency statistics and support vector machines,” _Computer Methods and Programs in Biomedicine_ , vol. 157, 2018.
  * [50] M. P. Hosseini, D. Pompili, K. Elisevich, and H. Soltanian-Zadeh, “Random ensemble learning for eeg classification,” _Artificial Intelligence in Medicine_ , 2018.
  * [51] R. He, T. Tan, L. Davis, and Z. Sun, “Learning structured ordinal measures for video based face recognition,” _Pattern Recognition_ , vol. 75, pp. 4–14, mar 2018.
  * [52] M. Yang, R. Shang, L. Jiao, W. Zhang, Y. Yuan, and S. Yang, “Feature selection based dual-graph sparse non-negative matrix factorization for local discriminative clustering,” _Neurocomputing_.
  * [53] R. Shang, W. Wang, R. Stolkin, and L. Jiao, “Non-negative spectral learning and sparse regression-based dual-graph regularized feature selection,” _IEEE Transactions on Cybernetics_ , vol. 48, no. 2, pp. 793–806, Feb 2018.
  * [54] H. Li, M. Wang, and X.-S. Hua, “Msra-mm 2.0: A large-scale web multimedia dataset,” in _IEEE International Conference on Data Mining Workshops_. IEEE, 2009, pp. 164–169.
  * [55] S. Lloyd, “Least squares quantization in pcm,” _IEEE Transactions on Information Theory_ , vol. 28, no. 2, pp. 129–137, 1982.
  * [56] B. J. Frey and D. Dueck, “Clustering by passing messages between data points,” _Science_ , vol. 315, no. 5814, pp. 972–976, 2007.
  * [57] A. Y. Ng, M. I. Jordan, Y. Weiss _et al._ , “On spectral clustering: Analysis and an algorithm,” _Advances in Neural Information Processing Systems_ , vol. 2, pp. 849–856, 2002.
  * [58] D. Cai, X. He, and J. Han, “Document clustering using locality preserving indexing,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 17, no. 12, pp. 1624–1637, 2005.
  * [59] C. Ding, T. Li, W. Peng, and H. Park, “Orthogonal nonnegative matrix t-factorizations for clustering,” in _the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. ACM, 2006, pp. 126–135.
  * [60] R. Liu, H. Wang, and X. Yu, “Shared-nearest-neighbor-based clustering by fast search and find of density peaks,” _Information Sciences_ , vol. 450, pp. 200 – 226, 2018.
  * [61] R. Real and J. M. Vargas, “The probabilistic basis of jaccard’s index of similarity,” _Systematic Biology_ , vol. 45, no. 3, pp. 380–385, 1996.
  * [62] C.-M. Hwang, M.-S. Yang, and W.-L. Hung, “New similarity measures of intuitionistic fuzzy sets based on the jaccard index with its application to clustering,” _International Journal of Intelligent Systems_ , vol. 33, no. 8, pp. 1672–1688, 2018.

|  Jielei Chu received the B.S. degree from Southwest Jiaotong University,
Chengdu, China in 2008, and is currently working toward the Ph.D. degree at
Southwest Jiaotong University. His research interests are deep learning, data
mining, semi-supervised learning and ensemble learning.
---|---
|  Hongjun Wang received his Ph.D. degree in computer science from Sichuan
University of China in 2009. He is currently Associate Professor of the Key
Lab of Cloud Computing and Intelligent Techniques in Southwest Jiaotong
University. His research interests are machine learning, data mining and
ensemble learning. He published over 30 research papers in journals and
conferences and he is a member of ACM and CCF. He has a reviewer for several
academic journals.
---|---
|  Jing Liu received his Ph.D. degree in management from Southwest Jiaotong
University.She is currently an Assistant Professor of Business SchoolinSichuan
University. Her research interestsare machine learning, financial
technologyand modelling and forecasting high-frequency data.
---|---
|  Zhiguo Gong received the Ph.D. degree in computer science from the
Institute of Mathematics, Chinese Academy of Science, Beijing,China. He is
currently a Professor with the Faculty of Science and Technology, University
of Macau, Macau, China. His current research interests include machine
learning, data mining, database, and information retrieval.
---|---
|  Tianrui Li (SM’11) received the B.S., M.S., and Ph.D. degrees in traffic
information processing and control from Southwest Jiaotong University,
Chengdu, China, in 1992, 1995, and 2002, respectively. He was a Post-Doctoral
Researcher with Belgian Nuclear Research Centre, Mol, Belgium, from 2005 to
2006, and a Visiting Professor with Hasselt University, Hasselt, Belgium, in
2008; University of Technology, Sydney, Australia, in 2009; and University of
Regina, Regina, Canada, in 2014. He is currently a Professor and the Director
of the Key Laboratory of Cloud Computing and Intelligent Techniques, Southwest
Jiaotong University. He has authored or co-authored over 150 research papers
in refereed journals and conferences. His research interests include big data,
cloud computing, data mining, granular computing, and rough sets. Dr. Li is a
fellow of the International Rough Set Society.
---|---