# Improving ClusterGAN Using Self-Augmented Information Maximization of
Disentangling Latent Spaces

Tanmoy Dam ${}^{\href https://orcid.org/0000-0003-3022-0971}$, Sreenatha G.
Anavatti ${}^{\href https://orcid.org/; 0000-0002-4754-8191}$, Hussein A.
Abbass, ${}^{\href https://orcid.org/0000-0002-8837-0748}$
School of Engineering and Information Technology, University of New South
Wales Canberra, Australia.

###### Abstract

Since their introduction in the last few years, conditional generative models
have seen remarkable achievements. However, they often need the use of large
amounts of labelled information. By using unsupervised conditional generation
in conjunction with a clustering inference network, ClusterGAN has recently
been able to achieve impressive clustering results. Since the real conditional
distribution of data is ignored, the clustering inference network can only
achieve inferior clustering performance by considering only uniform prior
based generative samples. However, the true distribution is not necessarily
balanced. Consequently, ClusterGAN fails to produce all modes, which results
in sub-optimal clustering inference network performance. So, it is important
to learn the prior, which tries to match the real distribution in an
unsupervised way. In this paper, we propose self-augmentation information
maximization improved ClusterGAN (SIMI-ClusterGAN) to learn the distinctive
priors from the data directly. The proposed SIMI-ClusterGAN consists of four
deep neural networks: self-augmentation prior network, generator,
discriminator and clustering inference network. The proposed method has been
validated using seven benchmark data sets and has shown improved performance
over state-of-the art methods. To demonstrate the superiority of SIMI-
ClusterGAN performance on imbalanced dataset, we have discussed two imbalanced
conditions on MNIST datasets with one-class imbalance and three classes
imbalanced cases. The results highlight the advantages of SIMI-ClusterGAN.

###### Index Terms:

Clustering, ClusterGAN, Generative Adversarial Networks, Information
Maximisation, Imbalanced Data.

##  I introduction

Clustering is a popular unsupervised representation learning method, which has
been widely studied in computer vision and machine learning communities such
as image segmentation [1], visual features representation learning [2], and 3D
object recognition [3]. The lower dimensional representation of high
dimensional semantics data has been described by many latent-space based
clustering methods, such as DCN [4], DEC [5], Dual-AE [6], and ClusterGAN [7].

Latent space clustering methods have been developed based upon the clustering-
assignment objective in the latent-space. Therefore, the objective still lies
in the discriminative learning representation blending with data
reconstruction loss. This discriminative lower dimensional representation can
capture all discrete factors in the lower dimensional latent space such as
data variations within the same group, which are the key factors for
clustering objective. However, it fails to reconstruct the real data due to
the clustering assignment objective. Hence, similarities based clustering
methods (such as K-means) have been incorporated with latent space to
determine the clustering assignments [4]. Further improvement using latent-
space clustering objectives and pretrained reconstruction are employed to
determine the optimal lower cluster assignment representation [8]. However,
this is a very laborious task to obtain both optimal lower representation and
a reconstruction of the real data at the same time. To obtain this
representation, the high-dimensional data is first represented in lower
dimensional latent space and then different distance-metric clustering
objectives are applied on latent space to determine clustering assignments [5]
[9].

Generative adversarial networks (GAN) are one of the most effective tools to
learn an implicit function [10, 11, 12, 13] from complex real data
distribution [14, 3, 15, 16, 17, 18, 7, 19]. The learning process is
formulated as an adversarial game using the min-max principle. The generator
network takes the sample from the noisy latent distribution to estimate the
real distribution, while the discriminator network is trained to discriminate
the real from fake samples, respectively. For learning complicated data
distributions, generative models have been extensively studied in most
circumstances. When it comes to downstream tasks like clustering, the
estimation of the posterior distribution of latent inference from the data is
an intractable problem in GAN models. Therefore, the researchers developed a
mode-matching network for estimating downstream tasks [7, 20]. In a mode
matching network, a clustering inference network estimates the multi-modal
latent prior through bounding loss. Thus, an inference network works as a
regularisation of the GAN objective where each mode is defined for each class
of real data distribution. However, it is well-defined that mode matching
network can’t be fully enforced the clustering objective on the generative
samples when dealing with high-dimensional data [21, 22, 23] . Therefore, for
high-dimensional data like CIFAR-10, STL-10, the mode matching network [7, 20]
is dependent on the semantics of the data, which may not be directly related
to the labels(class) information. In addition, in real-life scenarios, the
classes of real distribution are not uniformly distributed. For example, the
MNIST dataset contains ten distinctive classes, where mode matching methods
try to estimate ten distinctive modes through ten uniform priors [7, 20].
Although the latent priors originate from ten different distributions, the
uniform prior-based generative model can only create major classes. Thus, the
inverting inference network fails to provide appropriate clustering
performance. As a solution, we need to include regularisation that compel
inference networks to accurately estimate downstream tasks and create all
unique modes in imbalanced situations. To tackle both the issues, we propose a
self augmented information maximization improved ClusterGAN (SIMI-ClusterGAN)
in this paper. SIMI-ClusterGAN has four deep neural networks: self-
augmentation prior network, generator, discriminator and clustering inference
encoder. Prior network is used to learn the discrete representation of data in
discrete space by self-augmentation maximization principle. This learned prior
blends with continuous variable to form a discrete noise latent space. Then,
generative network takes continuous-discrete Gaussian noise latent space prior
to map in real data domain. Clustering inference encoder predicts the lower-
level representation of the generated as well as real data through clustering
assignment indices. To enforce the disentanglement clustering representation
of real data, the weight sharing generator-encoder networks pair have been
utilised. We propose three additional loss functions to modify the ClusterGAN
objectives. These three losses are used to create distinctive latent-space to
help maintaining diversity in generative samples in addition to improving the
clustering performance.

In summary, the main contributions of this work are as follows:

  * •

We propose self-augmented mutual information maximization based disentangle
categorical priors (one-hot encode)learning from the data.

  * •

The learned discrete categorical priors with continuous Gaussian noise are
utilised to form smooth disentangle noise latent spaces. This learned mixture
of continuous-discrete disentangle latent spaces is used to generate the real
class distribution of the data.

  * •

Three additional loss functions are introduced to modify ClusterGAN objectives
for improved performance. Theses loss functions account for reconstruction,
prior bounding and cross modality.

  * •

A comparison of the the proposed SIMI-ClusterGAN and other GAN clustering
methods is presented to validate the superiority of the proposed method in
clustering tasks. We have used seven datasets including CIFAR-10 and STL-10 to
accommodate complex high-dimensional data. We further demonstrate the
superiority of our proposed method for imbalanced dataset by comparing the
performance with ClusterGAN under two different imbalanced conditions.

##  II Related Work

###  II-A Disentangle latent space clustering

Most of the latent-space based approaches in practice have the encoder-decoder
(Enc-Dec) structure where lower dimensional encoding latent code is used for
clustering tasks [24, 25, 26, 5]. In deep embedding clustering (DEC) [5], an
Enc-Dec structure with reconstruction loss has been considered to train the
model parameters. Once pre-training has been done, then a clustering network
is added over the Enc network for further training with clustering
similarities loss. In improved DEC [26], to maintain the local structure in
the latent space, reconstruction loss is also considered in the final stage.
The whole learning process is jointly trained as in DEC [5]. The Deep
clustering network(DCN) is developed based on the Enc-Dec and K-means
clustering objective [25]. In order to maintain the local semantics in latent
space, the reconstruction loss is adopted along with the K-means objective.
However, the performance of the DCN method is still not seen to be adequate.

The variation of disentangle factor in autoencoder based generative models can
give explainable semantic latent code from the data. Most of the disentangle
representations are associated with defining the separate latent spaces using
variational factors [27, 28, 29, 30] in which a single-stage or two-stage
learning procedure can be found. Single stage disentanglement mostly deals
with variations of two factors [27, 31, 32, 33] or three factors [28, 29]. In
all single-stage methods, the disentanglement has improved due to the partial
knowledge of label information through cross-entropy loss. Meanwhile, two-
stage disentanglement methods, such as $\beta$-TCVAE [34], $\beta$-VAE [35],
factor-VAE [36], and joint continuous-discrete factors VAE [37] methods create
separate latent spaces without knowledge of class-label information. These
methods mostly deal with autoencoder structures with a Gaussian prior
distribution in the encoded space. The disentanglement is accounted through
reconstruction grade and regularisation of latent code factorization even
though the real world data require more disentanglement’s factors (eg.
labels), which can’t be obtained directly with continuous latent spaces.
Consequently, joint VAE [37] came up with continuous-discrete factors of
variations into the latent space.

###  II-B Generative prior-based inverse latent space clustering

The multi-modal prior based latent space clustering can be segregated into two
ways, 1) by applying a mixture of continuous distributions, such as the GMM
[38], and 2) by applying a combination of discrete and continuous
distributions, such as the InfoGAN [20] and ClusterGAN[7]. The latter is the
more popular of the two and is often accomplished by the concatenation of
discrete and continuous random variables. In InfoGAN, the disentanglement of
latent factor is maximised between the latent code and the generated data. The
modification of infoGAN, known as ClusterGAN [7], works on the same principle
except for the bounded non-smooth latent space. The non-smooth discrete
disentangle latent space is based on mixture of continuous-discrete (one-hot
code) latent variables. The one-hot code discrete variable is used to measure
the clustering performance by the encoder inference network. However, the
continuous-discrete latent space variables are unable to complete
disentanglement (e.g. categories) in ClusterGAN. It is reasonable to assume
that the cluster characteristics of real data will not be replicated in the
generated data, resulting in incorrect coverage of clusters in the generated
data, as described in [20]. To tackle such an issue, more disentanglement in
latent space is observed in [23]. However, the proposed NEMGAN [23], that has
the same ClusterGAN structure except a mode engineering network is used to
learn the discrete prior from the data. The Mode network is updated by
minimizing the KL divergence loss between mutual information data and mode
network output. Thus, mode network output depends on mutual information of the
data which is more relevant than the uniform discrete distribution of
ClusterGAN. While updating the mode network, NEMGAN accounts for partial true
label information. However, the performance is not significantly improved
without partial information of true labels compared to the ClusterGAN.
Therefore, NEMGAN is not a completely unsupervised method; instead, we would
say it is a semi-supervised method.

In SIMI-ClusterGAN, we have proposed a two-stage approach to improve
ClusterGAN performance. For learning the discrete prior assignments from the
data, the self-augmented information maximization principle is employed. Once
the learned prior from the data is obtained, it is used to form continuous-
discrete mixtures to improve our SIMI-ClusterGAN performance. Unlike NEMGAN,
the clustering inference network is used to separate the continuous-discrete
variable without any supervision.

##  III Proposed approach

This section begins with a brief introduction to the ClusterGAN algorithm. The
discrete prior is then discovered by applying the SIM principle directly to
the data. We created our novel SIMI-ClusterGAN approach by combining this
learnt discrete prior with three additional losses. The architecture of the
proposed method is depicted in Figure 1.

Figure 1: The SIMI-ClusterGAN Architecture. $E_{\omega}$ and $G_{\theta}$ are
sharing networks.

###  III-A ClusterGAN

Let, the given $i$-th sample of real data $\\{x_{i}^{r}\\}_{i=1}^{N}\in X\in
p_{r}$, represent the $M$-discrete representable form by the function
$Y=s(X)$, where $Y\in\\{y_{1},y_{2},...,y_{M}\\}\subseteq\textbf{c}$. Where,
$\textbf{c}\in\\{c_{1},c_{2},...,c_{M}\\}$ is the ground truth classes. The
ClusterGAN objective is estimate the $s(.)$ through adversarial GAN game
principle. Generative adversarial networks (GAN) [10] are defined by two
neural network(NN) components, the generative component(G) and the
discriminator component(D), which are parameterised by $\theta$ and $\eta$,
respectively. The generator $(G_{\theta})$ takes noise distribution to map
real distribution, $G_{\theta}:Z\mapsto X$ whereas the discriminator
$(D_{\eta})$ is assigned a probabilistic value for the data sample of being
real vs fake data, $D_{\eta}:X\mapsto R$. The GAN game is working on a two
players min-max game principle defined as follows,

$\mathop{\min}\limits_{\theta}\mathop{\max}\limits_{\eta}\mathbb{E}_{x^{r}\sim
p_{r}}[f(D_{\eta}(x^{r}))]+\mathbb{E}_{z\sim
p_{z}}[f(1-D_{\eta}(G_{\theta}(z)))]$ (1)

where, the real sample $(x^{r}\in X)$ belongs to real probabilities
distribution $x^{r}\in p_{r}$. $z\in p_{z}$ is the latent space samples, which
are drawn from a known prior distribution ($p_{z}$)(e.g. multivariate Gaussian
distribution). $p_{g}$ is the generated sample distribution which defines
$x^{g}=G_{\theta}(z)$ and $f$ is the standard GAN type selection function. For
vanilla GAN, $f(x^{r})=logx^{r}$ and for Wasserstein GAN, $f(x^{r})=x^{r}$
[39]. The adversarial GAN objective is to learn the function $f(.)$
$(p_{r}=f(p_{z}))$ through density estimation [40].

####  III-A1 Continuous-discrete prior based disentanglement in ClusterGAN

In ClusterGAN, the multi-modal latent space($z\in Z$) is defined by a mixture
of continuous and discrete prior distribution to form a discrete smooth
manifold. Generally, a discrete smooth manifold mixture is obtained by
cascading the normal distribution with uniform categorical distribution, and
its disentanglement in latent space is controlled through categorical
distributions. To be more specific, the latent space prior represents
$z=(z_{n},z_{M})$ where $z_{n}=\mathcal{N}({\mu=0,\sigma^{2}*I_{d_{n}}})$ and
$z_{M}\sim\mathcal{U}\\{1,2,...,M\\}$, $\mathcal{U}\in R^{M}$ is the discrete
uniform prior, which represents the number of $M$ classes present in the
dataset. However, the variance$(\sigma)$ of the normal distribution
($\mathcal{N}$) is kept to a low value for the $G_{\theta}$ network so that
all the discrete modes $(labels)$ can be generated with high purity. Hence,
the sigma value [7] has been chosen to ($\sigma=0.10$) for all experiments in
this paper. Thus, the continuous-discrete latent space ($z_{n}$) is bounded
within $z\in(-0.6,0.6)<1$. This uniform discrete latent priors allow us to
design an algorithm capable of detecting a number of clusters in the inverse
generated space.

####  III-A2 Inference Clustering Network in ClusterGAN

Many previous methods [41, 42] are mainly focused on the inverse latent space
to reconstruct $(z^{*})$ through an encoder inference $(E_{\omega})$ network.
However, this bidirectional mapping can’t guarantee the reconstruction of the
latent space due to a lack of consistency between the samples and the latent
space. Due to the non-convex optimization nature of the problem, the
reconstruction always suffers from different latent space embedding $z$.
DeliGAN [38] has used different initialization to reconstruct the discrete
latent space. Stochastic clipping of $z$ at each step is also found in [42].
However, none of the methods are related to the latent space based clustering.

In ClusterGAN [7], the bounded discrete latent space is computed by inference
autoencoder network $(E_{\omega})$. The generator takes the mixture of
continuous-discrete latent space $(z=z_{n},z_{M})$ to generate $x^{g}$ and the
encoder estimates back to the latent space
$(\hat{z}^{g}_{n},\hat{z}^{g}_{M})=E_{\omega}(x^{g})$. The encoding cyclic
loss for generated samples is computed by cross-entropy loss and
reconstruction losses between $(z_{n},z_{M})$ and
$(\hat{z}^{g}_{n},\hat{z}^{g}_{M})$. The generated samples cross entropy loss
$(J_{ce})$ between $z_{M}$ and $\hat{z}^{g}_{M}$ is defined as follows,

$J_{ce}=\operatorname{\mathbb{E}}_{z\sim
p_{z}}[l(z_{M},E_{\omega}(G_{\theta}(z_{n},z_{M})))]$ (2)

where, $l$ is associated with the cross-entropy loss. The reconstruction loss
$(J_{mse})$ between $z_{n}$ and $\hat{z}^{g}_{n}$ is defined by the mean
square error (MSE) as follows,

$J_{mse}=\operatorname{\mathbb{E}}_{z\sim p_{z}}\parallel
z_{n}-E_{\omega}(G_{\theta}(z_{n},z_{M}))\parallel_{2}$ (3)

Finally, the three-player ClusGAN game directs attention to the following
optimisation problem:

$\begin{split}\mathop{\min}\limits_{\theta,\omega}\mathop{\max}\limits_{\eta}\operatorname{\mathbb{E}}_{x\sim
p_{r}}[f(D(x^{r}))]+\operatorname{\mathbb{E}}_{z\sim p_{z}}[f(1-D(G(z)))]\\\
+\alpha_{cl}J_{ce}+\alpha_{mse}J_{mse}\end{split}$ (4)

The regularisation coefficients ($\alpha_{cl},\alpha_{mse}$) control the
disentanglement factors in the encoded latent space. To determine the
clustering accuracy, the real data distribution samples pass through the
$E_{\omega}$ network to predict continuous and discrete variables. Afterwards,
the $K$-means [9] clustering algorithm is applied to concatenated continuous
and discrete vectors to obtain clustering performance. For downstream tasks
like clustering, the cluster characteristics of real data can’t be completely
replicated through generated samples. This results in the disentanglement of
encoded latent space, which may not be enough because the $E_{\omega}$ network
has never been exposed to real distribution. As a result, adequate performance
can’t be obtained with this learning strategy. To tackle this issue, we first
learn the lower level representation by using the SIM principle. Afterwards,
the learned prior is used to guide more disentanglement in encoded latent
space by using three additional losses. Unlike NEMGAN [23], both the learning
stages don’t consider any true level of information.

###  III-B Self-augmented information maximization(SIM) for discrete prior
learning:

We use SIM principle to learn the discrete prior(${y_{1},y_{2},...,y_{M}}$)
from the data directly. Our objective is estimate the $s(.)$ by maximised
mutual information between $X\&Y$. According to Gomes et. al. [43],
regularised information maximisation (RIM) maximizes the mutual information
between the data and its discrete representation ($m=1,2,...,M$) by learning
the prior network ($P_{\phi}(Y|X)$). Therefore, the regulariser based RIM
objective function is defined as follows,

$R_{\phi}-\beta_{p}I(X:Y)$ (5)

where, $I(.)$ measures the mutual information between the data and the
discrete representation. The reguliser constraint, $R_{\phi}$, updates the
prior network parameters $\phi$. $\beta_{p}$ controls the reguliser and mutual
information. The prior network objective is to maximise the conditional
probabilities based on the similar representation of the data [43]. The
independent conditional probabilities of given model,
$P_{\phi}(y_{1},y_{2}...,y_{M}|X)$, are represented in the following form,

${P_{\phi}(y_{1},y_{2},...,y_{M}|X)=\prod_{m=1}^{M}P_{\phi}(y_{m}|X)}$ (6)

Maximising Eq. 6 is to represent the $M$ discrete classes based on the mutual
information between data. Hu. et. al. [44] suggested the regularisation
$(R_{\phi})$ can be represented in a flexible formation, which is defined as a
self-augmented training (SAT). The self-augmentation regularisation between
data $x^{r}$ and its augmented variations $T(x^{r})$ is represented as
follows,

${R_{SAT}({\phi;x^{r},T(x^{r}))}=-\sum_{m=1}^{M}\sum_{y_{m}=0}^{M-1}P_{\hat{\phi}}(y_{m}|x^{r})logP_{\phi}(y_{m}|T(x^{r}))}$
(7)

where, $T(.)$ is a self augmentation function. $P_{\hat{\phi}}(y_{m}|x^{r})$
is the current prediction of data $x^{r}$ and $\hat{\phi}$ is the current
update of $P$ network parameters. Equation 7 represents the ability of the
augmented data $(T(x^{r}))$ to push closer to the conditional probabilities
$P_{\phi}(y_{M}|x^{r})$. Generally, for image data, the augmentation is mostly
dealt with an affine transformation such as random rotation, scaling and
shearing [44].

The local perturbation of data can’t alter the invariant nature and the local
perturbation based self-augmentation can be defined as follows,

$T(x^{r})=x^{r}+l_{p}$ (8)

where $l_{p}$ is the local perturbation which does not change the data
representation in low dimensional regions/manifold [45]. The local
perturbation regularisation method is based on the virtual adversarial
training method [46] in which the local perturbation is defined as follows,

$l_{p}=\operatorname*{arg\,max}_{\hat{l}_{p}}[R_{SAT}({\hat{\phi}};x^{r},x^{r}+l_{p}),\,\,\,\parallel{l_{p}}\parallel_{2}\leq\beta_{t}]$
(9)

The local perturbation solution of Eq. 9 can be easily obtained from [46].

The discrete cluster representation of Eq. 5 can be reviewed in-terms of the
difference in the mutual information between the entropy and the marginal
conditional entropy [43] as follows,

$R_{SAT}-\beta_{p}[H(Y)-\beta_{mu}H(Y|X)]$ (10)

where, $H(Y)$ and $H(Y|X)$ represent the marginal entropy and conditionals
entropy, respectively, which are calculated as follows,

$\begin{split}H(Y)=h(P_{\phi}(y_{M}))=h(\frac{1}{N}\sum_{n=1}^{N}P_{\phi}(y_{M}|x^{r}_{n}))\\\
H(Y|X)=\frac{1}{N}\sum_{n=1}^{N}h(P_{\phi}(y_{M}|x^{r}_{n}))\end{split}$ (11)

where, $h(.)$ is an entropy function. The marginal entropy and conditionals
entropy are working together to represent the discrete representation
($y_{M}$) of the data. Hence, Eq. 10 is the final optimization problem that
can give us discrete prior index ($y_{M}$) from the data. After obtaining the
learned prior, it has to be represented in one-hot-code categorical
priors($z_{M}$) in the proposed SIMI-ClusterGAN. The one-hot code (OHC)
representation of $y_{M}$ is defined by the following form,

$z_{M}=1_{M}(y_{M})=\begin{cases}$1$&\text{when , $y_{M}\in M$ }\\\
$0$&\text{when, $y_{M}\not\in M$}\end{cases}$ (12)

###  III-C Inference network disentanglement representation in SIMI-ClusterGAN

The generated samples ($x^{g}$) are only considered in the $(E_{\omega})$, to
obtain cluster representations of the data in ClusterGAN. However, this can’t
guarantee that representation of data always lies in the categories of the
labels [7]. Hence, a suitable disentanglement is required to enforce the
representation to remain at categories labels. At the same time, it also
handles the mode diversity issue. The $G_{\theta}$ takes the continuous
$(z_{n})$-discrete$(z_{M})$ mixtures prior noise to generate samples $x^{g}$
and then, the generated samples passing through the $E_{\omega}$ to estimate
the $\hat{z}^{g}_{n}$ and $\hat{z}^{g}_{M}$ by the two equations 3 and 2,
respectively. These two equations are known as cyclic loss in ClusterGAN
objective Eq. 4. To estimate the posterior distribution from the real data
$(x^{r})$, the $E_{\omega}$ encodes as ($\hat{z}^{r}_{n}$, $\hat{z}^{r}_{M}$).
The $G_{\theta}$ takes the estimated categorical posterior $(\hat{z}^{r}_{M})$
with varying continuous distribution, $(z_{n})$, to generate realistic
samples, $(\hat{x}^{r})$. The encoding and decoding distributions match can be
achieved by two networks $E_{\omega}$ and $G_{\theta}$, respectively [47].
This $E_{\omega}-G_{\theta}$ reconstruction loss is defined as follows,

$J_{rec}=\mathbb{E}_{x^{r}\sim p_{r}}\parallel
x^{r}-(G_{\theta}(E_{\omega}(x^{r})))\parallel_{2}$ (13)

This loss can be looked upon as the real-reconstructing loss between
$E_{\omega}$ and $G_{\theta}$ pairs to force reconstruction levels
disentanglement.

The above cyclic losses and reconstruction losses are used to maintain the
local level disentanglement between the real space, $(x^{r})$, and the
generated space, $(x^{g})$. To enforce more disentanglement on the
$E_{\omega}$, we have added one regularisation penalty on the bounded real
discrete prior loss. We have used the same $E_{\omega}$ networks with sharing
weights that will help to update the parameters on different constraints. To
determine prior bounding loss, we draw samples from the real distribution,
$x^{r}\in p_{r}$ and $E_{\omega}$ maps real samples into real disentangle
representation variables, $\hat{z}^{r}_{n}$ and $\hat{z}^{r}_{M}$. The cross-
entropy prior bounded loss is defined as follows,

$J_{pce}=\mathbb{E}_{x^{r}\sim p_{r}}[l(z_{M},E_{\omega}(x^{r})]$ (14)

Moreover, to generate diverse samples within the same prior discrete
representation ($z_{M}$) [48], we have also introduced cross-modality loss by
considering two generated samples from the $G_{\theta}$ network. To generate
two samples $(x^{g},\hat{x}^{g})$ from $G_{\theta}$, two different variants
latent codes have been used with the same prior ($z_{M}$). They are obtained
by varying the continuous variable ($z_{n}$) only while maintaining $z_{M}$
fixed. The latent variables for cross-modality is represented as
$(z_{1},z_{2})$, where $z_{1}=(z_{n},z_{M})$ and
$z_{2}=(z^{\prime}_{n},\hat{z}^{r}_{M}=E_{\omega}(x^{r}))$. Cross-modality
loss continues to generate diverse images and penalises the generator. Cross-
modality is defined as follows,

$J_{cm}=\mathbb{E}_{x^{r}\sim p_{r}}\mathbb{E}_{z\sim p_{z}}\parallel
x_{g}-(G_{\theta}(z^{\prime}_{n},E_{\omega}(x^{r})))\parallel_{2}$ (15)

###  III-D Final objective function of SIMI-ClusterGAN

The proposed SIMI-ClusterGAN objective function incorporates all of the above
defined losses along with ClusterGAN cyclic losses. Thus, the SIMI-ClusterGAN
objective function is defined as follows,

$\begin{split}\mathop{\min}\limits_{\theta,\omega}\mathop{\max}\limits_{\eta}\operatorname{\mathbb{E}}_{x^{r}\sim
p_{r}}[f(D_{\eta}(x))]+\operatorname{\mathbb{E}}_{z\sim
p_{z}}[f(1-D_{\eta}(G_{\theta}(z)))]\\\
+\alpha_{cl}J_{ce}+\alpha_{mse}J_{mse}+\alpha_{re}J_{rec}+\alpha_{pcl}J_{pce}+\alpha_{cm}J_{cm}\end{split}$
(16)

where, the regularisation coefficients($\alpha_{re},\alpha_{pcl},\alpha_{cm}$)
are used to maintain disentanglement of encoded latent space and diversify the
generated samples.

Algorithm 1 SIMI-ClusterGAN algorithm

0: training data($D^{tra}$)= ${\left\\{{{{x^{r}}\in R^{d}},{y\in
R^{M}}}\right\\}^{S}}$, testing data($D^{tst}$)=${\left\\{{{{x^{r}}\in
R^{d}},{y\in R^{M}}}\right\\}^{T}}$, $epoch_{1}$=$60$, $epoch_{2}$=$200$,
$B_{p}$=$256$, $B_{m}$=$30$, $\beta_{p}=0.1$, $\beta_{t}=0.25$,
$\beta_{mu}=4$, ADAM optimizer($\beta_{1}=0.5$ & $\beta_{2}=0.999$), $l_{r1}$=
$0.002$ for $P_{\phi}$, $l_{r2}$= $0.0001$ for $D_{\eta}$, $G_{\theta}$,
$E_{\omega}$, Critic iter($C_{iter}$)=1, $\lambda$ =10, regularisation factors
$\alpha_{cl}=10,\alpha_{mse}=10,\alpha_{re}=1,\alpha_{pcl}=10$ and
$\alpha_{cm}=1$

0: $P_{\phi}$, $G_{\theta}$, $D_{\eta}$, $E_{\omega}$

1: Phase 1: The Prior Learning Algorithm ($P_{\phi}$)

2: for $epoch\,\,\,\,in\,\,\,\,epoch_{1}$ do

3: Samples $x^{r}\in p_{r}$ drawn in a mini-batch $B_{p}$.

4: Calculate the self-augmentation distance by using equ.(9)

5: Maximize the discrete representation($y_{M}$) by optimizing the equ.(10)

6: end for

1: Phase 2: SIMI-ClusterGAN using the learned prior($P_{\phi}$)

2: for $epoch\,\,\,in\,\,\,epoch_{2}$ do

3: for iter in $C_{iter}$ do

4: Samples $x^{r}\in p_{r}$ drawn in a mini-batch $B_{m}$.

5: Calculate discrete prior assignment $y_{M}$ = $P_{\phi}(x^{r})$.

6: One hot code representation of learned prior, $z_{M}$ by using equation
(12).

7: Samples $z_{n}\in p_{z}$ drawn in a mini-batch $B_{m}$.

8: the continuous-discrete mixture of latent space variable $z=(z_{n},z_{M})$.

9: $G_{\theta}(z)\rightarrow x^{g}$

10: WGAN 1-GP stability applied on $D_{\eta}$ parameters. Discriminator
parameters ($\eta$) are updated by the following equation,
$D_{\eta}\rightarrow D_{\eta}(x^{r})-D_{\eta}(x^{g})+\lambda(||\nabla
D_{\eta}(\hat{x}^{r})||-0)^{2}$ where, $\hat{x}^{r}=\alpha
x^{r}+(1-\alpha)x^{g}$

11: end for

12: Encodes the generated samples
$E_{\omega}(x^{g})\rightarrow(\hat{z}^{g}_{n},\hat{z}^{g}_{M})$.

13: Calculate the cyclic loss by using equations(3) and (2).

14: The reconstruction loss($J_{rec}$) between $E_{\omega}-G_{\theta}$
networks pair is obtained by equation (13).

15: The prior bounded cross entropy loss ($J_{pce}$) is obtained by equation
14.

16: The cross-modality loss ($J_{cm}$) is obtained by equation 15.

17: The generator($G_{\theta}$) network parameters ($\theta$)are updated by
the following form, $G_{\theta}\rightarrow
D_{\eta}(x^{g})+\alpha_{cl}J_{cl}+\alpha_{mse}J_{mse}+\alpha_{cm}J_{cm}+\alpha_{re}J_{rec}$

18: The inference($E_{\omega}$) network parameters ($\omega$) are updated by
the following form,
$E_{\omega}\rightarrow\alpha_{cl}J_{cl}+\alpha_{mse}J_{mse}+\alpha_{cm}J_{cm}+\alpha_{re}J_{rec}+\alpha_{pcl}J_{pce}$

19: if $epoch_{2}\mathbin{\%}1==0$ then

20: /* Encode latent spaces for $D^{tst}$ */

21: $E_{\omega}(x^{r}\in D^{tst})\rightarrow(\hat{z}_{n}^{r},\hat{z}_{M}^{r})$

22: Apply K-means on $(\hat{z}_{n}^{r},\hat{z}_{M}^{r})$ to calculate ACC and
NMI values.

23: end if

24: end for

##  IV Results & Experiments

We validate the proposed SIMI-ClusterGAN performance on several benchmark
datasets. We used an imbalanced dataset under two imbalanced situations to
verify the proposed method’s superiority over ClusterGAN. We also examine the
extensive ablation studies of each component of the proposed method’s
objective function to evaluate the clustering performance.

###  IV-A Datasets

SIMI-ClusterGAN clustering performance has been evaluated on several benchmark
datasets such as MNIST, Fashion-MNIST, USPS, Pendigits, $10\times{{}_{-}}73k$,
STL-10 [49] and CIFAR-10 [50] datasets. The information about these datasets
and their corresponding latent space dimensions are provided in Table I. The
SIMI-ClusterGAN algorithm is provided in Algorithm 1

TABLE I: Data and latent variable Dimensions Datasets | Samples | Labels |  Dimensions $\in R^{d}$ | $z_{n}$ | $z_{M}$
---|---|---|---|---|---
MNIST [51] | 70k | 10 | $1\times 28\times 28$ | 30 | 10
Fashion-MNIST [51] | 70k | 10 | $1\times 28\times 28$ | 30 | 10
USPS [6] | 9,298 | 10 | $1\times 16\times 16$ | 30 | 10
Pendigits [7] | 10992 | 10 | $1\times 16$ | 5 | 10
$10\times{\\_}73k$ [7] | 73233 | 8 | $1\times 720$ | 30 | 8
CIFAR-10 [44] | 60k | 10 | $3\times 32\times 32$ | 50 | 10
STL-10 [44] | 13k | 10 | $3\times 96\times 96$ | 100 | 10

###  IV-B Simulations Implementation

Four different neural networks $P_{\phi},G_{\theta},D_{\eta},E_{\omega}$ have
been developed for handling the four different tasks; prior assignment
learning, generator, discriminator and clustering inference network,
respectively. The $P_{\phi}$ is used to learn the prior indices from the data
using the self-augmentation maximization principle. This prior assignment is
used as the discrete one-hot code variable to train the improved clustering
GAN. The data is normalised in the range of $[-1,1]$ for all datasets except
STL-10 $[0,1]$. The tangent-hyperbolic and sigmoid activation functions are
used in the last layer of the $G_{\theta}$ network. However, for learning the
clustering prior from the data, we have used a $P_{\phi}$ network with three
MLP hidden layers given by ($d-1200-1200-M$) for the MNIST, Fashion-MNIST,
CIFAR-10, STL-10 datasets in which $d$ $\&$ $M$ represent input data dimension
and cluster assignments index respectively. For lower dimensional data such as
Pendigits, USPS, and $10\times{{}_{-}}73k$ , we have used the $P_{\phi}$
network as $d-256-256-M$. For MNIST and Fashion-MNIST, the conv-trans.conv
layer with ReLU and LeakyReLU activation functions have been used in
$G_{\theta},D_{\eta}$ and $E_{\omega}$ networks respectively [7]. The
clustering network $E_{\omega}$, has the same structure of $D_{\eta}$ to
estimate continuous($\hat{z}_{n}$) and discrete ($\hat{z}_{M}$) variables. For
Pendigits, $10\times{{}_{-}}73k$ and USPS datasets, we have used the two
layers MLP models for $G_{\theta},D_{\eta}$, and $E_{\omega}$. Our proposed
SIMI-ClusterGAN method has two learning stages. First, the discrete prior
learns $M$ categorical discretization representations from data by using the
$P_{\phi}$ network. The learning hyper-parameters are set to be
$\beta_{p}=0.1$, mutual information coefficient $\beta_{mu}=4$, perturbation
coefficient $\beta_{t}=0.25$ while updating the $P_{\phi}$ network. We have
used the ADAM optimizer with learning rate of $0.002$. Once the $M$
categorical representation is obtained then it is formulated into one-hot-code
$(z_{M})\in R^{M}$ to create $M$ discrete priors [7]. We have added the
Gaussian Normal distribution,
$z_{n}=\mathcal{N}({\mu=0,\sigma^{2}*I_{dn}},\sigma=0.1)$, into $z_{M}$ to
create the $M$ discrete non-smooth surfaces. The noise samples are drawn from
the continuous and discrete mixture variables to generate the realistic
samples. Moreover, the learning rate of $G_{\theta},D_{\eta}$, $E_{\omega}$
networks are set to $1e-4$. Similarly for $P_{\phi}$ network, we have adapted
the ADAM optimiser with learning hyper-parameters
$(\beta_{1}=0.5,\beta_{2}=0.999)$ [7, 40]. To determine the cluster labels
from the real datasets, the $K-means$ [9] algorithm is applied to estimate
encode latent space from the $E_{\omega}$ network [7]. The variation factors
of each loss are controlled by five multiplication factors given by
$\alpha_{cl}=10$ [7], $\alpha_{mse}=10$ [7], $\alpha_{re}=1$,
$\alpha_{pcl}=10$ and $\alpha_{cm}=1$ . The simulations are implemented on
PyTorch environment with NVIDIA GTX GPU.

###  IV-C Clustering performance Evaluation

TABLE II: Quantitative analysis of the clustering performance

Datasets | MNIST | Fashion-MNIST | USPS | Pendigits | $10\times{{}_{-}}73k$
---|---|---|---|---|---
Performance Index | Performance Index | Performance Index | Performance Index | Performance Index
Models | ACC | NMI | ACC | NMI | ACC | NMI | ACC | NMI | ACC | NMI
K-means [9] | 0.532 | 0.500 | 0.474 | 0.501 | 0.668 | 0.601 | 0.793 | 0.730 | 0.623 | 0.577
NMF [52] | 0.471 | 0.452 | 0.500 | 0.510 | 0.652 | 0.693 | 0.670 | 0.580 | 0.710 | 0.690
SC [53] | 0.656 | 0.731 | 0.660 | 0.704 | 0.649 | 0.794 | 0.700 | 0.690 | 0.400 | 0.290
AGGLO [54] | 0.640 | 0.650 | 0.550 | 0.570 | - | - | 0.700 | 0.690 | 0.630 | 0.580
DEC [5] | 0.863 | 0.834 | 0.518 | 0.546 | 0.762 | 0.767 | - | - | - | -
DCN [4] | 0.802 | 0.786 | 0.563 | 0.608 | - | - | 0.720 | 0.690 | - | -
JULE [25] | 0.964 | 0.913 | 0.563 | 0.608 | 0.950 | 0.913 | - | - | - | -
DEPICT [55] | 0.965 | 0.917 | 0.392 | 0.392 | 0.899 | 0.906 | - | - | - | -
SpectralNet [56] | 0.800 | 0.814 | - | - | - | - | - | - | - | -
Dual-AE [6] | 0.978 | 0.941 | 0.662 | 0.645 | 0.869 | 0.857 | - | - | - | -
InfoGAN [24] | 0.890 | 0.860 | 0.610 | 0.590 | - | - | 0.720 | 0.730 | 0.620 | 0.580
ClusterGAN [7] | 0.950 | 0.890 | 0.630 | 0.640 | $0.798^{*}$ | $0.703^{*}$ | 0.770 | 0.730 | 0.810 | 0.730
GAN with bp [7] | 0.950 | 0.900 | 0.560 | 0.530 | - | - | 0.760 | 0.710 | 0.650 | 0.59
CaTGAN [57] | 0.890 | 0.900 | 0.550 | 0.600 | - | - | - | - | - | -
GANMM [58] | 0.640 | 0.610 | 0.340 | 0.270 | - | - | - | - | - | -
NEMGAN_v [23] | 0.960 | 0.910 | 0.650 | 0.610 | - | - | - | - | - | -
SIMI-ClusterGAN | 0.986 | 0.958 | 0.745 | 0.705 | 0.951 | 0.894 | 0.857 | 0.825 | 0.917 | 0.878

No available data : $(-)$, simulated :$*$ .

We mainly focus on two popular evaluation indices given by accuracy (ACC) [5]
and normalised mutual information (NMI) [5] for comparing the clustering
performance. The ACC is defined as follows,

$ACC=max_{\delta}\frac{\sum_{i=1}^{N}\boldsymbol{1}\left\\{c_{i}=\delta(y^{k}_{i})\right\\}}{N}$
(17)

where, $c_{i}\in R^{M}$ is the $i$-th ground truth labels, $y^{k}_{i}$ is
$i$-th predicted labels which is obtained by applying $K$-means [9] method on
encoded latent-space($E_{\omega}$) [7] . The $\delta$ covers all potential
one-to-one mappings between true and predicted labels of data. This measure
identifies the best match between an unsupervised algorithm’s cluster
prediction and a ground truth assignment. The Hungarian method efficiently
computes the optimum mapping [59]. Similarly, NMI is defined as follows,

$NMI=\frac{2\times I(Y;\textbf{c})}{H(Y)+H(\textbf{c})}$ (18)

where, $I(Y;\textbf{c})$ is defined mutual information between predicted and
ground truth labels. NMI determines similarity between the true class and the
predicted class labels, which is bounded by $0$ (worst similarity) to $1$
(ultimate similarity).

Table II represents the clustering results of SIMI-ClusterGAN and other state-
of-the art methods on the five benchmark datasets which are mostly dealt with
single channel image datasets and a tabular dataset. The best evolution
matrices are highlighted in the table II with 5 runs [7]. It is clearly seen
from the Table II that deep neural network based clustering methods attained
better performance compared to traditional machine-learning approaches. In
comparison, our proposed SIMI-ClusterGAN achieved significant performance
improvement for all datasets. For the MNIST dataset, our proposed method
obtained better performance than the second best results of Dual-AE (ACC: .978
vs 0.986, NMI: .941 vs .958). While learning discrete prior ($P_{\phi}$) from
the Fashion-MNIST dataset, we have used the following affine distortion to
maximize the mutual information between categories($Y_{M}$). The following
transformations are adopted to learn the $P_{\phi}$ network parameters,

  * •

Uniformly drawn samples along with random rotation by $\chi$ within
$[-10^{\circ},10^{\circ}]$

  * •

Uniformly drawn samples along with $x$ axis and $y$-axis by scaling factor
$[.3,1]$

  * •

Uniformly drawn samples along with $x$ axis and $y$-axis by random shearing
$(-0.3,0.3)$

The performance has improved significantly due to random affine transformation
on the data since our proposed method is heavily dependent on the learning of
the prior. Besides the affine transformations of data, self-augmentation is
also applied to learn prior for the proposed SIMI-ClusterGAN method. The $ACC$
and $NMI$ values are $0.721$ and $0.675$, which are much better than the
ClusterGAN. Similarly, for $10\times{{}_{-}}73k$ dataset, our proposed method
performance is improved significantly compared with ClusterGAN (ACC: 0.810 vs
0.917 , NMI: 0.730 vs 0.878 ). Specifically, for $16$-dimensional pendigits
dataset, the K-means method outperformed all the GANs methods. But, our
proposed SIMI-ClusterGAN obtained better performance on both evaluation
indices (ACC:0.793 vs 0.857, NMI:0.730 vs 0.825). For the USPS dataset, the
best NMI is obtained by JULE but our proposed method achieved the best results
in ACC. The ClusterGAN method∗111https://github.com/zhampel/clusterGAN
obtained ACC and NMI values as 0.798 and 0.703 respectively.

(a)

(b)

Figure 2: Samples generated by SIMI-ClusterGAN for MNIST and Fashion-MNIST
datasets. Each row represents each class. The generated classes maintained
purity without mixtures of other classes

(a) Sandal

(b) Sneaker

Figure 3: Samples generated by ClusterGAN for Fashion-MNIST dataset

All the distinctive classes are fully achieved by SIMI-ClusterGAN method. It
is observed from Figure 2 that the SIMI-ClusterGAN is able to generate all the
modes without supervision with higher accuracy. For MNIST dataset, all the
distinctive classes are represented in each row of Figure 2(a).Similarly, for
Fashion-MNIST, our proposed method is able to generate all the classes without
supervision as shown in Figure 2(b). For fair comparisons of our proposed
method with ClusterGAN, we have plotted two distinctive generated
classes(”Sandal” and ”Sneaker”) for better understandings. ClusterGAN is not
able to generate two distinctive classes without purity, as can be seen in
Figure 3. As shown in Figure 4,however, the proposed SIMI-ClusterGAN is
capable of generating two different classes with a high degree of purity.

(a) Sandal

(b) Sneaker

Figure 4: Samples generated by SIMI-ClusterGAN for Fashion-MNIST dataset

###  IV-D Complex high-dimensional dataset performance

We evaluate the proposed SIMI-ClusterGAN clustering performance on complex
coloured image datasets: CIFAR-10 and STL-10. For learning the clustering
prior, the colour image can’t be directly used in the $P_{\phi}$ network
because associated raw pixels of images are colour dominant [44]. Therefore,
for learning the prior clustering assignments, we have used 50-layer
pretrained features extractor from raw data [44, 60]. We are not directly
using the class information to $P_{\phi}$ network. Instead, the well-separated
lower dimensional features are used to get the class assignment prior. For
both the cases, the extracted features size is $2048$. Once, the prior
assignment network is learned from extracted features data, the learned prior
is used to train SIMI-ClusterGAN. The clustering performance of feature space
data is listed in the Table III. It is clearly observed from the table III
that our proposed method has outperformed other state-of-the-art methods. The
uniform prior based ClusterGAN accuracy performance for CIFAR-10 is obtained
as $15.9$. This is because the clustering loss in clustering inference network
is focused on the generated samples’ semantics instead of categories of data
[7]. The ClusterGAN accuracy performance for STL-10 dataset is $12.6$.
Similarly for the STL-10 dataset, our proposed method SIMI-ClusterGAN obtained
better results.

TABLE III: Quantitative clustering performance for High-dimensional datasets

| CIFAR-10 | STL-10
---|---|---
Methods | ACC | NMI | ACC | NMI
K-means[44] | 0.344 | - | 0.856 | -
DAE+KMeans[44] | 0.442 | - | 0.722 | -
DEC[44] | 0.469 | - | 0.781 | -
Linear RIM[44] | 0.403 | - | 0.735 | -
Deep RIM[44] | 0.403 | - | 0.925 | -
Linear IMSAT[44] | 0.407 | - | 0.917 | -
IMSAT(RPT)[44] | 0.455 | - | 0.928 | -
IMSAT(VAT)[44] | 0.456 | - | 0.941 | -
VADE [61] | - | - | 0.844 | -
DDG [6] | - | - | 0.905 | -
Sarfaraz [62] | - | - | 0.952 | -
ClusterGAN | $0.159^{*}$ | $0.025^{*}$ | $0.126^{*}$ | $0.096^{*}$
SIMI-ClusterGAN | 0.512 | 0.412 | 0.954 | 0.861
Embedding feature-space clustering [44], simulated :$*$ . |  |  |  |

(a) ClusterGAN

(b) SIMI-ClusterGAN

Figure 5: Encoding latent space visualisation for CIFAR-10 datset where colors
are used to differentiate between distinct types of classes.

When dealing with high-dimensional datasets such as CIFAR10 and STL-10, the
disentanglement at $E_{\omega}$ is very important in determining the accuracy
of clustering. In order to facilitate comprehension, the superiority of the
SIMI-ClusterGAN over the ClusterGAN is displayed in Figure 5, which shows
estimated encoded latent spaces from both methods in comparison to each other.
Due to the joint learning framework of $E_{\omega}-G_{\theta}$ networks in the
ClusterGAN method, $E_{\omega}$ is more reliable on generated sample semantics
than focusing on downstream tasks like clustering. When we incorporate prior
learning with three additional losses, the encoded latent spaces are well
separated from ClusterGAN, which will lead to a significant improvement in
clustering performance. Similar phenomena are also being observed for the
STL-10 dataset. The encoded latent representation for the STL-10 dataset is
depicted in Fig 6 where both clustering methods’ results are shown.

(a) ClusterGAN

(b) SIMI-ClusterGAN

Figure 6: Encoding latent space visualisation for STL-10 datset where colors
are used to differentiate between distinct types of classes.

###  IV-E Latent space interpolation

The one-hot code $(z_{M})$ variable of the proposed SIMI-ClusterGAN method
represents the cluster information in the data. The continuous variable
($z_{n}$) is controlling the variation of Gaussian representation in the
latent space. In addition, we have also used the interpolation between
different classes while varying two discrete variables $z_{M}^{1}$,
$z_{M}^{2}$. The new continuous-discrete latent space vector is represented by
$z=(z_{n},\tau z_{M}^{1}+(1-\tau)z_{M}^{2})$ where $\tau\in[0,1]$, and $z_{n}$
is fixed. The smooth transition between one class to another class is depicted
in Figure 7 for the two datasets MNIST and Fashion-MNIST.

(a) MNIST

(b) Fashion-MNIST

Figure 7: Latent space interpolation between classes

###  IV-F Ablation studies

We have extensively analysed the proposed SIMI-ClusterGAN components in the
following subsections.

####  IV-F1 different losses in SIMI-ClusterGAN

We also evaluate each loss function in the proposed SIMI-ClusterGAN objective
function. The performance on Fashion-MNIST data by SIMI-ClusterGAN is reported
in Table IV. In Table IV, the first loss is related when only the generated
samples are passed to clustering inference network [7]. The uniform
distribution based prior learning is not adequate to improve the clustering
performance while using only generated samples. The performance is below the
ClusterGAN performance. While considering the reconstruction loss into
proposed objective function, the performance has improved significantly. All
losses have an individual impact on the clustering performance to get accurate
categories in fully unsupervised way. Figure 8 depicts the encoding latent
space in 2D for Fashion-MNIST dataset. Figure 8(a) depicts the raw-data in 2D
encoding latent space. Considering the generated samples to the encoding
network, the t-SNE [63] plot of encoded latent space is illustrated in Figure
8(b) which gives better representation than the raw-data. In addition, the
encoding space gets better separated while considering the reconstruction loss
into the proposed objective function. As illustrated in the Fig 8(d), the
SIMI-ClusterGAN gives better discrimination on encoding space while
considering all the losses into proposed objective function.

(a) Raw-data

(b) $J_{ce}+J_{mse}$

(c) $J_{ce}+J_{mse}+J_{rec}$

(d) $J_{ce}+J_{mse}+J_{rec}+J_{cm}+J_{pce}$

Figure 8: Visualisation of individual loss performance through t-SNE [63] TABLE IV:  Effects of different losses on clustering performance of Fashion-MNIST dataset Dataset | Fashion-MNIST
---|---
NMI | ACC
$J_{ce}$ \+ $J_{mse}$ | 0.5796 | 0.5853
$J_{ce}$ \+ $J_{mse}$ \+ $J_{rec}$ | 0.6580 | 0.6996
$J_{ce}$ \+ $J_{mse}$ \+ $J_{rec}$ +$J_{cm}$ | 0.6620 | 0.7056
all losses | 0.7053 | 0.7459

####  IV-F2 Variations of $\sigma$ in $z_{n}$

A non-smooth latent variable is formed when the continuous variable($z_{n}$)
is mixed with the discrete variables($z_{M}$). Because the non-smooth bounded
prior has an effect on clustering performance, it is thus necessary to assess
the performance of clustering while changing the variance($\sigma$) of the
continuous variable($z_{n}$). Therefore, we have illustrated the clustering
performance on the Figure 9 when $\sigma$ is varied incrementally by $0.3$.
When the variance is at its highest ($\sigma=1.0$), ClusterGAN suffers a
significant performance drop. Our suggested SIMI-ClusterGAN, in contrast to
ClusterGAN, has no performance issues. Therefore, SIMI-ClusterGAN has a more
robust performance than ClusterGAN.

(a) ACC

(b) NMI

Figure 9: Varying $\sigma$ in $z_{n}$

####  IV-F3 Uniform Prior based SIMI-ClusterGAN

We also examined the clustering performance of our proposed method with the
ClusterGAN when the $P_{\phi}$ network is not involved in the adversarial
game. Thus, the SIMI-ClusterGAN prior comes from uniform distribution, as it
is like ClusterGAN. Table V depicts the quantitative performance comparison
between ClusterGAN and SIMI-ClusterGAN. It is clearly observed from the Table
V that SIMI-ClusterGAN produces more disentanglement’s by considering the real
data distribution into $E_{\omega}$ network. However, for CIFAR-10 dataset,
the adequate performance can’t be obtained by considering uniform prior
distribution. When the $E_{\omega}-G_{\theta}$ networks work together to
learn, the $E_{\omega}$ focus is on semantics of the data rather than
clustering. Therefore, prior learning is required for forcing $E_{\omega}$
network to match the real data distribution.

TABLE V: Uniform prior based SIMI-ClusterGAN performance Methods | MNIST | CIFAR10
---|---|---
| ACC | NMI | ACC | NMI
ClusterGAN | 0.950 | 0.890 | 0.159 | 0.025
SIMI-ClusterGAN \$P_{\phi}$ | 0.972 | 0.921 | 0.228 | 0.067

###  IV-G Imbalanced Dataset Performance

We have also validated the performance of the proposed method with ClusterGAN
under two different imbalanced conditions: single class imbalance and multi-
class imbalance. MNIST dataset has been considered in which two class
imbalanced settings have been tested. For single class imbalanced settings,
$0.1$ fraction of zero samples (‘digit-0’) and remaining all the samples have
been taken for training whereas all the test samples($10K$) are used for
testing the clustering performance. Similarly for multi-class imbalanced
settings, three imbalanced classes are ‘digit-1’, ‘digit-3’and ‘digit-5’with
fraction of $0.1,0.3\ $, and $\,0.5$. Similar to the one class-imbalanced
case, all the testing data samples are used to test the clustering inference
network. The performance of the proposed SIMI-ClusterGAN is shown in the Table
VI. It is clearly observed from the Table VI that our proposed method out-
performs uniform prior based ClusterGAN performance. For better understanding
of $E_{\omega}$ performance for both the methods, we have visualised encoding
latent space is shown in Figure 10. It is clearly observed from the figure
that our proposed method gives better separable decision boundary than the
ClusterGAN.

TABLE VI: Imbalanced Clustering Performance Methods | MNIST
---|---
| 1-Class | 3-Classes
| ACC | NMI | ACC | NMI
ClusterGAN | 0.892 | 0.836 | 0.812 | 0.787
SIMI-ClusterGAN | 0.969 | 0.923 | 0.949 | 0.909

(a) ClusterGAN

(b) SIMI-ClusterGAN

Figure 10: Encoding latent space visualisation by t-SNE[63] for one class
imbalanced condition

For three classes imbalanced conditions, both the methods failed to generate
the minor ‘digit-1’which is shown in the Figure 11. However, for other two
minor classes (‘digit-3’and ‘digit-5’), proposed SIMI-ClusterGAN is able to
generate pure classes. For better visual understanding, we have plotted with
two minor classes data in Figure 13 in which ‘50’samples were generated for
each class. It is clearly observed from the Figure 13 that our proposed method
is able to generate both classes with high purity. However, ClusterGAN,
generated mixture of samples of ‘digit-3’and ‘digit-5’and $50$ samples were
generated for digit-8. We have also visualised encoding latent space in 2D for
both the methods in Figure 12.

(a) ClusterGAN

(b) SIMI-ClusterGAN

Figure 11: Samples generated by ClusterGAN and SIMI-ClusterGAN under three
classes imbalanced condition.

(a) ClusterGAN

(b) SIMI-ClusterGAN

Figure 12: Encoding latent space visualisation by t-SNE[63] for three classes
imbalanced condition

(a) ClusterGAN:’digit-3’ and ’digit-5’

(b) SIMI-ClusterGAN: ’digit-3’ and ’digit-5’

Figure 13: Two minor classes generated by ClusterGAN and SIMI-ClusterGAN

##  V conclusion

In this paper, we have proposed a SIMI-ClusterGAN which is the modification of
uniform prior latent space based ClusterGAN method. The proposed method
consists of four networks, a prior network, a generator, a discriminator and a
clustering inference network. The proposed method is followed by two stage
learning methods: learning categorical prior from the data directly and then
the learned prior is used to modify clustering performance through adversarial
training between conditional generator and discriminator. The simulation
results indicate that the proposed SIMI-ClusterGAN achieved better performance
than the state-of-the-art methods. The effectiveness of each loss has been
studied through an ablation study. It is also observed that the proposed
method can handle imbalanced conditions. However, if the data distribution is
highly imbalanced in nature, the SIM-based $P_{\phi}$ won’t be able to capture
how the real data is distributed. Consequently, SIMI-ClusterGAN fails to
generate the minority classes. Rather than relying on two-stage techniques, we
will instead leverage self-supervised losses[64] to develop a single-stage
learning technique that simultaneously updates prior and clustering inference
networks.

## References

  * [1] K.-S. Chuang, H.-L. Tzeng, S. Chen, J. Wu, and T.-J. Chen, “Fuzzy c-means clustering with spatial information for image segmentation,” _computerized medical imaging and graphics_ , vol. 30, no. 1, pp. 9–15, 2006\.
  * [2] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 132–149.
  * [3] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.-A. Manzagol, and L. Bottou, “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.” _Journal of machine learning research_ , vol. 11, no. 12, 2010.
  * [4] B. Yang, X. Fu, N. D. Sidiropoulos, and M. Hong, “Towards k-means-friendly spaces: Simultaneous deep learning and clustering,” in _international conference on machine learning_ , 2017, pp. 3861–3870.
  * [5] J. Xie, R. Girshick, and A. Farhadi, “Unsupervised deep embedding for clustering analysis,” in _International conference on machine learning_ , 2016, pp. 478–487.
  * [6] X. Yang, C. Deng, F. Zheng, J. Yan, and W. Liu, “Deep spectral clustering using dual autoencoder network,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4066–4075.
  * [7] S. Mukherjee, H. Asnani, E. Lin, and S. Kannan, “Clustergan: Latent space clustering in generative adversarial networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 4610–4617.
  * [8] E. Min, X. Guo, Q. Liu, G. Zhang, J. Cui, and J. Long, “A survey of clustering with deep learning: From the perspective of network architecture,” _IEEE Access_ , vol. 6, pp. 39 501–39 514, 2018.
  * [9] J. MacQueen _et al._ , “Some methods for classification and analysis of multivariate observations,” in _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_ , vol. 1, no. 14. Oakland, CA, USA, 1967, pp. 281–297.
  * [10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in neural information processing systems_ , 2014, pp. 2672–2680.
  * [11] L. Ma, Y. Ma, Q. Lin, J. Ji, C. A. C. Coello, and M. Gong, “Snegan: Signed network embedding by using generative adversarial nets,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 2020.
  * [12] J. He, L. Shen, Y. Yao, H. Wang, G. Zhao, X. Gu, and W. Ding, “Finger vein image deblurring using neighbors-based binary-gan (nb-gan),” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 2021.
  * [13] Q. Li, H. Qu, Z. Liu, N. Zhou, W. Sun, S. Sigg, and J. Li, “Af-dcgan: Amplitude feature deep convolutional gan for fingerprint construction in indoor localization systems,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 2019.
  * [14] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 1125–1134.
  * [15] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “Wavenet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [16] H. Tang, D. Xu, Y. Yan, J. J. Corso, P. H. Torr, and N. Sebe, “Multi-channel attention selection gans for guided image-to-image translation,” _arXiv preprint arXiv:2002.01048_ , 2020.
  * [17] L. Jing and Y. Tian, “Self-supervised visual feature learning with deep neural networks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [18] C.-H. Liu, H. Chang, and T. Park, “Da-cgan: A framework for indoor radio design using a dimension-aware conditional generative adversarial network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , 2020, pp. 498–499.
  * [19] C. Zheng, S. Yang, J. M. Parra-Ullauri, A. Garcia-Dominguez, and N. Bencomo, “Reward-reinforced generative adversarial networks for multi-agent systems,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , 2021.
  * [20] X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel, “Infogan: Interpretable representation learning by information maximizing generative adversarial nets,” in _Advances in neural information processing systems_ , 2016, pp. 2172–2180.
  * [21] S. Arora, A. Risteski, and Y. Zhang, “Do gans learn the distribution? some theory and empirics,” in _International Conference on Learning Representations_ , 2018.
  * [22] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” in _International Conference on Learning Representations_ , 2018.
  * [23] D. Mishra, J. Aravind, and A. Prathosh, “Effect of the latent structure on clustering with gans,” _IEEE Signal Processing Letters_ , 2020.
  * [24] J. Chang, L. Wang, G. Meng, S. Xiang, and C. Pan, “Deep adaptive image clustering,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5879–5887.
  * [25] J. Yang, D. Parikh, and D. Batra, “Joint unsupervised learning of deep representations and image clusters,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5147–5156.
  * [26] X. Guo, L. Gao, X. Liu, and J. Yin, “Improved deep embedded clustering with local structure preservation.” in _IJCAI_ , 2017, pp. 1753–1759.
  * [27] M. F. Mathieu, J. J. Zhao, J. Zhao, A. Ramesh, P. Sprechmann, and Y. LeCun, “Disentangling factors of variation in deep representation using adversarial training,” in _Advances in neural information processing systems_ , 2016, pp. 5040–5048.
  * [28] A. Gonzalez-Garcia, J. Van De Weijer, and Y. Bengio, “Image-to-image translation for cross-domain disentanglement,” in _Advances in neural information processing systems_ , 2018, pp. 1287–1298.
  * [29] M. Tschannen, O. Bachem, and M. Lucic, “Recent advances in autoencoder-based representation learning,” _arXiv preprint arXiv:1812.05069_ , 2018.
  * [30] Z. Pan, W. Yu, B. Wang, H. Xie, V. S. Sheng, J. Lei, and S. Kwong, “Loss functions of generative adversarial networks (gans): Opportunities and challenges,” _IEEE Transactions on Emerging Topics in Computational Intelligence_ , vol. 4, no. 4, pp. 500–522, 2020.
  * [31] N. Hadad, L. Wolf, and M. Shahar, “A two-step disentanglement method,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 772–780.
  * [32] M. Patacchiola, P. Fox-Roberts, and E. Rosten, “Y-autoencoders: disentangling latent representations via sequential-encoding,” _arXiv preprint arXiv:1907.10949_ , 2019.
  * [33] M. Ye and J. Shen, “Probabilistic structural latent representation for unsupervised embedding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5457–5466.
  * [34] R. T. Chen, X. Li, R. B. Grosse, and D. K. Duvenaud, “Isolating sources of disentanglement in variational autoencoders,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 2610–2620.
  * [35] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, “beta-vae: Learning basic visual concepts with a constrained variational framework,” 2016.
  * [36] H. Kim and A. Mnih, “Disentangling by factorising,” _arXiv preprint arXiv:1802.05983_ , 2018.
  * [37] E. Dupont, “Learning disentangled joint continuous and discrete representations,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 710–720.
  * [38] S. Gurumurthy, R. Kiran Sarvadevabhatla, and R. Venkatesh Babu, “Deligan: Generative adversarial networks for diverse and limited data,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 166–174.
  * [39] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, “Improved training of wasserstein gans,” in _Advances in neural information processing systems_ , 2017, pp. 5767–5777.
  * [40] T. Dam, M. M. Ferdaus, S. G. Anavatti, S. Jayavelu, and H. A. Abbass, “Does adversarial oversampling help us?” in _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, 2021, pp. 2970–2973.
  * [41] A. Creswell and A. A. Bharath, “Inverting the generator of a generative adversarial network,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 7, pp. 1967–1974, 2018.
  * [42] Z. C. Lipton and S. Tripathi, “Precise recovery of latent vectors from generative adversarial networks,” _arXiv preprint arXiv:1702.04782_ , 2017\.
  * [43] A. Krause, P. Perona, and R. G. Gomes, “Discriminative clustering by regularized information maximization,” in _Advances in neural information processing systems_ , 2010, pp. 775–783.
  * [44] W. Hu, T. Miyato, S. Tokui, E. Matsumoto, and M. Sugiyama, “Learning discrete representations via information maximizing self-augmented training,” _arXiv preprint arXiv:1702.08720_ , 2017.
  * [45] Y. Grandvalet and Y. Bengio, “Semi-supervised learning by entropy minimization,” in _Advances in neural information processing systems_ , 2005, pp. 529–536.
  * [46] T. Miyato, S.-i. Maeda, M. Koyama, K. Nakae, and S. Ishii, “Distributional smoothing with virtual adversarial training,” _arXiv preprint arXiv:1507.00677_ , 2015.
  * [47] J. Zhao, Y. Kim, K. Zhang, A. Rush, and Y. LeCun, “Adversarially regularized autoencoders,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 5902–5911.
  * [48] Q. Mao, H.-Y. Lee, H.-Y. Tseng, S. Ma, and M.-H. Yang, “Mode seeking generative adversarial networks for diverse image synthesis,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1429–1437.
  * [49] A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , 2011, pp. 215–223.
  * [50] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [51] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” _arXiv preprint arXiv:1708.07747_ , 2017.
  * [52] D. D. Lee and H. S. Seung, “Learning the parts of objects by non-negative matrix factorization,” _Nature_ , vol. 401, no. 6755, pp. 788–791, 1999\.
  * [53] J. Shi and J. Malik, “Normalized cuts and image segmentation,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 22, no. 8, pp. 888–905, 2000.
  * [54] W. Zhang, X. Wang, D. Zhao, and X. Tang, “Graph degree linkage: Agglomerative clustering on a directed graph,” in _European Conference on Computer Vision_. Springer, 2012, pp. 428–441.
  * [55] K. Ghasedi Dizaji, A. Herandi, C. Deng, W. Cai, and H. Huang, “Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 5736–5745.
  * [56] U. Shaham, K. Stanton, H. Li, B. Nadler, R. Basri, and Y. Kluger, “Spectralnet: Spectral clustering using deep neural networks,” _arXiv preprint arXiv:1801.01587_ , 2018.
  * [57] J. T. Springenberg, “Unsupervised and semi-supervised learning with categorical generative adversarial networks,” _arXiv preprint arXiv:1511.06390_ , 2015.
  * [58] Y. Yu and W.-J. Zhou, “Mixture of gans for clustering.” in _IJCAI_ , 2018, pp. 3047–3053.
  * [59] H. W. Kuhn, “The hungarian method for the assignment problem,” _Naval research logistics quarterly_ , vol. 2, no. 1-2, pp. 83–97, 1955.
  * [60] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [61] Z. Jiang, Y. Zheng, H. Tan, B. Tang, and H. Zhou, “Variational deep embedding: an unsupervised and generative approach to clustering,” in _Proceedings of the 26th International Joint Conference on Artificial Intelligence_ , 2017, pp. 1965–1972.
  * [62] S. Sarfraz, V. Sharma, and R. Stiefelhagen, “Efficient parameter-free clustering using first neighbor relations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8934–8943.
  * [63] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [64] Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng, “Contrastive clustering,” in _2021 AAAI Conference on Artificial Intelligence (AAAI)_ , 2021.