# Clustering-Induced Generative Incomplete Image-Text Clustering(CIGIT-C)

Xiaoming Su${}^{1,2\corr}$ Dongjin Guo1,2 Jiatai Wang1,2, Limin Liu1,2
1College of Data Science and Application,Inner Mongolia University of
Technology, Huhhot, China 2Big Data Laboratory of Inner Mongolia Discipline
Supervision and Investigation, Huhhot, China Correspondence
Xiaoming Su, College of Data Science and Application, Inner Mongolia
University of Technology, Huhhot, China
Email<EMAIL_ADDRESS>
###### Abstract

The target of image-text clustering (ITC) is to find correct clusters by
integrating complementary and consistent information of multi-modalities for
these heterogeneous samples. However, the majority of current studies analyse
ITC on the ideal premise that the samples in every modality are complete. This
presumption, however, is not always valid in real-world situations. The
missing data issue degenerates the image-text feature learning performance and
will finally affect the generalization abilities in ITC tasks. Although a
series of methods have been proposed to address this incomplete image text
clustering issue (IITC), the following problems still exist: 1) most existing
methods hardly consider the distinct gap between heterogeneous feature
domains. 2) For missing data, the representations generated by existing
methods are rarely guaranteed to suit clustering tasks. 3) Existing methods do
not tap into the latent connections both inter and intra modalities. In this
paper, we propose a Clustering-Induced Generative Incomplete Image-Text
Clustering(CIGIT-C) network to address the challenges above. More
specifically, we first use modality-specific encoders to map original features
to more distinctive subspaces. The latent connections between intra and inter-
modalities are thoroughly explored by using the adversarial generating network
to produce one modality conditional on the other modality. Finally, we update
the corresponding modality-specific encoders using two KL divergence losses.
Experiment results on public image-text datasets demonstrated that the
suggested method outperforms and is more effective in the IITC job.

##  1 INTRODUCTION

The two primary ways of displaying information in daily life are visual media
and natural language. Therefore, as a fundamental research topic in machine
learning and data science communities, ITC has drawn an increasing amount of
interest in recent years. It is advantageous for a variety of applications,
including vehicle detection Gomaa et al. [2020] Gomaa et al. [2018] Gomaa et
al. [2019], clinical medicine Horne et al. [2020], marketing research,
crowdsourced mobile testing Li et al. [2021], and recommender system Do et al.
[2020].

Existing ITC methods are divided into two categories in the literature:
conventional and deep approaches. The conventional methods, while they have
progressed, still rely on manually created features and linear embedding
functions. They are unable to grasp the non-linear structure of complex cross-
modal data as a consequence. In DNN-based ITC methods, they are usually
classified into the following two categories. The first category employs a
two-stage procedure, in which features based on DNN are initially extracted,
and then the final clustering results are learned using the conventional
clustering method. The second category establishes a clustering loss to steer
the multi-modal feature learning process. The success of ITC relies on the
assumption of information completeness: instances in all modalities correspond
to each other.

Figure 1: The reason for incomplete multi-view clustering approach does not
solve the IITC task well. Because the incomplete multi-view approach targets
homogeneous modalities, as shown in the two yellow boxes, i.e., (Image Space)
and (Image Space (HOG)). IITC essentially solves the problem of heterogeneous
modalities, which belong to the paradigm of multi-modal clustering methods, as
shown in yellow and blue rectangles in the figure, i.e., (Image Space) and
(Text Space). Figure 2: Four imputation strategies for missing values. (a)
Zero or mean-based methods. (b) Common space learning-based method. (c) Cycle
GAN-based method. (d) Ours proposed CIGIT-C. Because of their heterogeneity,
the solid triangles and circles in the figure represent the image and text
modalities. Circles and triangles of the same colour represent the same
instance. The graph drawn in the dashed box is a representation of the missing
modalities.

However, in real-life cases, some instances could lack paired instances of
different modalities. For instance, on Twitter, tweets can contain texts or
images, but not all contain paired modalities. The missing modalities not only
lead to information loss but also increase the difficulties of excavating
complementary information. In general, most present IITC algorithms rely
heavily on the idea of incomplete multi-view clustering. Over the past few
years, several efforts have been dedicated to addressing IITC, with various
imputation strategies (i.e., zero or mean-based methods, common space
learning-based methods, generative methods) to fill in the missing samples. As
shown in Figure2(a), these methods Wen et al. [2021a] adopt zero or mean
values to pad missing data firstly and then design specific machine learning
techniques to conduct multi-view clustering. These simply padded data are very
different from the actual data and can reduce the performance of the final
clustering. To alleviate this challenge, a common space learning-based method
has been introduced, as shown in Figure2(b). The goal of the method is to
learn a common representation using complete and aligned modalities and then
use the common representation to complete the representation of missing
modalities. Such as the kernel Shao et al. [2013] Liu et al. [2018] Liu et al.
[2020a] method, the matrix Hu and Chen [2019a] Rai et al. [2016] method and
the contrastive learning Lin et al. [2021] method. However, these methods
cannot explicitly compensate for the missing data in each modality. To solve
this issue, generative-based(GANTran et al. [2017]) methods (see Figure2(c)
and 2(d)) are proposed. As shown in the Figure2(c), the third imputation
strategy for missing values is based on Cycle GAN Zhu et al. [2017]. To tackle
a more complex problem when each instance is represented in only one modality,
for instance, TPIT-C Guo et al. [2022] integrates the adversarial learning
framework approach.

Although the aforementioned methods provide some schemes to address the IITC
problem, these methods still suffer from the following issues: 1) The majority
of previously incomplete multi-view clustering techniques extract distinct
view features from various visual datasets that are fundamentally homogenous
using various feature description techniques (e.g., SIFT, LBP, HOG Routray et
al. [2017]). Therefore, as shown in Figure 1, the direct use of these methods
on heterogeneous data (i.e. visual and textual data) may cause adverse effects
and even unsuccessful clustering tasks. This is since they neglect distinct
characteristics of visual and textual patterns. The reason for this since
images and text each have a different signal space. Language tasks have a
discrete signal space (words, sub-word units, etc.). In contrast, the original
signals of images are in a continuous, high-dimensional space and are not
constructed for human communication (e.g., unlike words). 2) The inaccurate
imputation or padding for missing data harms clustering performance. In
addition, the missing data processing tends to lean towards the generation
task rather than the clustering task. Therefore, the performance of missing
value completion and inversion is poor. 3) Existing methods ignore tapping
into the latent connections within and between modalities.

To address the above issues, we propose a novel model named CIGIT-C, which
first maps the original features of the image and text into their respective
subspaces using their specific encoders. The different modalities provide
cluster-level distinctive representations. This step takes into account the
intra-modal connections. Second, inspired by conditional GAN Mirza and
Osindero [2014] , as shown in Figure 2(d), the two generators use conditional
information (cluster label knowledge) provided by their respective subspaces
to standardize the generation process to generate representations, which is
beneficial to the clustering results. Clustered adversarial networks can not
only generate representations to compensate for missing data. Moreover, these
generated representations are beneficial for clustering tasks. The generators
thoroughly learn the knowledge of clustering-level across modalities,
exploiting and fully exploiting the complementarity between modalities. KL
clustering loss is used to update the encoder, which attempts to make the
distribution of the learned representations consistent and compact. The main
contributions of our approach are listed below:

(a) We have argued theoretically (in Section 1) and experimentally (in Section
4) that existing incomplete multi-view methods do not solve the IITC task
well. Such a theoretical view is remarkably different from existing works,
which treat image-text as homogeneous data.

(b) For the processing of missing data, we are driven by clustering tasks
rather than generating tasks, which are ignored in the existing IITC methods.

(c) We fully explore the potential relationships within and between
modalities. Extensive experiments demonstrate that our method achieves
superior clustering performance compared to state-of-the-art methods.

##  2 RELATED WORK

In this section, we briefly review recent advances in two related topics,
namely incomplete multi-view clustering and knowledge distillation.

###  2.1 Imcomplete Multi-view Clustering

Existing works can be roughly divided into three categories based on the
imputation strategy.

a) In the pioneering works, imcomplete multi-view clustering (IMVC) uses zero
or mean Wen et al. [2021a] values to fill in the missing values. However,
these simple padded data are very different from the real data, which avoids
learning a consistent clustering structure and severely degrades the final
clustering performance.

b) The second padding strategy for missing values is establishing a common
latent subspace using the complete data and then padding the missing data with
the learned common representation, i.e., matrix factorization (MF) based IMVC,
kernel-based methods, contrastive learning based IMVC. The MF-based approach
focuses on obtaining a consensus representation shared by all views from the
original feature space. For example, DAIMC Hu and Chen [2019b] establishes a
consensus basis matrix with the help of ${\ell_{2,1^{-}}}norm$ . IMG Zhao et
al. [2016] employs the graph laplacian to regularize the latent subspaces of
each data view. Hu et al. Hu and Chen [2019a] propose an efficient and
effective method for handling large-scale incomplete multi-view clustering
problems using regularized matrix decomposition and weighted matrix
factorization, taking full account of the missing information of the
instances. The kernel-based method of main idea are to leverage the kernel
matrices of the complete views for completing the kernel matrix of incomplete
view. The first partial multi-modal clustering algorithm is proposed by Rai et
al. Rai et al. [2010]. To fill the missing modality kernel matrix, they employ
a modality kernel representation and the Laplace regularisation approach. Liu
et al. Liu et al. [2020b] propose a joint optimization of kernel imputation
and clustering that seamlessly integrates these two learning processes to
achieve better clustering. EE-IMVC Liu et al. [2021] incorporates prior
knowledge to regularize the learning of the consensus clustering matrix. The
motivation for contrastive learning Hadsell et al. [2006] is to maximize the
similarity of positive pairs and distance of negative pairs. Motivated from
the information theory, Lin et al. Lin et al. [2021] propose a unified
framework to jointly learn consistent representation and recover the missing
view by maximizing the mutual information while minimizing the conditional
entropy of multiple views. Additionally, it shows promising results when
applied to complete multi-view data. However, this imputation strategy focuses
on the available information of present views. PMVC Li et al. [2014], IMGZhao
et al. [2016], MIC Shao et al. [2015], and DAIMC are all developed as joint
partial alignment or weighted matrix decomposition models to investigate data
on the alignment complementarity of current viewpoints and reach consensus.
They generally use some regularization or add some restrictions to the new
representation. The second imputation strategy focuses on available
information about present views without attempting to generate missing
information related to the missing views. However, it is challenging to
discover correlations across all the various modalities when there is a high
percentage of missing data.

c) The third imputation strategy is to use the generative method GAN to fill
in the missing views. The GAN model is intended to discover the connection
between various modalities. The model from references Wang et al. [2018] was
used to complete the missing data. Cycle GAN Zhu et al. [2017] uses generative
methods and their inverse directions to achieve unpaired image style
transformation. Inspired by this, Guo et al. Guo et al. [2022] suggest an
efficient approach for extracting the global representation of visual
instances. Furthermore, this method integrates the adversarial learning
framework methodology to handle a more difficult job in which each instance is
represented in just one modality. In addition to the above three imputation
strategies, some graph-based learning methods Wen et al. [2021b]Xue et al.
[2021] are proposed to handle the arbitrary IMC cases. These work fill in
missing values by extracting currently available multi-view information, i.e.,
data structures. CDIMC-net Wen et al. [2020] incorporates the view-specific
encoders and the graph embedding strategy to handle the incomplete multi-view
data. However, obtaining the complete data structure is difficult when some
views are missing.

We have borrowed some ideas from work above, namely the use of generative
models, and the idea is natural to use GAN to fill in the missing values. The
differences between this study and existing works are given below.

1) CIGIT-C uses clustering labels as a supervision to guide the generation
process, allowing the generation process to be regularized. CIGIT-C can
guarantee that the generative representation is suitable for the clustering
task.

2) Instead of using the raw feature space, we apply the generative technique
to the latent subspace. This promises to investigate the data’s structure and
produce more distinctive feature representations.

###  2.2 Knowledge Distillation

Knowledge distillation Buciluǎ et al. [2006] is a compression technique in
which a compact model (student) is trained to reproduce the behaviour of a
larger model (teacher) or set of models.

Distilling models Zhai et al. [2020] Zhang et al. [2018] convey extra
information beyond the traditional supervised learning target by mimicking the
teacher’s class probability or feature representation. Recently, in order to
address the inconsistency between training and inference caused by the
introduction of dropout, Liang et al. liang et al. [2021] introduce a simple
consistent training strategy that normalizes dropout. As some modalities in
the IITC task might be inherent in low quality or inconsistent cross-modality
clustering prediction, they will negatively affect the fusion process, in
turn, the clustering result. Motivated by these, we design a loss function
that minimizes KL-divergence between the clustering prediction scores of the
respective modalities and fused representations clustering probability.

Figure 3:  The framework of our proposed Clustering-Induced Generative Incomplete Image-Text Clustering(CIGIT-C). The model contains two core components, namely modality-specific encoders and subspace conditional clustering GAN, as shown in Figure with the red and black dashed boxes. The role of the modality-specific encoders is to map the original features into the subspace with the aim of seeking distinctive representations in the subspace. The subspace conditional clustering GAN function is to increase the cross-modal representation diversity and improve the model robustness. These generated representations are beneficial for clustering tasks. In addition, we introduce KL-divergence loss, the purple dashed box in the figure, making the latent distributions for all the modalities consistent and compact cluster-wise.  Table 1:  Notations and descriptions Notations | Descriptions
---|---
${{\mathbf{X}}^{(img)}}$, ${{\mathbf{X}}^{(txt)}}$ | Image/Text original features.
${\widetilde{\mathbf{X}}^{(img)}}$, ${\widetilde{\mathbf{X}}^{(txt)}}$ | Text/Image features of data missing modalities.
${\mathbf{Z}^{(img)}}$,${\mathbf{Z}^{(txt)}}$ | Subspace representation of images/text.
${\mathbf{E}^{(img)}}$,${\mathbf{E}^{(txt)}}$ | Encoders of image/text modalities.
${\mathbf{Z}_{\text{fake }}^{(img)}}$,${\mathbf{Z}_{\text{fake }}^{(txt)}}$ | Generated fake images/text representations.
${\mathbf{Z}^{\text{(fusion)}}}$ | Fusion representations
Figure 4: Notions for incomplete image-text data. The green triangle and the
orange circle represent the image and text modalities, respectively. The red
dashed boxes ${\mathbf{X}_{n}^{(\text{img) }}}$ and
${\mathbf{X}_{n}^{(\text{txt) }}}$ indicate the same instance, respectively,
and the samples’ modalities were accessible and strictly aligned.
${\widetilde{\mathbf{X}}^{(img)}}$ and ${\widetilde{\mathbf{X}}^{(txt)}}$
denote the image modality only, and the text modality only, respectively.
${n}$ denotes the number of data points of the whole dataset. ${m}$ denotes
the number of examples presented in all modalities.

##  3 CIGIT-C: THE PROPOSED MODEL

In this section, we first introduce the notations and descriptions. After
clarifying this, we present an overview of the CIGIT-C model. Since CIGIT-C
consists of three components, we describe each component and the loss function
in detail. Then, we provide a method to optimize the CIGIT-C model. Finally,
we integrate the above to provide a discussion of the CIGIT-C model.

###  3.1 Notations and Descriptions

In the legacy ITC task, each sample contains features in all the modalities.
This condition, however, cannot always be held in the incomplete ITC task.
Given a dataset
${\mathcal{D}=\left\\{{\mathbf{X}}^{(img,txt)},\tilde{\mathbf{X}}^{(img)},\tilde{\mathbf{X}}^{(txt)}\right\\}}$
of n instances, where ${{\mathbf{X}}^{(img,txt)}}$,
${\tilde{\mathbf{X}}^{(img)}}$, ${\tilde{\mathbf{X}}^{(txt)}}$ denote the
examples presented in both modalities, the visual modality only, and the text
modality only, respectively. For clarity, we denote variables originated from
images and texts by superscript img and text, respectively. The frequently
used notations and their descriptions are listed in Table 1. Because complete
modal data are few and expensive, it is advantageous to fully use the
substantial amount of incomplete-modal data.

###  3.2 Model Overview

The original features of the image and text are first mapped to a more
distinctive subspace through encoders ${\mathbf{E}^{(img)}}$ and
${\mathbf{E}^{(txt)}}$, respectively. More distinctive representations were
found by subspace clustering. The two generators generate a clustering task-
driven representation based on the clustering information provided by the
other subspace. The generative mechanism fully uses the characteristics and
cluster distribution of ${\mathbf{Z}^{(img)}}$ and ${\mathbf{Z}^{(txt)}}$. The
generated representation is fused with the original representation for
clustering. Furthermore, at the same time, aligned clustering loss is designed
to obtain a better modality-specific encoder.

###  3.3 Modality-Specific Encoders

The role of the modality-specific encoders ${\operatorname{E}^{(img)}}$ and
${\operatorname{E}^{(txt)}}$ is to map the original features into a more
distinctive subspace. The reason for this is twofold: 1) Given that the
frequencies, formats and receptive domains of images and text are very
different. Furthermore, since there may be higher-order noise in the original
features, directly applying GAN to the original features would hurt the
clustering predictions and make GAN harder to train and converge. 2) Different
modalities can provide uniqueness at different clustering levels, and
exploring the correlation between different categories and modalities is
critical to improving the clustering performance further. The clustering label
information obtained from the subspace is also used to standardise the
generation of Subspace Conditional Clustering GAN to benefit the features of
the clustering task.

$Z_{n}^{(fus)}=\sum_{t=1}^{n}(1-\beta)E^{(img)}(\mathbf{X}_{t}^{img})+\beta
E^{(txt)}(\mathbf{X}_{t}^{txt})$ (1)

${\operatorname{E}^{(img)}\left(\boldsymbol{x}^{img};\theta^{img}\right)}$ and
${\operatorname{E}^{(txt)}\left(\boldsymbol{x}^{txt};\theta^{txt}\right)}$ in
Eq 1 represent the encoders for images and text, respectively, where
${\theta}$ is the parameter that can be learned by back-propagation.
${\boldsymbol{z}_{i}^{img}=E^{img}\left(\boldsymbol{x}_{i}^{img}\right)}$ and
${\boldsymbol{z}_{i}^{txt}=E^{txt}\left(\boldsymbol{x}_{i}^{txt}\right)}$
represent the subspace features, while ${\mathbf{Z}^{(fus)}}$ is the subspace
features of the fused image and text, where ${\beta}$ is a weighting factor to
balance the ratio of text and visual modality. Moreover, in order to make the
projected samples more distinctive across modalities, thus, the Student’s
t-distribution is utilized, where the goal of Student’s t-distribution is to
make the projected representations closer to the samples of the same the
clustering centre than it is to any other cluster centre. As follow Eq 2,
${\delta}$ represents the degrees of freedom of the Student’s t-distribution.
Following Xie et al. Xie et al. [2016], we set ${\delta}$ to 1. The clustering
centroid of the ${K-th}$ cluster in the subspace for the ${m-th}$ modality is
denoted by ${\mu_{k}^{(m)}}$. The centroids are initialized by ${K}$-means and
updated by Stochastic Gradient Descent (SGD). ${\mathbf{z}_{n}^{(m)}}$
represents learning the subspace representation learned through the encoder,
where m=1 represents the visual modality ${\mathbf{Z}^{(img)}}$, m=2,
represents the text modality ${\mathbf{Z}^{(txt)}}$, and m=3 represents the
fusion modality ${\mathbf{Z}^{(fus)}}$. ${q_{nk}^{(m)}}$ represents the
probability of assigning instance n to group K of the ${m-th}$ modality (i.e.,
soft assignment).

$q_{nk}^{(m)}=\frac{\left(1+\left\|\mathbf{z}_{n}^{(m)}-\mu_{k}^{(m)}\right\|_{2}^{2}/\delta\right)^{-\frac{\delta+1}{2}}}{\sum_{k^{\prime}}\left(1+\left\|\mathbf{z}_{n}^{(m)}-\mu_{k^{\prime}}^{(m)}\right\|_{2}^{2}/\delta\right)^{-\frac{\delta+1}{2}}}$
(2)

To enhance cluster compactness, we prioritise data points assigned with high
confidence by calculating the emphasised target distribution ${p_{nk}}$ as
follows:

$p_{nk}=\frac{q_{nk}^{2}/f_{k}}{\sum_{k^{\prime}}q_{nk^{\prime}}^{2}/f_{k^{\prime}}}$
(3)

where ${f_{k}=\sum_{n}q_{nk}}$ denotes soft cluster frequencies. Before
normalising it by frequency per cluster, it first squares ${q_{nk}}$. Thus,
emphasis will be placed on high probability. Thus the modality-specific
encoder is updated by the following loss ${\mathcal{L}_{E_{m}}}$, as follows:

$\displaystyle\mathcal{L}_{E_{m}}$
$\displaystyle=KL\left(P^{(m)}\|Q^{(m)}\right)+\alpha
KL\left(P^{(3)}\|Q^{(3)}\right)$ (4)
$\displaystyle=\sum_{n}\sum_{j}p_{nk}^{(m)}\log\frac{p_{nk}^{(m)}}{q_{nk}^{(m)}}+\alpha\sum_{n}\sum_{j}p_{nk}^{(3)}\log\frac{p_{nk}^{(3)}}{q_{nk}^{(3)}},$

where m=1 and m=2 correspond to the losses of encoders
${\operatorname{E}^{(img)}}$ and ${\operatorname{E}^{(txt)}}$, respectively,
and ${\alpha}$ refers to a trade-off parameter. The objective
${\mathcal{L}_{E_{m}}}$ is to match the soft assignment ${p_{nk}}$ with the
desired distribution ${q_{nk}}$. Loss ${\mathcal{L}_{E_{m}}}$ is designed to
resolve inconsistencies in cluster predictions for different specific
modalities. Loss ${\mathcal{L}_{E_{m}}}$ forces the distribution of fused
features to be consistent with that of different modalities. Specifically,
minimize the KL-divergence of fusing feature with particular modality image
and text.

Figure 5: ${p_{nk}^{(1)}}$ and ${p_{nk}^{(2)}}$ represent the logits scores of
the respective modalities, while ${p_{nk}^{(3)}}$ represents the fused logits
scores. We use KL-divergence to add consistency constraints, i.e. different
modalities and fused modalities predict the clustering knowledge consistently.
This learns a common semantic representation of the visual and text.

###  3.4 Subspace Conditional Clustering GAN

Subspace conditional clustering GAN module contains two generators, ${G_{12}}$
and ${G_{21}}$, and their corresponding discriminators, ${D_{1}}$ and
${D_{2}}$, which are trained in the inverse direction. The generator is
trained to generate samples while the discriminator attempts to differentiate
the samples. Both networks are compelled to improve their capabilities through
a competition strategy. Since ${G_{12}}$, ${D_{1}}$, and ${G_{21}}$,
${D_{2}}$, have symmetrical positions and identical objective equations. So
here, we will only discuss G12 and D1.

$\displaystyle L_{G_{12}d}$ $\displaystyle=-E_{z\sim
p_{z}(z)}\log\Big{(}1-D_{1}\Big{(}G_{12}\Big{(}z\mid
E^{(img)}\Big{(}X_{t}^{(img)}\Big{)}$ (5)
$\displaystyle\Big{)}\Big{)}\Big{)},$

where ${z}$ refers to noise, subspace conditional clustering GAN differs from
traditional GAN in focusing on clustering rather than generation tasks. ${z}$
is an a prior sampling method consisting of a cascade of normal random
variables and a one-hot label noise. Since the subspaces
${\mathbf{Z}^{(img)}}$ and ${\mathbf{Z}^{(txt)}}$ are modified when the
encoders ${\mathbf{E}^{(img)}}$ and ${\mathbf{E}^{(txt)}}$ are tuned, it is
challenging to achieve stable generative outputs directly. Therefore, we
introduce a similarity constraint that pulls produced samples and real samples
toward subspace similarity. The objective phrase is depicted below:

$\displaystyle L_{G_{12}s}$ $\displaystyle=E_{z\sim
p_{z}(z)}\Big{(}\Big{\|}G_{12}\left(z\mid
E^{(img)}\left(X^{img}\right)\right)$ (6)
$\displaystyle-E^{(txt)}\left(X^{txt}\right)\Big{\|}_{\mathrm{F}}^{2}\Big{)}$

The final overall objective ${L_{G_{12}}}$ is obtained by combining Eq.5 and 6
, as shown in Eq.7 below, where ${\mu}$ is used to balance discriminator loss
and similarity loss.

$L_{G_{12}}=L_{G_{12}d}+\mu L_{G_{12}s}$ (7)

The function of ${D_{1}}$ is to distinguish the generated samples as much as
possible from the real samples in subspace ${\mathbf{Z}^{(txt)}}$.

$\displaystyle L_{D_{1}}$ $\displaystyle=E_{X\sim p_{X}(X)}\log
D_{1}\left(E^{(txt)}\left(X^{txt}\right)\right)$ (8) $\displaystyle+E_{z\sim
p_{z}(z)}\log\left(1-D_{1}\left(G_{12}\left(z\mid
E^{(img)}\left(X^{img}\right)\right)\right)\right)$

After the representation is generated in the subspace of modality-specific,
the false representation and the true representation are fused, and Eq.1 is
updated as:

$\displaystyle Z_{n}^{(fus)}=$
$\displaystyle\sum_{t=1}^{n}(1-\beta)E^{(img)}(\mathbf{X}_{t}^{img})$ (9)
$\displaystyle+\beta
E^{(txt)}(\mathbf{X}_{t}^{txt})+\sum_{m=1}^{2}\eta_{m}Z_{\text{fake }}^{(m)},$

where ${\eta_{m}}$ is a trade-off parameter that balances the weight between
the real representation and the generated representation, m=1 represents the
visual modality, while m=2 represents the text modality.

Finally, we define the overall loss function of CIGIT-C as:

$\mathcal{L}_{\text{total}}=\min_{E_{m},G_{12},G_{21}}\max_{D_{1},D_{2}}\mathcal{L}_{E_{m}}+\mathcal{L}_{G_{12}}+\mathcal{L}_{G_{21}}+\mathcal{L}_{D_{1}}+\mathcal{L}_{D_{2}}.$
(10)

For a clear understanding, we summarize the optimization steps of the proposed
method CIGIT-C in Algorithm 1.

###  3.5 Model Analysis and Learning

Our model, CIGIT-C, differs from other approaches in three main ways: 1) The
generation mechanism focuses on clustering tasks rather than generating tasks,
ensuring that the generated representation is beneficial to the task. 2) The
CIGIT-C model taps into latent relationships within and between modalities, on
the one hand, mapping raw features to more distinctive subspaces aiming to
allow different modalities to provide cluster-level unique representations.
This aspect considers intra-modal connections. On the other hand, adversarial
learning is encapsulated in the CIGIT-C model to explore complementary
information across modalities. This aspect takes into account the inter-modal
connections. In addition, subspace conditional clustering GAN is also used as
regularization to enhance the robustness and accuracy of the model. 3) We
consider and resolve inconsistent specific-modality clustering predictions. We
introduce the effect of knowledge distillation loss, on the one hand, to make
the posterior probability of the fused feature consistent with the posterior
probability of each modality subspace, and on the other hand, to enable the
encoder to better extract the semantic knowledge within the subspace and use
it in the representation of the embedded subspace.

Algorithm 1 Clustering-Induced Generative Incomplete Image-Text Clustering
(CIGIT-C) Algorithm.

0:
${\mathcal{D}=\left\\{{\mathbf{X}}^{(img,txt)},\tilde{\mathbf{X}}^{(img)},\tilde{\mathbf{X}}^{(txt)}\right\\}}$,
where some samples either lack image or text modality; learing rate $\omega$;
hyper-parameter $\beta,\alpha,\eta_{1},\eta_{2}$; iteration number ${I}$;
cluster number ${K}$.

0: The optimal parameters ${\Theta_{G_{12}},\Theta_{G_{21}},\Theta_{D_{1}},}$
and ${\Theta_{D_{2}}}$; clustering results $\mathbf{R}$;

1: Map original features
${\left\\{{\mathbf{X}}^{(img,txt)},\tilde{\mathbf{X}}^{(img)}\right\\}}$ to
more distinctive subspaces
${\left\\{{\mathbf{Z}^{(img)},\mathbf{Z}^{(txt)}}\right\\}}$. Initialize the
parameters of CIGIT-C with Xavier initializer. Calculate fusion features
${\mathbf{Z}^{\text{(fusion)}}}$ and clustering centres ${\mu_{k}^{(m)}}$.
${m\in\\{1=img,2=txt,3=fus\\}}$.

2: while ${t\leq\text{ MaxIter }}$ do

3: Train the encoders with ${\mathcal{L}_{E_{m}}}$ according to Eq.4.
${\forall m=1,2}$

4: Train generators ${G_{12}}$ and ${G_{21}}$ using loss functions
${L_{G_{12}}}$ and ${L_{G_{21}}}$ according to Eq. 7.

5: Train discriminators ${D_{1}}$ and ${D_{2}}$ using loss functions
${L_{D_{1}}}$ and ${L_{D_{2}}}$ according to Eq. 8.

6: The fusion features are updated according to Eq.9, and ${\mu_{k}^{(m)}}$ is
recalculated. ${m\in\\{1=img,2=txt,3=fus\\}}$.

7: end while

8: The final updated fusion representation ${{\mathbf{Z}^{\text{(fusion)}}}}$
is obtained, and ${\mu_{k}^{(3)}}$ and ${q_{nk}^{(3)}}$ are computed in turn
by using Eq.2 and 3, respectively.

9: Obtain the clustering results $\mathbf{R}$;

10: return $\mathbf{R}$;

##  4 EXPERIMENTS

###  4.1 Experimental settings

All tests are carried out on the Ubuntu 16.04 platform, with Tesla P100
Graphics Processing Units and 32GB of RAM. The PyTorch 1.6.0 framework serves
as the foundation for our model, technique, and baseline.

####  4.1.1 Compared methods

We use state-of-the-art methods to demonstrate the effectiveness of our model
CIGIT-C. Comparison baselines are briefly introduced below. BestSM: We report
the best clustering results achieved by performing k-means on modality-
specific, i.e., visual features ${{\mathbf{X}}^{(img)}}$ and text features
${{\mathbf{X}}^{(txt)}}$. DAIMCHu and Chen [2018]: It is based on weighted
semi-nonnegative matrix factorization to obtain cluster results. UEAFWen et
al. [2019]: A locality-preserved reconstruction term is introduced to infer
the missing views so that all views can be aligned. OPIMCHu and Chen [2019a]:
It adopts regularized and weighted matrix factorization to obtain clustering
results. PICWang et al. [2019]: It learns a consensus Laplacian matrix from
incomplete multi-view data for clustering. CDIMC-netWen et al. [2020]: It
incorporates the view-specific encoders and the graph embedding strategy to
handle the incomplete multi-view data. CASENXue et al. [2021]: It is an end-
to-end trainable framework that jointly conducts multi-view structure
enhancing and data clustering. COMPLETERLin et al. [2021]: It is a two-stage
contrastive multi-view clustering method by learning consistent features.

####  4.1.2 Datasets and Features

Wikipedia Pereira et al. [2013] contains 2,866 image-text pairs, including ten
semantic categories, sourced from the Wikipedia collection of "featured
articles". The text in each pair is a paragraph describing the content of the
corresponding image. For the test set, we chose 30${\%}$ randomly from the
dataset. The result forms a training set of 1910 samples and a test set with
956 samples.  NUS-WIDE-10k consists of 10,000 image-text pairs, selected on
average from the ten largest semantic categories of the NUS-WIDE Chua et al.
[2009] dataset. The 10,000 selected image-text pairs were divided into two
parts: 8,000 image-text pairs for training and 2,000 image-text pairs for
testing.

BDGP Cai et al. [2012] contains 2500 instances about drosophila embryos of 5
categories.

For all subsequent comparisons, we use the same image and text features for a
fair comparison. For the BDGP dataset, the feature files are provided by the
authors. For the image samples of Wikipedia and NUS-WIDE-10K, we first resize
it into 224${\times}$224 then employ the VGGNet-19 Simonyan and Zisserman
[2014] model pre-trained on the ImageNet to output 4096-dimensional features
from its fc _7_ layer. For the text samples of Wikipedia, we adopt a pre-
trained BERT to extract 768-dimensional text features. Similar to the BERT
paper Devlin et al. [2018], we take the embedding associated with ${[CLS]}$ to
represent the whole sentence. For the text samples of NUS-WIDE-10K, we adopt
the widely-used bag-of-words(BoW) vector with the TF-IDF weighting approach to
get 1000-dimensional features. The detailed statistics of the three datasets
are summarized in Table 2.

####  4.1.3 Incomplete data construction

To demonstrate the effectiveness of the CIGIT-C model in processing missing
image-text data. We set different missing rates
${p\in\\{10\%,30\%,50\%,70\%\\}}$. The missing rate ${p}$ is defined as
${p=(n-m)/n}$, where m is the number of complete examples, and n is the number
of the whole dataset. We randomly remove a modality by the missing rate ${p}$
to select some instances as incomplete data. A high missing rate is positively
correlated with task complexity.

Table 2: The statistics of the datasets. Datasets | Classes | Modality | Samples | Feature
---|---|---|---|---
Wikipedia | 10 |  | Image
---
Text
| 1919/956
---
1919/956
| 4096D VGG
---
768d BERT
NUS-WIDE-10K | 10 |  | Image
---
Text
| 8000/2000
---
8000/2000
| 4096D VGG
---
1000d BoW
BDGP | 5 |  | Image
---
Text
| 2000/500
---
2000/500
| 1000D lateral
---
79d texture

####  4.1.4 Evaluation Metric

We choose two classical and widely used clustering metrics to measure the
consistency of cluster assignments and ground-truth memberships:

(1) Clustering accuracy (ACC) maps one-to-one the learned clusters to the
ground-truth classes by the Hungarian algorithm and measures the
classification accuracy;

(2) Normalised mutual information (NMI) quantifies the labelling consistency
by the normalised MI between the predicted and ground truth labels of all
samples.

For all the metrics, the value 1 means a perfect clustering.

####  4.1.5 Implementation details

Table 3: The architecture of the encoders in CIGIT-C Dataset |  ${\mathbf{E}^{(img)}}$ |  ${\mathbf{E}^{(txt)}}$
---|---|---
Wikipedia |  |  Input (size=(bathsize,4096))
---
Dense (LeakyReLU, size =2048)
Dense (LeakyReLU, size =1024)
Dense (LeakyReLU, size =256)
|  Input (size=(bathsize,768))
---
Dense (LeakyReLU, size =256)
Dense (LeakyReLU, size =256)
Dense (LeakyReLU,size =128)
NUS-WIDE-10K |  |  Input (size=(bathsize,4096))
---
Dense (LeakyReLU, size =2048)
Dense (LeakyReLU, size =1024)
Dense (LeakyReLU,size =256))
|  Input (size=(bathsize,1000))
---
Dense (LeakyReLU, size =256)
Dense (LeakyReLU,size =256))
Dense (LeakyReLU,size =128)
BDGP |  |  Input (size=(bathsize,1000))
---
Dense (LeakyReLU, size =256)
Dense (LeakyReLU, size =256)
Dense (LeakyReLU,size =128)
|  Input (size=(bathsize,79))
---
Dense (LeakyReLU, size =79)
Dense (LeakyReLU, size =79)
Dense (LeakyReLU,size =64)
Table 4: The architecture of the generators in CIGIT-C Dataset |  ${G_{12}}$ |  ${G_{21}}$
---|---|---
Wikipedia |  |  Input (batchsize,4096+noise-dim=4396)
---
Dense (BatchNorm1d,LeakyReLU, size =1024)
Dense (BatchNorm1d, LeakyReLU, size = 1024)
Dense (BatchNorm1d,LeakyReLU, size =768)
|  Input (batchsize,768+noise-dim=1,068)
---
Dense (BatchNorm1d,LeakyReLU, size =2048)
Dense (BatchNorm1d,LeakyReLU, size =2048)
Dense (BatchNorm1d,LeakyReLU, size =4096)
NUS-WIDE-10K |  |  Input (batchsize,4096+noise-dim=4396)
---
Dense (BatchNorm1d, LeakyReLU, size = 1024)
Dense (BatchNorm1d,LeakyReLU, size =1024)
Dense (BatchNorm1d,LeakyReLU, size =1000)
|  Input (batchsize,1000+noise-dim=1300)
---
Dense (BatchNorm1d,LeakyReLU, size =2048)
Dense (BatchNorm1d,LeakyReLU, size =2048)
Dense (BatchNorm1d,LeakyReLU, size =4096)
BDGP |  |  Input (batchsize,1000+noise-dim=1300)
---
Dense (BatchNorm1d,LeakyReLU, size =256)
Dense (BatchNorm1d,LeakyReLU, size =128)
Dense (BatchNorm1d,LeakyReLU, size =79)
|  Input (batchsize,79+noise-dim=379)
---
Dense (BatchNorm1d,LeakyReLU, size =512)
Dense (BatchNorm1d,LeakyReLU, size =512)
Dense (BatchNorm1d,LeakyReLU, size =1000)
Table 5: The architecture of the discriminators in CIGIT-C Dataset |  ${D_{1}}$ |  ${D_{2}}$
---|---|---
Wikipedia |  |  Input (batchsize,${\mathbf{Z}_{\text{fake }}^{(txt)}}$+${q_{nk}^{(2)}}$ =768+10=778)
---
Dense (LeakyReLU, size =512)
Mini-batch(LeakyReLU, size = 512)
Dense (sigmoid, size =1)
|  Input (batchsize, ${\mathbf{Z}_{\text{fake }}^{(img)}}$+${q_{nk}^{(1)}}$
=4096+10=4160))
---
Dense (LeakyReLU, size =1024)
Mini-batch(LeakyReLU, size =512)
Dense (sigmoid, size =1)
NUS-WIDE-10K |  |  Input (batchsize, ${\mathbf{Z}_{\text{fake }}^{(img)}}$+${q_{nk}^{(2)}}$ =768+10=778)
---
Dense (LeakyReLU, size = 512)
Mini-batch(LeakyReLU, size =512)
Dense (sigmoid,size =1)
|  Input (batchsize, ${\mathbf{Z}_{\text{fake }}^{(img)}}$+${q_{nk}^{(1)}}$ =
4096+10=4160)
---
Dense (LeakyReLU, size =1024)
Mini-batch (LeakyReLU, size =512)
Dense (sigmoid,size =1)
BDGP |  |  Input (batchsize,${\mathbf{Z}_{\text{fake }}^{(txt)}}$+${q_{nk}^{(2)}}$ =79+10=89))
---
Dense (LeakyReLU, size =64)
Mini-batch (LeakyReLU, size =64)
Dense (sigmoid, size =1)
|  Input (batchsize, ${\mathbf{Z}_{\text{fake }}^{(img)}}$+${q_{nk}^{(1)}}$
=1000+10=1010)
---
Dense (BatchNorm1d,LeakyReLU, size =512)
Mini-batch (BatchNorm1d,LeakyReLU, size =512)
Dense (sigmoid,size =1)
Table 6: Clustering results (ACC and NMI) of different methods on the Wikipedia dataset with different missing rate ${p\%}$. Method |  Accuracy |  NMI
---|---|---
10${\%}$ |  30${\%}$ |  50${\%}$ |  70${\%}$ |  10${\%}$ |  30${\%}$ |  50${\%}$ |  70${\%}$
BestSM |  44.79 |  40.70 |  33.45 |  23.32 |  48.55 |  39.51 |  31.80 |  22.31
DAIMCHu and Chen [2018] |  55.05 |  43.70 |  31.96 |  18.99 |  45.18 |  28.43 |  16.84 |  9.86
UEAFWen et al. [2019] |  53.03 |  43.53 |  33.79 |  25.12 |  49.83 |  38.12 |  26.64 |  18.59
OPIMCCHu and Chen [2019a] |  45.15 |  28.28 |  16.92 |  9.010 |  54.43 |  39.92 |  33.63 |  25.75
PICWang et al. [2019] |  45.60 |  40.66 |  30.81 |  26.74 |  34.42 |  28.84 |  15.02 |  11.37
CDIMC-netWen et al. [2020] |  31.28 |  28.45 |  27.43 |  23.18 |  36.88 |  33.24 |  28.75 |  21.16
CASENNXue et al. [2021] |  57.46 |  49.44 |  44.94 |  40.71 |  56.96 |  47.99 |  40.54 |  36.54
COMPLETERLin et al. [2021] |  34.47 |  31.79 |  30.35 |  31.30 |  26.48 |  20.93 |  17.54 |  17.17
CIGIT-C |  54.71 |  50.52 |  46.68 |  42.22 |  56.83 |  42.84 |  42.11 |  39.92
Table 7: Clustering results (ACC and NMI) of different methods on the NUS-WIDE-10K dataset with different missing rate ${p\%}$. Method |  Accuracy |  NMI
---|---|---
10${\%}$ |  30${\%}$ |  50${\%}$ |  70${\%}$ |  10${\%}$ |  30${\%}$ |  50${\%}$ |  70${\%}$
BestSM |  50.82 |  47.14 |  42.49 |  32.88 |  57.71 |  48.76 |  40.41 |  33.25
DAIMCHu and Chen [2018] |  61.84 |  51.06 |  39.04 |  26.49 |  51.75 |  35.25 |  24.21 |  17.12
UEAFWen et al. [2019] |  60.47 |  51.03 |  40.91 |  32.18 |  56.72 |  45.54 |  34.12 |  25.42
OPIMCCHu and Chen [2019a] |  51.90 |  35.91 |  23.67 |  15.95 |  55.60 |  45.72 |  40.76 |  33.34
PICWang et al. [2019] |  51.52 |  47.09 |  36.73 |  32.91 |  40.41 |  34.91 |  20.94 |  17.47
CDIMC-netWen et al. [2020] |  59.98 |  58.41 |  51.28 |  47.94 |  54.46 |  52.82 |  49.68 |  34.23
CASENNXue et al. [2021] |  62.39 |  56.34 |  50.74 |  46.51 |  62.63 |  52.81 |  47.97 |  31.24
COMPLETERLin et al. [2021] |  48.66 |  43.79 |  42.88 |  39.67 |  40.66 |  36.66 |  39.51 |  33.17
CIGIT-C |  63.26 |  57.44 |  52.48 |  48.39 |  63.56 |  54.93 |  49.65 |  38.73

The real number of clusters ${k}$ is set as the true number of classes for all
data sets under the assumption that it is known.

We initialise 20 times and choose the best answer for each approach using the
k-means algorithm to produce clustering assignments. In order to prevent
extreme occurrences, we run each method 10 times and present the average
results.

For DAIMC, we set the parameters of the method by following the setting of the
original paper.

For UEAF, we set parameters ${\gamma}$ to 3. The role of parameter ${\gamma}$
is to control the smoothness of the weight distribution.

For OPIMC, in hyperparameter search, we select the parameter ${\alpha}$ within
${\\{1e^{-4},1e^{-3},1e^{-2},1e^{-1},1e^{0},1e^{1},1e^{2},1e^{3}\\}}$ and
report the best results.

For PIC, we set parameters ${\tilde{\beta}}$ to 0.1.

For CDIMC-net and CASEN, we choose the recommended network structure according
to the original paper.

For CDIMC-net, the graph embedding hyper-parameter ${\alpha}$ is fixed to
${1e^{-5}}$, and the learning rate is fixed to ${1e^{-3}}$.

For CASEN, the important parameters are set as ${\eta_{1}=0.1}$,
${\eta_{2}=0.01}$, ${\lambda=0.5}$, ${\theta=0.1}$, ${r=2}$.

For COMPLETER, the softmax activation function is used at the last layer of
the encoders. The entropy parameter ${\alpha}$ is fixed to 9 and trade-off
hyper-parameters ${\lambda_{1}}$ and ${\lambda_{2}}$ are fixed to 0.1.

For CIGIT-C, the proposed method contains two training modules, i.e.,
modality-specific encoders and subspace conditional clustering GAN. Tables 3,
4 and 5 present the details of the network architectures in these two training
modules, respectively. Take the Wikipedia dataset, ${\mathbf{E}^{(img)}}$ and
${\mathbf{E}^{(txt)}}$ are three-layer networks in our implementation. We set
the dimension of the ${\mathbf{E}^{(img)}}$ to 4096${-}$1024${-}$256, where
4096 is the feature dimension of the visual modality, 1024 denotes the hidden
layer dimension, 256 denotes the subspace representation dimension. We set the
dimension of the ${\mathbf{E}^{(txt)}}$ to 728${-}$256${-}$128\. ${G_{12}}$
and ${G_{21}}$ are three-layer neural networks with a batch normalization
layer to normalize the input vector and stabilize the training procedure. The
proposed ${D_{1}}$ and ${D_{2}}$ are mainly made up of a fully connecteded
layer with ReLU Nair and Hinton [2010] activation, a mini-batch layer that can
increase the diversity of fake representations, a sigmoid function which
outputs the fake-real possibility of input representations.

Table 8: Clustering results (ACC and NMI) of different methods on the BDGP dataset with different missing rate ${p\%}$. Method |  Accuracy |  NMI
---|---|---
10${\%}$ |  30${\%}$ |  50${\%}$ |  70${\%}$ |  10${\%}$ |  30${\%}$ |  50${\%}$ |  70${\%}$
BestSM |  47.52 |  41.44 |  34.47 |  27.95 |  35.74 |  25.20 |  16.39 |  9.29
DAIMCHu and Chen [2018] |  74.76 |  62.88 |  52.45 |  35.82 |  55.64 |  47.87 |  28.33 |  9.17
UEAFWen et al. [2019] |  76.56 |  57.13 |  50.41 |  36.84 |  65.13 |  44.10 |  33.13 |  12.60
OPIMCCHu and Chen [2019a] |  71.69 |  56.44 |  55.17 |  35.82 |  61.77 |  41.47 |  25.94 |  8.35
PICWang et al. [2019] |  68.26 |  63.05 |  53.23 |  44.31 |  66.70 |  55.77 |  27.78 |  11.43
CDIMC-netWen et al. [2020] |  87.50 |  76.99 |  60.47 |  40.98 |  77.24 |  57.09 |  35.64 |  14.77
CASENNXue et al. [2021] |  79.92 |  70.96 |  59.25 |  46.93 |  72.21 |  48.23 |  29.77 |  12.45
COMPLETERLin et al. [2021] |  59.60 |  55.20 |  54.10 |  52.90 |  52.80 |  51.10 |  33.04 |  14.77
CIGIT-C |  78.12 |  72.14 |  62.74 |  51.82 |  68.41 |  51.76 |  36.11 |  16.78

###  4.2 Experimental Results

Tables 6 to 8 list the clustering performance of all the compared methods on
the three datasets, from which we can observe the following:

1) It is clear that when the missing rate changes from 0.1 to 0.7, the
clustering performance of all approaches is decreased.

2) Although CIGIT-C performed moderately compared to other baselines with a
low miss rate, it was not outstanding. Compared with the second-best methods,
CIGIT-C has considerable improvements on Wikipedia, NUS-WIDE-10K, and BDGP
dataset, when the missing rate is 50% or 70%. Specifically, for Wikipedia
datasets, CIGIT-C achieves a 0.78%, 1.74%, and 1.64% increment at a miss rate
of 30%, 50%, and 70%, with respect to accuracy against the second best. For
the indicator NMI, CGIG-C is higher than 1.57% and 3.38% compared to the
second best baseline at a miss rate of 50% and 70%. For missing rate
${p\%=70}$, compared to the second best method, CIGIT-C improves NMI by 4.5%,
2.01% on NUS-WIDE-10K and BDGP datasets, respectively, which demonstrates the
effectiveness of CIGIT-C.

3) For Wikipedia dataset, some multi-view clustering methods (i.e., CDIMC-net
and COMPLETER ) results even perform worse than single-view methods (i.e.,
BestSM) results.

4) For BDGP with low missing rates, the performance of certain methods (like
OPIMC and COMPLETER) is bad.

The reasons for the above observations can be explained as follows:

1) As the missing rate rises, complementary information between multi-
modalities becomes scarce, and all methods inevitably decline.

2) CIGIT-C methods are superior in most databases, especially when the partial
ratio is large. The reasons are 2.1) our model is generative and serves to
augment the data, and the augmented data is helpful for the clustering task.
2.2) the model exploits potential representations within and between
modalities. 2.3) our model not only generates data but also introduces a KL
loss function to achieve predictive consistency across heterogeneous
modalities.

3) For Wikipedia dataset, the performance of certain methods (like CDIMC-net
and COMPLETER) is bad. The possible reasons are as follows: 3.1) Both method
belong to the imcomplete multi-view method and do not consider the
heterogeneity between images and text. 3.2) COMPLETER aims to maximise the
mutual information between different views through contrastive learning to
obtain a common representation to fill in the missing data. In the case of
high missing rates, it is difficult to find connections between modalities.
3.3) The CDIMC-net padding strategy depends on the estimation of the data
distribution. As a result, the cumulative error increases when the missing
rate is high.

4) The similarity of the data is the primary variable influencing clustering
performance. The critical component is still the multi-modal learning of
features since similarity depends on characteristics. The low quality of the
modal features in the BGDP dataset has a direct impact on the performance of
the method.

5) DAIMC and OPIMC are based on matrix factorization. However, their
clustering performance is limited because they mainly make use of some
regularization and add some constraints to the new representation but fail to
compensate for the missing data in each view explicitly.

6) UEAF and PIC explore intra-modality connections and structure for multi-
view clustering yet neglect to explore the underlying relationships and
structure between modalities. Therefore valid clustering results can not be
obtained.

7) The direct and brutal fusion of heterogeneous data inevitably leads to
performance degradation. The introduction of KL-divergence loss in CIGIT-C
largely circumvents the inconsistency of predictions across modal clusters by
means of consistency constraints.

Figures 6 and 7 show the ACC and NMI metric values for all comparison methods
on the three datasets at missing rates from 0.1 to 0.9 with an interval of
0.1, respectively.

Through these results, we can draw the following conclusions: 1) CIGIT-C
methods are superior in most databases, especially when the missing ratio is
large, i.e. missing rates ${p\%}$ from 50 to 90. For example, as shown in
figure 6(b), at a missing rate of 90, CGIT-C surpasses the optimal baseline by
5.57%. 2) With increasing the missing rate, the performance degradations of
the compared methods are much larger than that of ours. For example, when the
missing rate is 50%, as shown in Figure 6(a) , CASEN and CGIG-C reach 44.94%
and 46.68%, respectively. However, as the missing rate rise, CGIG-C is
superior to CASEN.

In addition, as shown in Figure 6(a), an interesting phenomenon is that the
results of some incomplete multi-view (UEAF, COMPLETER,DAIMC,CDIMC-net)
methods are even worse than those of single-view(BestSM) methods, which also
proves from the experimental results that the existing incomplete multi-view
methods cannot effectively solve IITC tasks.

Figure 6: The average clustering accuracy performance with respect to
different missing rate on the (a) Wikipedia dataset, (b) NUS-WIDE-10K dataset
and (c) BDGP dataset.

###  4.3 Parameter Analysis and Ablation Study

This section examines the CIGIT-C on the Wikipedia, NUS-WIDE-10K, and BDGP
datasets from the perspective of parameter analysis and ablation studies. Our
method contains four user-defined parameters, weighting factor ${\beta}$ to
balance the ratio of image modality to text modality and weighting factor
${\alpha}$ to control KL loss. ${\eta_{m}}$ is the generated and true
representation weighting coefficient for the m-th order modality. ${\eta_{1}}$
is for image modality, and ${\eta_{2}}$ is for text modality.

As shown in Figure 7(a), when the missing rate is 0.5, parameter ${\beta}$ has
the best effect when it is 0.6, 0.4 and 0.4 on the three datasets of
Wikipedia, NUS-WIDE-10K and BDGP, respectively. The reason for the different
values of parameter ${\beta}$ may be that, in contrast to the Wikipedia
dataset, the text features extracted from the tags are very simple. Therefore,
the semantic information provided by text modalities in the other two datasets
is weaker than that provided by text modalities in the Wikipedia dataset. The
parameter ${\alpha}$ is tuned from $\\{{0.01,0.1,1,10,100\\}}$ and the ACC
performance is shown in Figure 7(b).

The ${\eta_{1}}$ and ${\eta_{2}}$ parameters are tuned in the same range as
${\alpha}$. The best results are achieved on the Wikipedia dataset for a
missing rate of 0.5, with both taking the value of 0.1. The parametric
analysis of CIGIT-C on the NUS-WIDE-10K and BDGP datasets is analysed using
the same methods. Due to space constraints, we will not continue our
conversation and show here.

To demonstrate the importance of the Clustering GAN module and KL loss in
CIGIT-C, we performed ablation experiments with a miss rate of 0.5. As shown
in Figure 9, where"None Clustering GAN" means that the conditional clustering
GAN does not participate in the CIGIT-C work, similarly, "None KL-Divergence
losses" means that CGIG-C contains only the clustered GAN.

We can draw the following conclusions:

1) It can be seen that when both clustering GAN and KL-loss exist, CIGIT-C
performs the best, indicating that modules and loss functions in CIGIT-C
complement each other and promote each other.

2) The CIGIT-C model does not improve much when only Kl-loss is available.
Even with the consistency constraint, i.e. the different and fused modalities
predict the clustering knowledge consistently. However, the performance of
CIGIT-C inevitably decreases because no treatment is given to the missing
values.

3) When only clustering GAN is available, although clustering GAN serves to
increase the diversity of cross-modal representations and solve the problem of
missing modalities, it lacks consideration of the inconsistencies arising from
inter-modal clustering predictions.

Figure 7: The average clustering NMI performance with respect to different
missing rate on the (a) Wikipedia dataset, (b) NUS-WIDE-10K dataset and (c)
BDGP dataset . Figure 8: Accuarcy (%) performance with different ${\beta}$ and
${\alpha}$ when the missing rate ${p}$ is 0.5. Figure 9:  Effectiveness of
clustering GAN, KL-divergence losses, when the missing rate ${p}$ is 0.5.

Figure 10: Parameter analysis on Wikipedia dataset.

###  4.4 Qualitative Analysis

In order to verify the effectiveness of clustering GAN in CIGIT-C, we
visualize the distribution of real and generated representations of samples in
${\mathbf{Z}^{(img)}}$ and ${\mathbf{Z}^{(txt)}}$ by t-SNE Van der Maaten and
Hinton [2008]. The results are shown in Figure 11, which illustrates that the
true and generated representations belonging to the same cluster category are
close to each other and vice versa. The visualisation in Figure 12
demonstrates the advantages of our proposed CIGIT-C in terms of clustering
results. In contrast to CGIG-C, COMPLETERLin et al. [2021] incorrectly
clusters text with the semantic category art and media images together. CDIMC-
netWen et al. [2020] incorrectly clusters text with the semantic category
music and sports images together.

(a) Real and Fake Samples in ${\mathbf{Z}^{(img)}}$

(b) Real and Fake Samples in ${\mathbf{Z}^{(txt)}}$

Figure 11:  t-SNE Van der Maaten and Hinton [2008] visualization results of
the real and the generated test sample representations in
${\mathbf{Z}^{(img)}}$ and ${\mathbf{Z}^{(txt)}}$ respectively.

##  5 Conclusion

We demonstrate theoretically and experimentally that the existing incomplete
multi-view clustering method can not solve the IITC task well, which is not
considered in the existing works. We propose a clustering Gan to ensure that
the populated data benefits the clustering task. We also add consistency
constraints by introducing the KL divergence loss function to reduce the
inconsistency caused by different modality predictions.

Figure 12: The SOTA method and the CIGIG-C method visualise the clustering
results. The bottom white clustering result in the figure is correct, with
both images and text belonging to the same semantic category. The categories
are art and music from top to bottom, while the yellow box shows the wrong
clustering results.

## Acknowledgments

This work was supported in part by the Major Program of the Inner Mongolia
Natural Science Foundation under Grant 2021ZD0015, in part by the Natural
Science Foundation of China under Grant 52069018, in part by Natural Science
Foundation of Inner Mongolia under Grant 2020MS03009, in part by Higher
Education Scientific Research Program of Inner Mongolia under Grant NJZY22370,
in part by Science and Technology Program of Inner Mongolia under Grant
2020GG0078.

## References

  * Gomaa et al. [2020] Ahmed Gomaa, Moataz M Abdelwahab, and Mohammed Abo-Zahhad.  Efficient vehicle detection and tracking strategy in aerial videos by employing morphological operations and feature points motion analysis.  _Multimedia Tools and Applications_ , 79(35):26023–26043, 2020.
  * Gomaa et al. [2018] Ahmed Gomaa, Moataz M Abdelwahab, and Mohammed Abo-Zahhad.  Real-time algorithm for simultaneous vehicle detection and tracking in aerial view videos.  In _2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS)_ , pages 222–225. IEEE, 2018.
  * Gomaa et al. [2019] Ahmed Gomaa, Moataz M Abdelwahab, Mohammed Abo-Zahhad, Tsubasa Minematsu, and Rin-ichiro Taniguchi.  Robust vehicle detection and counting algorithm employing a convolution neural network and optical flow.  _Sensors_ , 19(20):4588, 2019.
  * Horne et al. [2020] Elsie Horne, Holly Tibble, Aziz Sheikh, Athanasios Tsanas, et al.  Challenges of clustering multimodal clinical data: review of applications in asthma subtyping.  _JMIR medical informatics_ , 8(5):e16452, 2020\.
  * Li et al. [2021] Bing Li, Jiahua Li, and Xijun Ou.  Hybrid recommendation algorithm of cross-border e-commerce items based on artificial intelligence and multiview collaborative fusion.  _Neural Computing and Applications_ , pages 1–10, 2021.
  * Do et al. [2020] Truong Dong Do, Kyungbaek Kim, Hyukro Park, and Hyung-Jeong Yang.  Image and encoded text fusion for deep multi-modal clustering.  In _The 9th International Conference on Smart Media and Applications_ , pages 308–312, 2020.
  * Wen et al. [2021a] Jie Wen, Zheng Zhang, Zhao Zhang, Lei Zhu, Lunke Fei, Bob Zhang, and Yong Xu.  Unified tensor framework for incomplete multi-view clustering and missing-view inferring.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 10273–10281, 2021a.
  * Shao et al. [2013] Weixiang Shao, Xiaoxiao Shi, and S Yu Philip.  Clustering on multiple incomplete datasets via collective kernel learning.  In _2013 IEEE 13th International Conference on Data Mining_ , pages 1181–1186. IEEE, 2013.
  * Liu et al. [2018] Xinwang Liu, Xinzhong Zhu, Miaomiao Li, Lei Wang, Chang Tang, Jianping Yin, Dinggang Shen, Huaimin Wang, and Wen Gao.  Late fusion incomplete multi-view clustering.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(10):2410–2423, 2018.
  * Liu et al. [2020a] Xinwang Liu, Miaomiao Li, Chang Tang, Jingyuan Xia, Jian Xiong, Li Liu, Marius Kloft, and En Zhu.  Efficient and effective regularized incomplete multi-view clustering.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(8):2634–2646, 2020a.
  * Hu and Chen [2019a] Menglei Hu and Songcan Chen.  One-pass incomplete multi-view clustering.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 3838–3845, 2019a.
  * Rai et al. [2016] Nishant Rai, Sumit Negi, Santanu Chaudhury, and Om Deshmukh.  Partial multi-view clustering using graph regularized nmf.  In _2016 23rd International Conference on Pattern Recognition (ICPR)_ , pages 2192–2197. IEEE, 2016.
  * Lin et al. [2021] Yijie Lin, Yuanbiao Gou, Zitao Liu, Boyun Li, Jiancheng Lv, and Xi Peng.  Completer: Incomplete multi-view clustering via contrastive prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11174–11183, 2021.
  * Tran et al. [2017] Luan Tran, Xiaoming Liu, Jiayu Zhou, and Rong Jin.  Missing modalities imputation via cascaded residual autoencoder.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1405–1414, 2017.
  * Zhu et al. [2017] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2223–2232, 2017.
  * Guo et al. [2022] Dongjin Guo, Xiaoming Su, Yahong Lian, Limin Liu, and Haibo Wang.  Two-stage partial image-text clustering (tpit-c).  _IET Computer Vision_ , 2022.
  * Routray et al. [2017] Sidheswar Routray, Arun Kumar Ray, and Chandrabhanu Mishra.  Analysis of various image feature extraction methods against noisy image: Sift, surf and hog.  In _2017 Second International Conference on Electrical, Computer and Communication Technologies (ICECCT)_ , pages 1–5. IEEE, 2017.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Hu and Chen [2019b] Menglei Hu and Songcan Chen.  Doubly aligned incomplete multi-view clustering.  _arXiv preprint arXiv:1903.02785_ , 2019b.
  * Zhao et al. [2016] Handong Zhao, Hongfu Liu, and Yun Fu.  Incomplete multi-modal visual data grouping.  In _IJCAI_ , pages 2392–2398, 2016.
  * Rai et al. [2010] Piyush Rai, Anusua Trivedi, Hal Daumé III, and Scott L DuVall.  Multiview clustering with incomplete views.  In _Proceedings of the NIPS Workshop on Machine Learning for Social Computing_. Citeseer, 2010.
  * Liu et al. [2020b] Xinwang Liu, Xinzhong Zhu, Miaomiao Li, Lei Wang, En Zhu, Tongliang Liu, Marius Kloft, Dinggang Shen, Jianping Yin, and Wen Gao.  Multiple kernel $k$k-means with incomplete kernels.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(5):1191–1204, 2020b.  10.1109/TPAMI.2019.2892416.
  * Liu et al. [2021] Xinwang Liu, Miaomiao Li, Chang Tang, Jingyuan Xia, Jian Xiong, Li Liu, Marius Kloft, and En Zhu.  Efficient and effective regularized incomplete multi-view clustering.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(8):2634–2646, 2021.  10.1109/TPAMI.2020.2974828.
  * Hadsell et al. [2006] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , volume 2, pages 1735–1742. IEEE, 2006.
  * Li et al. [2014] Shao-Yuan Li, Yuan Jiang, and Zhi-Hua Zhou.  Partial multi-view clustering.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 28, 2014.
  * Shao et al. [2015] Weixiang Shao, Lifang He, and Philip S Yu.  Multiple incomplete views clustering via weighted nonnegative matrix factorization with ${L_{2,1}}$ regularization.  In _Joint European conference on machine learning and knowledge discovery in databases_ , pages 318–334. Springer, 2015.
  * Wang et al. [2018] Qianqian Wang, Zhengming Ding, Zhiqiang Tao, Quanxue Gao, and Yun Fu.  Partial multi-view clustering via consistent gan.  In _2018 IEEE International Conference on Data Mining (ICDM)_ , pages 1290–1295. IEEE, 2018.
  * Wen et al. [2021b] Jie Wen, Zhihao Wu, Zheng Zhang, Lunke Fei, Bob Zhang, and Yong Xu.  Structural deep incomplete multi-view clustering network.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 3538–3542, 2021b.
  * Xue et al. [2021] Zhe Xue, Junping Du, Changwei Zheng, Jie Song, Wenqi Ren, and Meiyu Liang.  Clustering-induced adaptive structure enhancing network for incomplete multi-view data.  In _IJCAI_ , pages 3235–3241, 2021.
  * Wen et al. [2020] Jie Wen, Zheng Zhang, Yong Xu, Bob Zhang, Lunke Fei, and Guo-Sen Xie.  Cdimc-net: Cognitive deep incomplete multi-view clustering network.  In Christian Bessiere, editor, _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pages 3230–3236. International Joint Conferences on Artificial Intelligence Organization, 7 2020.
  * Buciluǎ et al. [2006] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil.  Model compression.  In _Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 535–541, 2006.
  * Zhai et al. [2020] Yunpeng Zhai, Qixiang Ye, Shijian Lu, Mengxi Jia, Rongrong Ji, and Yonghong Tian.  Multiple expert brainstorming for domain adaptive person re-identification.  In _European Conference on Computer Vision_ , pages 594–611. Springer, 2020.
  * Zhang et al. [2018] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu.  Deep mutual learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4320–4328, 2018.
  * liang et al. [2021] xiaobo liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, Min Zhang, and Tie-Yan Liu.  R-drop: Regularized dropout for neural networks.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 10890–10905. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/5a66b9200f29ac3fa0ae244cc2a51b39-Paper.pdf.
  * Xie et al. [2016] Junyuan Xie, Ross Girshick, and Ali Farhadi.  Unsupervised deep embedding for clustering analysis.  In _International conference on machine learning_ , pages 478–487. PMLR, 2016.
  * Hu and Chen [2018] Menglei Hu and Songcan Chen.  Doubly aligned incomplete multi-view clustering.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pages 2262–2268. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.  10.24963/ijcai.2018/313.  URL https://doi.org/10.24963/ijcai.2018/313.
  * Wen et al. [2019] Jie Wen, Zheng Zhang, Yong Xu, Bob Zhang, Lunke Fei, and Hong Liu.  Unified embedding alignment with missing views inferring for incomplete multi-view clustering.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 5393–5400, 2019.
  * Wang et al. [2019] Hao Wang, Linlin Zong, Bing Liu, Yan Yang, and Wei Zhou.  Spectral perturbation meets incomplete multi-view data.  _arXiv preprint arXiv:1906.00098_ , 2019.
  * Pereira et al. [2013] Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Nikhil Rasiwasia, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos.  On the role of correlation and abstraction in cross-modal multimedia retrieval.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(3):521–535, 2013.
  * Chua et al. [2009] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng.  Nus-wide: a real-world web image database from national university of singapore.  In _Proceedings of the ACM international conference on image and video retrieval_ , pages 1–9, 2009.
  * Cai et al. [2012] Xiao Cai, Hua Wang, Heng Huang, and Chris Ding.  Joint stage recognition and anatomical annotation of drosophila gene expression patterns.  _Bioinformatics_ , 28(12):i16–i24, 2012.
  * Simonyan and Zisserman [2014] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Devlin et al. [2018] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Nair and Hinton [2010] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Icml_ , 2010.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.