This work was supported by the ANR LabEx CIMI (grant ANR-11-LABX-0040) within
the French State Programme “Investissements d’Avenir”, and the EU ICT-48
2020 project TAILOR (No. 952215). We thank Robert Mattmüller, Umberto Grandi,
Bernhard von Stengel, and Martin Antonov for their helpful feedback.


#  MM811 Project Report
Cloud Detection and Removal in Satellite Images

Dale Chen-Song, Erfan Khalaji, Vaishali Rani  Department of Computing Science
University of Alberta
Edmonton, Canada
{chensong, khalaji<EMAIL_ADDRESS>
###### Abstract

For satellite images, the presence of clouds presents a problem as clouds
obscure more than half to two-thirds of the ground information. This problem
causes many issues for reliability in a noise-free environment to communicate
data and other applications that need seamless monitoring. Removing the clouds
from the images while keeping the background pixels intact can help address
the mentioned issues. Recently, deep learning methods have become popular for
researching cloud removal by demonstrating promising results, among which
Generative Adversarial Networks (GAN) have shown considerably better
performance. In this project, we aim to address cloud removal from satellite
images using AttentionGAN and then compare our results by reproducing the
results obtained using traditional GANs and auto-encoders. We use RICE
dataset. The outcome of this project can be used to develop applications that
require cloud-free satellite images. Moreover, our results could be helpful
for making further research improvements.

###### Index Terms:

Cloud detection, cloud removal, deep learning, generative adversarial network
(GAN), AttentionGAN

##  I Introduction

Cloud coverage on satellite images makes it challenging to use such data for
applications that require seamless input. With recent advances in machine
learning, generative adversarial networks (GANs) have shown promising results
in reconstructing satellite images without cloud coverage. While the
importance of GANs is highlighted in recent studies, the lack of
implementation for real-world applications remains unaddressed. StyleGAN is an
extension of GAN architecture, which proposes large changes to the generator
model. In this project, we aim to use the dataset introduced in [1] to train
StyleGANs and compare our results by reproducing the obtained precision in [1,
2, 3]. Further, we aim to train StyleGANs by modifying loss function, network
structure, and hyperparameters. The purpose of doing so is to complete our
research and explore different network configurations. Comparing our obtained
results with [1, 2, 3] is a significant contribution to understanding how
StyleGANs, GANs, and auto-encoders behave differently while being fed the same
satellite data. The novelty of our project is centered around two points.
Firstly, to the best of our knowledge, we are the first research group that is
training StyleGANs with the dataset introduced in [1]. Secondly, auto-encoders
are not trained with the dataset that [1] published.

##  II Brief Summary of what exists

###  II-A Convolution Neural Network

[3] used a dataset of images from ZiYuan-3 (ZY-3) high resolution satellite
images [3]. For their cloud detection, they used a convolution neural network
(CNN) network due to the ZY-3 satellite images only having three visible bands
and on the near-infrared band, posing a challenge. [3] used traditional CNN,
consisting of stacking multiple players of convolution and pooling in
alternating fashion, but changing the fully connected layer with global
average pooling (GAP). GAP’s advantage is that it is more native to
convolution structure, improving the cloud detection CNN architecture. For
cloud removal, they proposed using content-texture-spectral CNN (CTS-CNN) to
recover areas that were covered by clouds. CTS-CNN can remove the thick and
thin clouds and cloud shadows. Content generation networks were used to
reconstruct the missing objects in the image with clouds. The spectral
generation network restores the spectral information of the missing objects. A
texture generation network is used to enhance the content of the content
network. Rectified linear units were applied in every output of the
convolutional layer. Images with clouds were used as input data and images
without clouds were used as targets to recover the region with clouds. While
this proposed algorithm performs decently with reconstructing image
information under clouds with the ZY-3 images, it still cannot be used in
cases where land cover may change a lot.

###  II-B Multi-Scale Convolutional Feature Fusion

Li et al. used multi-scale convolutional feature fusion (MSCFF) for remote
sensing images of different sensors [2]. It uses a symmetric encoder-decoder
module to extract multi-scale and high-level spatial features. Cloud and cloud
shadow detection is a challenge that needed to be resolved by the algorithm.
Rule-based classification technique is used for extracting clouds from images.
In the case of images having limited spectral information, bright non-cloud
objects are delegated and thin clouds are omitted. o overcome this challenge,
they try to create masks for multiple types of satellite images. MSCFF model
consists of a symmetric encoder decoder module and a multi-scale feature
fusion module. In the training phase, the model is made to learn iteratively
and we receive cloud and cloud shadow maps as output features maps of MSCFF.
In the testing phase, pre-processed test images are fed into a pre-trained
MSCFF model which will predict cloud and shadow for each pixel of the image.
The output map of the feature will then be fed into a binary classifier
outside the model for pixel-wise binarization. The output of the previous step
will yield us binary cloud and shadow mask combined together. The same dataset
which was used to train MSCFF was used to train the Deeplab and DCN network.
In order to adjust the inputs from multiple channels [2]. They changed the
first layer of both models i.e Deeplaband DCN. If one pixel is expanded, it
can boost the performance in accuracy assessment for cloud shadow detection
results for Fmask whereas the result of MSCFF remains constant. On checking
the efficiency of Fmask, Deeplab, DCN, and MSCFF, the outcome that they can
infer from the implementation of the above-mentioned algorithms is that deep
learning-based algorithms take less time than the traditional Fmask technique.

Some limitation is that, even though Li et al. can achieve higher accurate
results using MSCFF, the architecture of MSCFF has restrictions for the size
of the input image that can be processed on. Also, MSCFF can identify it as
the wrong class. For example, snow can be classified as clouds because central
areas for bright surfaces are identical and deep learning models cannot pull
out discriminative features in the patch-wise processing.

###  II-C Generative Adversarial Network

GANs are a generative model that is an unsupervised learning task in machine
learning that can determine patterns and regularities from the input data to
generate new output that could be part of the original dataset. GANs use two
neural networks, a generator to train generate new examples, against a
discriminator model that classifies the examples from the generator as real or
fake, until the discriminator model is fooled more than half the time. GANs
were used to generate cloud-free images such as done by Ebel et al. [1].

###  II-D Style Generative Adversarial Network

StyleGANs were proposed for the first time in 2014. Their purpose is to
synthesize artificial examples, such as pictures that are obscure from
authentic photographs. A typical example of a GAN application is to produce
artificial face pictures by learning from a dataset of notable faces. However,
in this project, we aim to use StyleGANs to reconstruct cloudy images. Given
that we have access to the segmentations (pixel labels) and the original
images, we are aiming to modify style GANs in such a way that they are capable
of reconstructing the image without the clouds. With respect to the fact that
style GANs perform significantly better than traditional GANs, we are hoping
to outperform the results mentioned in [1].

###  II-E Auto-Encoder

Auto-encoders are an unsupervised learning technique, that leverages neural
networks for the task of representation learning. They are a specific type of
feedforward neural network where the input is the same as the output. It
composes of 3 components, the encoder, which compresses the input, the code,
which is produced by the encoder, and the decoder, which then reconstructs the
input only using the code. They are used as another approach for image
reconstruction. Denoising auto-encoders, in particular, are used for outlier
detection, and in general, detecting noise in a search space. While auto-
encoders are computationally less expensive than GAN models, their promising
performance has led them to be used in various computing applications.
Therefore, we see it as a crucial step to have it as a part of our benchmark
for model evaluation.

##  III Milestones and Breakdown

This is the general milestones and breakdown of tasks.

  * •

Literature Review: All, October 28

  * •

Project: November 30

    * –

Creating the Pipeline: All, November 10

    * –

Training GAN: Dale, November 21

    * –

Training StyleGAN: Erfan, November 21

    * –

Training Auto-Encoder: Vaishali, November 21

  * •

Project Demo: All, December 5

  * •

Final Report Draft: All, December 7

  * •

Project Implementation: All, December 17

  * •

Final Report: All, December 17

##  IV Literature Review

Cloud removal is a matter of extracting the background, and in general,
background extraction has various applications in industry and academic
research. Furthermore, there are different techniques for background removal,
each of which might be suitable for a particular application. Authors in [4]
introduce an approach titled Stability of Adaptive Features (SoAF) that
adaptively weighs the contribution of pixels by utilizing the stability of
different features to extract the image foreground. By computing local
invariances, [4] aims to tackle the issue of object sensitivity to shadows. To
this end, the authors improved the SILTP algorithm dominating mechanism in
addressing the mentioned issue. The results indicate that the algorithm works
with high performance for office-related, pets, cubicles, and other datasets.
[5] proposes a method that works based on optical extraction. In this study,
after extracting the optical features of an image, authors perform image
decomposition through RPCA, and after that, they use the integration of the
foreground mask to obtain their final result. For model evaluation, precision
and recall were used, and they are among the commonly used metrics. Looking at
the reported numbers and the visuals, incorporating RPCA with the models’
authors of [5] proposed looks promising.

In [6], authors propose a method for tackling this challenge. The core idea
behind this study is to consider pixel variation rather than pixel
distribution when it comes to foreground detection. Foreground detection is a
binary classification problem, and [6] uses variation transformation to
address it. For variation representation, [6] forms patches for every image
pixel so that spatial information can be preserved and assumed to be
beneficial for this method. On top of variation transformation, they run a
mechanism called Deep Variation Transformation Network (DVTN). DVTN tries to
learn pixel variations using the processed data in the previous stage. Then,
The temporal information obtained by the network is used for foreground
detection that shows promising results. Given the recent advancements in the
foreground and background detection, it would be beneficial to incorporate
such progress with cloud removal techniques. In GLF-CR, there are two streams
in which SAR features are hierarchically fused into optical features for the
sake of restoring the lost information [7]. At the top of the network, there
are shallow feature extraction and cloud-free image reconstruction networks
which then pass their outputs to a network that consists of convolution layers
with a Relu activation function. Convolution layers are integrated into the
network to learn the pixel values that are at the end passed to an image
reconstruction network. The authors state that they obtain slightly higher
SSIM compared to the state-of-the-art method.

In addition to convolution networks, residual networks also proved to be
useful for classification problems. One of the recent and most cited
publications in the field of cloud removal uses residual networks to address
this issue [8]. The aim of this model is to learn the pixel values of the
parts of the cloud-free images so that it can reconstruct the cloudy images by
adding all the learned features at the bottom of the network. Even though the
structure of the network introduced in [8] is not as complex as the one in
[7], it demonstrates better results. Authors in [9] introduce a novel
methodology for semantic learning in images. They modified VGG19 as a part of
their model and define regions of interest for processing the images.
According to their result, Resnet50 has a less fluctuating learning process.
This model obtains 17.4 compression ratio while maintaining a score of 0.85
SSIM. Even though the model does not perform the state-of-the-art, the novelty
in designing a simple network given the obtained scores shows that a complex
network is not necessarily the best, and image semantic learning is possible
through designing a simple network.

Generative Adversarial Networks (GAN) are relatively newer deep learning
methods widely used for image translation, reconstruction, and transformation.
One of the recent studies uses Conditional GANs (cGAN) to learn the non-linear
mapping of two images in two different domains [10]. This study first uses
image reconstruction to generate images closer to real images. After having
enough reconstructed data samples, in the next phase of the network, both real
images and reconstructed images are given to a discriminator. The loss is
computed and propagated to the beginning of the network, where the image
generator will have the chance to use the feedback to reconstruct more
realistic images. After learning the optical features through cGAN, these
features are used to reconstruct the cloudy SAR images. Using cGANs leverages
this study by helping to obtain more accurate results. Alternatively, [11]
uses Spatial Attention GANs (SpaGANs) to simulate a human visual system where
humankind focuses on the cloudy part to collect as much as visual information
as they could. This method enhances the possibility of reconstructing the
image in a more efficient and accurate manner. The authors of [11] used
spatial attentive blocks as well as residual entities beneath a convolutional
layer in which the result is activated using a Relu function. [11] uses SSIM
to compare their work with cycle GANs and conditional GANs. The SSIM scores
demonstrate that the SpaGANs perform better compared to the cycle/conditional
GANs.

Unlike [10] that uses RBG satellite images to train cGANs with them, authors
of [12] use an extended version of RBG images in the forms of multi-spectral
images. Therefore, they turn RBG images to have four channels as a part of
their data preparation pipeline. The network architecture of this study
consists of convolution layers. Relu activation function, batch normalization,
and drop-out functionalities. In the Relu layers, authors used leaky Relu for
both encoders and discriminators. In this research paper, the visual results
seem to be clear. however, they have no table or clear methodology to make the
readers able to compare the results with other studies. With an image-to-image
translation method, [13] uses a hybrid two-stage network where CNN and GANs
work together to accomplish cloud removal using SAR data. In the first step, a
specially designed convolutional neural network (CNN) translates the synthetic
aperture radar (SAR) images into simulated optical images in an object-to-
object manner; in the second step, the simulated optical image, together with
the SAR image and the optical image corrupted by clouds, is fused to
reconstruct the corrupted area by a generative adversarial network (GAN) with
a particular loss function. The proposed model achieves an SSMI score of over
0.9 which makes it more accurate compared to the other approaches that exist
in the body of the literature.

One of the more popular methods to solve this is synthetic aperture radar
(SAR)-based methods as SAR imaging isn’t affected by clouds, and can reflect
ground information differences and changes. [14] proposed a cloud removal
method using SAR-optical data fusion and graph-based feature aggregation
network (G-FAN). Cloudy optical images and SAR images are concatenated and
transformed into hyper-feature maps, which are then inputted into G-FAN to
obtain the missing data of the cloud-covered areas, as it aggregates the
information from the SAR image and spectral information of neighbourhood and
non-neighbourhood pixels of optical images. A loss function based on smooth L1
loss function and Multi-Scale Structural Similarity Index is used for the
model. This proved to show better results than the traditional deep learning
model.

[15] on the other hand proposed an alternative way of detecting and removing
clouds that are less complicated for a common user to use, by using simulated
top-of-atmosphere radiance fields. Based on these fields, a simple scheme is
provided to reconstruct the cloud-contaminated images based on near-infrared
and visible bands. Then, to recover the band radiance of the contaminated
pixels based on simulated radiance fields. This proposed method can be used in
common satellite channels.

For example, [16] describes a method for background subtraction by determining
foreground and background cues, differing from the norm of improving the
accuracy of motion estimation. The framework they created was the Integration
of Foreground and Background (IFB). Extracting the foreground cues was done by
using a Gaussian mixture model. Background cues were obtained from finding the
static spatiotemporal features filtered by homography transformation. The
integration of the alternate cues was determined by super-pixels.

[17] proposed another method for background subtraction using Deep Pixel
Distribution Learning (DPDL), which uses Random Permutation of Temporal Pixels
(RPoTP) to represent the distribution of past observations for a particular
pixel. By looking at a variety of pixel values, they can determine whether the
pixel varies due to illumination or a breeze for example, as it follows a
pattern that is detected. Then, a CNN is used to learn if the pixel is in the
foreground or background. [18] further refined their method by proposing a
dynamic DPDL by dynamically generating RPoTP features to prevent overfitting.
A Bayesian refinement model was used to handle the random noise that was
generated from the random permutation. Another approach for background
subtraction from [19] was a framework for Arithmetic Distribution Neural
Network, which can learn the distribution of temporal pixels. Using an
improved Bayesian refinement model based on neighbouring information

The Smart MultiMedia conference is a forum where it promotes the exchange of
the latest in multimedia, whether it is technologies, systems, and
applications in research, development, or industrial perspectives. Authors are
invited to submit papers for the conference. Here are some of the papers we
reviewed from the conference in 2019.

A method that [20] proposed using clustering to differentiate the background
and foreground pixels, using the difference vectors between pixels in the
background image set and the current frame, saving computational time by only
iterating once. The clustering method used was the quartile method, but only
needed 2 clusters as it is just the background and foreground clusters. This
method saves a lot of time compared to other background subtraction methods
while providing similar results.

[21] worked on removing fog from images to improve pilots’ perception of their
safety in take-off and landing situations. Image defogging removes undesirable
factors from the image, such as weather and blurring. The authors proposed a
method to defog the image by correcting the original image with Gamma as the
guided map to correct contrast and brightness. The gamma-corrected images are
then put through a guided filter, which then was put through a Retinex model
to defog the image. The images are then combined with a histogram truncation
technique to output the mapping between 0 and 255. The algorithm proposed in
this paper removed the fog, improve the contrast and maintain consistency in
colour. [22] proposed a method to classify the race based on iris image
segmentation using machine learning. Iris recognition can be used in many
various fields of identification, as iris patterns are unique. [22] first used
feature extraction and classification, then used iris image segmentation. For
feature extraction and classification, they built a classifer, local Gabor
binary pattern, and support vector machine. This was used to separate the
irises into 2 groups, human vs non-human. Afterward, they use Hough transform
to segment the interior boundary and localize the exterior boundary by using
localizing region-based active contour model.

Deep learning is proposed for cleaning and imputing satellite images. In this
approach, supplementary data are used to obtain a coherent and cloud-free
image for a specific target date and region using a modified version of the
well-known U-Net architecture [23]. Clouds can obscure or cover the land
surface, making it difficult to analyze the data. The FMASK approach was
developed to detect distortions in satellite imagery, but it is still far from
perfect. We propose a model based on deep learning that can efficiently handle
large amounts of satellite time series data instances. The time series data
are aggregated via a special gating function, and the U-Net architecture is
used to generate a cloud- and distortion-free image. This approach outperforms
the commonly used baseline approach.

[24] proposes a cloud detection and removal method based on deep learning
architecture. The problem statement is addressed by dividing it into two sub-
problems which are a convolution neural network (CNN) architecture used for
cloud detection and also used CNN. However, multi-source data is used
(content, texture, and spectral) as an input of the unified framework. All
being said, the proposed algorithm cannot be applied to cases in which the
land cover may significantly change. In another study, [25] proposes a method
based on using YUV colour space to reconstruct cloud-free images, and a
residual symmetrical encoding-decoding architecture was used to recover
detailed information without downsampling and upsampling layers. By using deep
convolutional neural networks (CNNs), CR-GAN-PM decomposes cloud-free
background layers and cloud distortion layers. In this paper, we introduce a
YUV-GAN, a residual encoding-decoding network, and a fidelity loss in YUV
colour space for thin cloud removal on satellite imagery. The experimental
results prove that the adversarial learning between the generator and residual
discriminator can improve the accuracy of ground scene identification. Authors
of [26] describe a method for creating cloud artifact masks from satellite
imagery by combining traditional image processing with deep learning based on
U-Net, in this paper. The underlying data is a 1-dimensional elevation model,
and the general goal is to visualize scientific data in a cinematic manner. To
create realistic landscapes and backdrops in computer-generated movies,
CloudFindr masks cloud artifacts in satellite-collected DEM data. A popular
U-Net algorithm is used in this paper to detect and mask out cloud regions
using deep-learning image segmentation. CloudFindr is a method for labeling
pixels as ”cloud” or ”non-cloud” from a single-channel DEM image. It has high
accuracy.

In another study, [27] proposes a deep-learning neural network that is applied
to the reconstruction of hazy satellite images with a MSE of 0.125 and a PSNR
of 80. During rainy seasons, the proposed model does not require pre-training
datasets. The deep learning model was found to be highly effective, efficient,
adaptive, and can be applied in all environments as a means of detecting and
removing clouds from images. We only have rectangular masks to apply, and the
MSE value is quite acceptable, but some hyperparameters and parameter tuning
are needed to produce better results. Similarly, in another study concerned
with landscape image reconstruction ([28]), a generative adversarial network
is used to generate satellite imagery conditioned on population changes, also
known as the latent space, by generating random noises. Using an Adversarial
Latent AutoEncoder (ALAE) model trained on satellite data and population data,
we generated and edited images via a trained generative model. They use ALAE
for the basis of our model, which is in turn based on StyleGAN.

Given that they could think of addressing cloud removal with unsupervised
methods, [29] proposes an efficient unsupervised method for detecting relevant
changes between two temporally different images of the same scene. The method
uses a convolutional neural network for semantic segmentation. It is necessary
to train a convolutional neural network at various levels of a trained CNN in
order to create an effective DI. The CNN consists of an encoder and a decoder,
which together extract features from the input image and build a segmented
representation. Using PCC2, the semantic segmentation results were evaluated,
and the best results were obtained at epoch 15 when 89.2% accuracy was
achieved.

##  V Methods

The proposed approach for this project is using AttentionGAN, an architecture
shown below in Fig. 1. The novelty in this project is by using GANs while
preserving cycle consistency, computing loss for clouds to clear, computing
loss for clear to clouds and also preserving attention which leads to
preserving background pixels through computing pixel loss and shifting
attention towards the changes in images (clouds, shadows, etc. of images).

Figure 1:  Architecture of AttentionGAN The attention-guided generators have
built-in attention mechanisms, which can detect the most discriminative part
of images. After that, the input image, content mask, and attention mask are
fused together to synthesize the targeted image. Moreover, an attention-guided
discriminator Dya is added to distinguish only the most discriminative content

###  V-A Attention Guided Generator

GANs are composed of two competing modules; the generator G and the
discriminator D. These two neural networks contest with each other in the form
of a zero-sum game, where one’s gain is the other’s loss. X denotes training
images in the source and Y denotes the target image domain. Therefore, the
generator Gx-¿y maps x from the source domain to the generated image Gy in the
target domain Y and tries to trick the discriminator Dy. Discriminator Dy will
improve itself so that it can determine whether the sample is a generated or a
real photo. This is similar to Gy-¿x and Dx.

There are two mappings between X and Y via two generators with built-in
attention mechanisms. Gx-¿y:x-¿[My, Ry, Gy] and Gy-¿x:y-¿[Mx, Rx, Gx], where M
is the mask, R is the content mask, and G is generated images. The attention
masks M define a per-pixel intensity specifying to which extend each pixel of
the content masks R will contribute to the final rendered image. After that,
the input image x, the generated attention mask My, and the content mask Ry
are fused to generate the target image Gy.

###  V-B Attention Guided Discriminator

The generators only act on the attended regions. However, the basic
discriminators only consider the whole image currently. The discriminator Dy
takes the generated image Gy or the real image y as input and tries to
distinguish them, while the discriminator Dx takes the generated image Gx or
the real image x. Therefore, the attention-guided discriminator would be
similar to the basic discriminator but will take the attention mask as input.
Therefore, the attention-guided discriminator Dya tries to distinguish the
fake image pairs, My and Gy, and the real image pairs, My and y, while Dxa
tries to distinguish the fake image pairs, Mx and Gx, and the real image
pairs, Mx and x.

###  V-C Loss function

The loss function that we wanted to initially implement was the Tversky index
loss function [30]. However, even though this was used as it provided more
generalization and better optimization with an imbalanced dataset, and this
proved to work effectively with attention U-Net, we had to use other loss
functions in our method. AttentionGAN has various loss functions such as pixel
loss for both generator and discriminator, given that we have two generators
and two discriminators, the number of loss functions multiplies. In our model,
we optimize two generator and two discriminator loss functions. Last but not
least, we optimize a consistency loss function mostly because AttentionGAN has
a cycle consistency attribute that we had to attend to.

###  V-D Dataset

Remote sensing Image Cloud rEmoving (RICE) [31] dataset contains two parts:
RICE1 contains 500 pairs of images, each pair has images of cloud and
cloudless lands with the size of 512 x 512 without overlap; RICE2 contains 450
sets of images, each set contains three 512 x 512 size images, respectively,
the reference picture without clouds, the picture of the cloud and the mask of
its cloud. RICE1 image set was used in this project as RICE1 contained thin
clouds, which made for easier computation for training the model, while RICE2
has mask associated and images having thick clouds. The dataset is available
at: https://github.com/BUPTLdy/RICE_DATASET The image dataset was separated so
that 450 images were used for training, and 50 images were used for testing.

###  V-E Parameter Settings

We also performed data augmentation by rotating, flipping (vertically and
horizontally), cropping, and changing the background color. As a result, we
managed to change the number of data values from 450 to over 1600 images. In
order to avoid the vanishing gradient issue and the possibility of our
optimization not diverging, we decided to use an epoch decay. In other words,
we gradually decrease the epochs until the model diverges and the training
process ends. In this project, we used a different number of epochs. We
initially started with three epochs and continued to 30 and 300. However, the
results we obtained for 300 were not very different than 30 epochs so we ruled
it out. The last parameter we adjusted was the batch size. Since we did not
have a great volume of data points, we decided to keep the batch size at 1.
Therefore, for each epoch, every single image would be fed into the model
individually until all images in the training folder go through the same
process.

##  VI Results

According to the available computational resources we had to complete this
project, we decided to train AttentionGAN based on three different epoch
values. In the first stage, we trained the model with only 3 epochs to
understand how well the model will perform on the test set, therefore it was
not trained for a long period of time. In order to demonstrate our results, we
chose three photos each with a different texture background.

Figure 2 demonstrates a cloudy image of land without any plant/forest cover,
ocean, and land with plant cover. Furthermore, Figure 2 demonstrates the same
images generated after 3 and 30 epochs using AttentionGAN. Looking at the
image and analyzing it qualitatively illustrates that the background pixel
values are distorted and this is not a desirable output. Therefore, we
increased the epochs to 30 to let the model train more. After 30 epochs
background pixels are matching the reference image. Further, the thin cloud is
less in the image compared to the cloudy reference image. This can be
qualitatively assessed by comparing the background color intensity. It is
worth noting that after 30 epochs, AttentionGAN shows a promising result in
preserving background pixels.

Figure 2: Versions of satellite images with clouds, then no clouds, then 3
epochs through AttentionGAN, and then 30 epochs through AttentionGAN

##  VII Discussion and Conclusion

We proposed using AttentionGAN to generate cloudless images from cloudy
images. Even though we could only partially remove the thin cloud in RICE1
dataset, our approach demonstrates promising results in preserving background
pixel values. Moreover, to the best of our knowledge, this study is the first
implementation of AttentionGAN adapted for cloud removal, which can be
considered a considerable contribution to the body of the literature for the
reasons we describe.

Firstly, we provide empirical evidence that AttentionGAN shows considerably
good results in preserving background images when we are concerned with
removing foreground pixels. Secondly, we show that AttentionGAN, only after 3
epochs, reaches a point where it can distinguish foreground and background
depending on the provided data. Thirdly, our results demonstrate that
AttentionGAN is potentially a useful method for cloud removal from satellite
images. And last but not least, we highlight the importance of two concepts in
GANs; Cycle Consistency and Attention. These two features of our model show
that the model can be useful in removing clouds from satellite images.

It goes without saying that we experienced limitations in this study. GANs are
extremely data-hungry and we managed to perform data augmentation and change
the number of data points from 450 to 1600 by rotating, flipping, cropping,
and changing the background color of images. Also, we would have tested the
model for 3000 epochs if we had access to more data and computational
resources. Furthermore, the clouds were not fully removed from the images.

As a future work, it would be an asset to try training AttentionGAN on a
dataset with a greater volume of images using better graphical processing
units. Also, one potential approach would be using attention-U-Net, or only
U-Net as a replacement for the generator in AttentionGAN model. However, we
would look to fully remove the clouds with the next model.

## References

  * [1] P. Ebel, A. Meraner, M. Schmitt, and X. X. Zhu, “Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 7, pp. 5866–5878, 2021.
  * [2] Z. Li, H. Shen, Q. Cheng, Y. Liu, S. You, and Z. He, “Deep learning based cloud detection for medium and high resolution remote sensing images of different sensors,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 150, pp. 197–212, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0924271619300565
  * [3] Y. Chen, L. Tang, X. Yang, R. Fan, M. Bilal, and Q. Li, “Thick clouds removal from multitemporal zy-3 satellite images using deep learning,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 143–153, 2020.
  * [4] D. Yang, C. Zhao, X. Zhang, and S. Huang, “Background modeling by stability of adaptive features in complex scenes,” _IEEE Transactions on Image Processing_ , vol. 27, no. 3, pp. 1112–1125, 2017.
  * [5] Y. Ma, G. Dong, C. Zhao, A. Basu, and Z. Wu, “Background subtraction based on principal motion for a freely moving camera,” in _International Conference on Smart Multimedia_. Springer, 2019, pp. 67–78.
  * [6] Y. Ge, J. Zhang, X. Ren, C. Zhao, J. Yang, and A. Basu, “Deep variation transformation network for foreground detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 9, pp. 3544–3558, 2020\.
  * [7] F. Xu, Y. Shi, P. Ebel, L. Yu, G.-S. Xia, W. Yang, and X. X. Zhu, “Glf-cr: Sar-enhanced cloud removal with global–local fusion,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 192, pp. 268–278, 2022.
  * [8] A. Meraner, P. Ebel, X. X. Zhu, and M. Schmitt, “Cloud removal in sentinel-2 imagery using a deep residual neural network and sar-optical data fusion,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 166, pp. 333–346, 2020.
  * [9] K. Mahalingaiah, H. Sharma, P. Kaplish, and I. Cheng, “Semantic learning for image compression (slic),” in _International Conference on Smart Multimedia_. Springer, 2019, pp. 57–66.
  * [10] J. Bermudez, P. Happ, D. Oliveira, and R. Feitosa, “Sar to optical image synthesis for cloud removal with generative adversarial networks.” _ISPRS Annals of Photogrammetry, Remote Sensing & Spatial Information Sciences_, vol. 4, no. 1, 2018.
  * [11] H. Pan, “Cloud removal for remote sensing imagery via spatial attention generative adversarial network,” _arXiv preprint arXiv:2009.13015_ , 2020\.
  * [12] K. Enomoto, K. Sakurada, W. Wang, H. Fukui, M. Matsuoka, R. Nakamura, and N. Kawaguchi, “Filmy cloud removal on satellite imagery with multispectral conditional generative adversarial nets,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2017, pp. 48–56.
  * [13] J. Gao, Q. Yuan, J. Li, H. Zhang, and X. Su, “Cloud removal with fusion of high resolution optical and sar images using generative adversarial networks,” _Remote Sensing_ , vol. 12, no. 1, p. 191, 2020.
  * [14] S. Chen, W. Zhang, Z. Li, Y. Wang, and B. Zhang, “Cloud removal with sar-optical data fusion and graph-based feature aggregation network,” _Remote Sensing_ , vol. 14, no. 14, 2022. [Online]. Available: https://www.mdpi.com/2072-4292/14/14/3374
  * [15] T. Wang, J. Shi, H. Letu, Y. Ma, X. Li, and Y. Zheng, “Detection and removal of clouds and associated shadows in satellite imagery based on simulated radiance fields,” _Journal of Geophysical Research: Atmospheres_ , vol. 124, 07 2019.
  * [16] C. Zhao, A. Sain, Y. Qu, Y. Ge, and H. Hu, “Background subtraction based on integration of alternative cues in freely moving camera,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 7, pp. 1933–1945, 2019.
  * [17] C. Zhao, T.-L. Cham, X. Ren, J. Cai, and H. Zhu, “Background subtraction based on deep pixel distribution learning,” in _2018 IEEE International Conference on Multimedia and Expo (ICME)_ , 2018, pp. 1–6.
  * [18] C. Zhao and A. Basu, “Dynamic deep pixel distribution learning for background subtraction,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 11, pp. 4192–4206, 2020.
  * [19] C. Zhao, K. Hu, and A. Basu, “Universal background subtraction based on arithmetic distribution neural network,” _IEEE Transactions on Image Processing_ , vol. 31, pp. 2934–2949, 2022.
  * [20] X. Wu, X. Gao, C. Zhao, J. Wu, and A. Basu, “Background subtraction by difference clustering,” in _Smart Multimedia_ , T. McDaniel, S. Berretti, I. D. D. Curcio, and A. Basu, Eds. Cham: Springer International Publishing, 2020, pp. 45–56.
  * [21] X. Liu, C. Liu, and H. Lan, “Fog removal of aerial image based on gamma correction and guided filtering,” _Smart Multimedia_ , pp. 472–479, 2020\.
  * [22] X. Ke, L. An, Q. Pei, and X. Wang, “Race classification based iris image segmentation,” in _Smart Multimedia_ , T. McDaniel, S. Berretti, I. D. D. Curcio, and A. Basu, Eds. Cham: Springer International Publishing, 2020, pp. 383–393.
  * [23] S. Oehmcke, T.-H. K. Chen, A. V. Prishchepov, and F. Gieseke, “Creating cloud-free satellite imagery from image time series with deep learning,” in _Proceedings of the 9th ACM SIGSPATIAL International Workshop on Analytics for Big Geospatial Data_ , 2020, pp. 1–10.
  * [24] Y. Chen, L. Tang, X. Yang, R. Fan, M. Bilal, and Q. Li, “Thick clouds removal from multitemporal zy-3 satellite images using deep learning,” _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ , vol. 13, pp. 143–153, 2019.
  * [25] P. Ebel, A. Meraner, M. Schmitt, and X. X. Zhu, “Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 59, no. 7, pp. 5866–5878, 2020.
  * [26] K. Borkiewicz, V. Shah, J. P. Naiman, C. Shen, S. Levy, and J. Carpenter, “Cloudfindr: A deep learning cloud artifact masker for satellite dem data,” in _2021 IEEE Visualization Conference (VIS)_. IEEE, 2021, pp. 1–5.
  * [27] J. Saxena, A. Jain, and P. R. Krishna, “Deep learning for satellite image reconstruction,” in _Proceedings of the International Conference on Paradigms of Communication, Computing and Data Sciences_. Springer, 2022, pp. 569–577.
  * [28] T. Langer, N. Fedorova, and R. Hagensieker, “Formatting the landscape: Spatial conditional gan for varying population in satellite imagery,” _arXiv preprint arXiv:2101.05069_ , 2020.
  * [29] K. L. de Jong and A. S. Bosman, “Unsupervised change detection in satellite images using convolutional neural networks,” in _2019 International joint conference on neural networks (IJCNN)_. IEEE, 2019, pp. 1–8.
  * [30] N. Abraham and N. M. Khan, “A novel focal tversky loss function with improved attention u-net for lesion segmentation,” 2018. [Online]. Available: https://arxiv.org/abs/1810.07842
  * [31] D. Lin, G. Xu, X. Wang, Y. Wang, X. Sun, and K. Fu, “A remote sensing image dataset for cloud removal,” _arXiv preprint arXiv:1901.00600_ , 2019.


# MORI-RAN: Multi-view Robust Representation Learning via Hybrid Contrastive
Fusion

Guanzhou Ke  DSIDS 1
Beijing Jiaotong University
Beijing, China
<EMAIL_ADDRESS>1Institute of Data Science and Intelligent Decision
Support Yongqi Zhu  DSIDS 1
Beijing Jiaotong University
Beijing, China
<EMAIL_ADDRESS>Yang Yu* * Prof. Yang Yu, the corresponding author, is
with the Institute of Data Science and Intelligent Decision Support (DSIDS)
and the Frontiers Science Center for Smart High-speed Railway System, Beijing
Jiaotong University. DSIDS 1
Beijing Jiaotong University
Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Multi-view representation learning is essential for many multi-view tasks,
such as clustering and classification. However, there are two challenging
problems plaguing the community: i)how to learn robust multi-view
representations from mass unlabeled data and ii) how to balance the view
consistency and specificity. To this end, in this paper, we proposed a novel
hybrid contrastive fusion method to extract robust view-common representations
from unlabeled data. Specifically, we found that introducing an additional
representation space and aligning representations on this space enables the
model to learn robust view-common representations. At the same time, we
designed an asymmetric contrastive strategy to ensure that the model does not
obtain trivial solutions. Experimental results demonstrated that the proposed
method outperforms 12 competitive multi-view methods on four real-world
datasets in terms of clustering and classification. Our source code will be
available soon at https://github.com/guanzhou-ke/mori-ran.

###### Index Terms:

Multi-view Representation Learning, Multi-view Clustering, Multi-view Fusion,
Contrastive Learning

##  I Introduction

Multi-view Representation Learning (MRL)[1] has received a lot of attention in
recent years, especially in downstream tasks such as multi-view clustering and
classification [2]. In other words, the high-quality representation of multi-
view can significantly improve the performance of downstream tasks. Unlike
single-view representation learning, the multi-view context requires balancing
the representation of heterogeneous information from multiple sources. We
argued that methods for processing multi-source information could roughly be
divided into two categories: implicit fusion [3, 4, 5, 6, 7, 8, 9] and
explicit fusion [10, 11, 12, 13, 14, 15, 16, 17], as shown in Figure1.
Implicit fusion concatenates all view representations as input to a downstream
task, while the latter maps all view representations into a more appropriate
representation space through a fusion layer, such as the neural network.

Figure 1: Illustration of the difference between (a) implicit and (b) explicit
fusion, where $h$ denotes view-specific representations, $[\cdot]$ is
concatenation operation, $f(\cdot)$ implies the fusion function, and
$I(\cdot,\cdot)$ is mutual information measurement. Implicit fusion is usually
straightforward in maximizing mutual information between view-specific
representations, meaning each view representation contains a portion of
redundant information. It is then used as input for downstream tasks by
concatenation, when the dimensionality of the data to be processed by the
downstream task grows linearly with the number of views. Instead, explicit
fusion outputs view-specific representations as view-common representations
$z$ in either dimension via a mapping function $f(\cdot)$, such as the neural
network or the kernel function. Hence that can effectively improve the
scalability of downstream tasks.

The approach of implicit fusion has received much attention due to the
simplicity of the fusion manner. In contrast, explicit fusion requires to
design of complex auxiliary tasks to drive the fusion process. However, the
former cannot be well suited to address learning from large-scale data and
balance view consistency with view specificity [13, 11]. Some approaches [12,
14, 4] have designed fusion modules based on reconstruction to alleviate this.
These methods extract common representations from multiple data sources and
reconstruct the original data from common representations. Nevertheless, the
reconstruction forces the model to learn detailed information about the data,
which means that the learned representations contain noise, thus reducing
generalization. Although some methods introduce adversarial networks [9, 15]
to enhance the robustness of multi-view representation, these improvements
make the training process unstable.

Contrastive learning [18] has recently emerged as a promising research
direction in single-view representation learning [19, 20, 21]. Some new
approaches [11, 13, 5, 6] to extend contrastive learning to MRL have also
achieved remarkable success. Although these methods can learn high-quality
representations from a large amount of unlabeled data, these methods align
view-specific representations directly, resulting in a poor balance between
view consistency and view specificity. To this end, we propose a novel Multi-
view rObust RepresentatIon learnIng method via hybRid contrAstive fusioN
(MORI-RAN) to preserve view consistency and specificity. Specifically, we
introduce an additional alignment space in the contrastive fusion module to
align view-specific representations. At the same time, we leverage instance-
level and class-level contrastive objective functions to maximize the mutual
information between view-specific and view-common representations. Moreover,
we design an asymmetrical contrastive strategy to preserve view diversity
information in view-common representations to prevent model collapse. The
major contributions of this study are summarized as follows:

  1. 1.

We proposed a novel hybrid contrastive fusion method, called MORI-RAN, which
can balance the consistency and specificity of views by an additional
alignment representation space. Moreover, the ablation study and visualization
results demonstrate that the proposed method can preserve the diversity among
views without destroying the original representation space.

  2. 2.

We design an asymmetric representation contrastive strategy to prevent model
collapse during the fusion process. The ablation experiment shows that our
proposed fusion strategy could learn a more robust representation than the
cross-view implicit fusion.

  3. 3.

We conduct experiments on four real-world datasets to measure the clustering
and classification performance of MORI-RAN. Extensive experiments verify the
effectiveness of the proposed fusion module.

The remainder of the paper is as follows. Section II reviews related work,
including multi-view representation learning and multi-view fusion. Section
III introduces the details of the proposed framework, while Section IV
presents the experimental results demonstrating the performance of MORI-RAN in
different experimental settings. Finally, the discussion and conclusion of our
work is presented in section V and section VI, respectively.

##  II Related Work

###  II-A Multi-view Representation Learning

Multi-view representation learning aims to learn representations from multi-
view data that facilitate extracting rich information for downstream tasks,
such as clustering or classification. MRL methods can be roughly divided into
two categories: alignment and fusion. The alignment approach focuses on
capturing the relationships among multiple resource data, and the other aims
to fuse the view-specific representation into a single compact (view-common)
representation. The CCA-based methods [3, 4, 22] are representative of the
alignment. These approaches explore the statistical properties between two
views, searching for two projections to map the two views onto a low-
dimensional view-common representation. Furthermore, some methods based on
distance and similarity have been proposed [23, 24]. These methods address the
cross-view similarity learning problem by embedding multi-view data into a
common metric space. Alignment-based approaches have achieved awe-inspiring
success for tasks such as information extraction and recommender systems. This
paper falls into multi-view fusion, and we will describe related works on
fusion in the next part.

###  II-B Multi-view Fusion

The Multi-view fusion aims to extract view-common representations of arbitrary
dimensions from multiple view-specific representations instead of the direct
concatenation of all view-specific representations [3, 4, 5, 6, 8, 9].
Although the concatenation strategy may appear more intuitive, it also means
that the computational pressure is transferred to the downstream task. Our
approach falls into the explicit fusion. Unlike other shallow fusions, such as
the kernel fusion[7, 25], the subspaces fusion[26, 17], or the weight-sum
fusion[15], we leverage the neural network as the fusion mapper. The benefit
of neural networks is the ability to map representations to feature spaces of
arbitrary shape. The literature on [15, 12] employed the attention mechanism
fusion technique. These methods require a long training time and a large
amount of training data to converge. Some other methods [11, 13] using
contrastive fusion can alleviate these drawbacks, but literature [13] aligns
directly on view-specific representations, which could destroy the diversity
between views. The literature [11] drives network training based on
clustering, which is less compatible for various downstream tasks.

##  III Method

Our goal is to learn a set of robust view-common representation $z$ from $n$
data points consisting of $V$ views
$\mathcal{D}=\\{\textbf{X}^{1},\textbf{X}^{2},...,\textbf{X}^{V}\\}$, where
$\textbf{X}^{v}\in\mathbb{R}^{n\times d_{v}}$ denotes the samples of the
dimension $d_{v}$ from the $v$-th views. In this work, we suggest a non-linear
multi-view fusion method, i.e., hybrid contrastive fusion, to obtain view-
common representations $\mathbb{z}$. In this section, we first describe the
network architecture of MORI-RAN. Next, we introduce the hybrid contrastive
fusion method and the asymmetrical contrastive strategy. Finally, we describe
the objective function of MORI-RAN.

###  III-A Networks Architecture

The proposed approach, MORI-RAN, consist of $V$ view-specific encoder networks
$e_{v}(\cdot)$, a fusion block $f(\cdot)$, and a hybrid contrastive module
(HCM), as illustrated in Figure2.

  * •

View-specific encoder networks $e_{v}(\cdot)$ that extracts view-specific
representations(vectors) $h$ from each view data point, i.e.,
$\mathbb{h^{v}}=e_{v}(\mathbb{X_{v}})$. MORI-RAN is model-agnostic, meaning
that various forms of the network architecture can be treated as encoders,
e.g., Fully-Connected Networks (FCNs) or Convolution Neural Networks (CNNs).
For simplicity, we employ FCNs to build view-specific encoders.

  * •

A fusion block $f(\cdot)$ that fusions concatenating vectors
$\mathbb{\bar{h}}=[\mathbb{h^{1}},...,\mathbb{h^{v}}]$ to obtain view-common
representations $\mathbb{z}=f(\mathbb{\bar{h}})$, where
$\mathbb{\bar{h}}\in\mathbb{R}^{n\times(V\times
d_{h})},\mathbb{z}\in\mathbb{R}^{n\times d_{h}}$. We build it with fully-
connected layers with ReLU and employ the skip connection to connect the
fusion block’s input and output. The major advantage of our fusion block is
that it can map any complex data into a compatible representation space[27].

  * •

The hybrid contrastive module (HCM) treats view-common representations
$\mathbb{\bar{z}}$ and view-specific representations $\mathbb{\bar{h}}$ as
inputs. It leverage instance-level contrastive method to maximize the mutual
information between $\mathbb{\bar{z}}$ and $\mathbb{\bar{h}}$ with the
asymmetrical contrastive strategy, reducing the view-redundant representations
with class-level contrastive method. We describe the detail of HCM in the next
part.

Figure 2: Illustration of MORI-RAN. It (left) consist of $V$ view-specific
encoder networks $e_{v}(\cdot)$ to obtain view-specific representations
$\mathbb{h^{v}}$, a fusion block $f(\cdot)$ to obtain the view-common
representations $\mathbb{z}$ via fusion the concatenating vectors
$\mathbb{\bar{h}}$. We employed multiple linear layers in the fusion block
with ReLU non-linear activation function and leveraged skip-connection to
connect the input and output’s embeddings. In HCM (right), we treat
$\mathbb{z}$ and $\mathbb{\bar{h}}$ as input, and we employed an asymmetrical
contrastive strategy to maximize the mutual information between $\mathbb{z}$
and each $\mathbb{\bar{h}}$. Moreover, we apply the soft label assignment
method on $\mathbb{z}$ and each $\mathbb{\bar{h}}$, reducing their category-
agnostic information.

###  III-B The Hybrid Contrastive Module and The Asymmetrical Strategy

We introduce an additional alignment space $\mathbb{z}$ in the fusion block.
Our motivation is that learns robust view-common representations $\mathbb{z}$.
In other words, we desire to balance the view consistency and specificity. To
achieve this, we design the hybrid contrastive module (HCM), in which we
leverage the instance-level contrastive method to preserve view-specific
information, i.e., we aim to maximize the mutual information between views, as
shown in (1). Moreover, to maximize the view consistency, we optimize the
consistency of the assignment of soft labels for view-specific representations
$\mathbb{h}$ and view-common representations $\mathbb{z}$ using the class-
level contrastive method[28]. We integrate and trade-off the instance-level
and the class-level contrastive method into HCM, elaborating their objective
functions in the next part.

$\max I(\mathbb{z};\mathbb{h}^{v});\quad\text{where}\quad v=1,2,\cdots,V$ (1)

However, in a multi-view setting, it is prone to model collapse if directly
performed contrastive method between view-common representations $\mathbb{z}$
and view-specific representations $\mathbb{h}$. It is because
$\mathbb{z}=f([\mathbb{h}^{1},\cdots,\mathbb{h}^{v}])$, so a trivial solution
is that f equals the constant mapping function, meaning that $\mathbb{z}$ has
not learned effective representations for downstream tasks.

To this end, we designed an asymmetrical contrastive strategy to prevent the
model from obtaining trivial solutions. Concretely, unlike previous methods
[5, 6, 13], we disable the alignment between each view-specific representation
h and allow only view-specific representations and the introduced view-common
representation z to be aligned. The experimental result demonstrates that the
model can learn robust representations when employing the asymmetric strategy
only on the instance-level contrastive method.

###  III-C The Objective Function

With the above definitions, we design the following objective function for
MORI-RAN:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{ins}+\lambda_{2}\mathcal{L}_{cls}$ (2)

where $\mathcal{L}_{ins}$ and $\mathcal{L}_{cls}$ are instance-level
contrastive loss and class-level contrastive loss, respectively. The
parameters $\lambda_{1}$ and $\lambda_{2}$ are the trade-off factors for
instance-level and class-level contrastive loss, respectively. According to
our experimental results, we simply fix these two parameters to 1 and 0.5,
respectively. We describe the detail of the objective function in the
following.

####  III-C1 The instance-level contrastive loss

In previous works [5, 6, 13, 11], various forms of contrastive loss have been
proposed, such as InfoNCE-based[21], momentum-based[20], etc. However, all
these works require a larger batch size or extra memory during training [29].
These limitations become bottlenecks for multi-view learning. Therefore,
inspired by [30], we extend the redundancy-reduction principle to the multi-
view. We define the multi-view cross-correlation matrix as following:

$\mathcal{C}_{ij}\triangleq\frac{\sum_{b}z_{b,i}h^{v}_{b,j}}{\sqrt{\sum_{b}(z_{b,i})^{2}}\sqrt{\sum_{b}(h^{v}_{b,j})^{2}}}\
\text{where}\ v=1,2,\cdots,V$ (3)

where $b$ is batch samples and $i,j$ denotes the representation dimension of
the networks’ outputs. $\mathcal{C}$ is a square matrix of the same size as
the dimension of the networks’ output, with values range $[-1,1]$. According
to the asymmetric contrastive strategy, we fix the view-common representations
$\mathbb{z}$ and then compute the cross-correlation between $\mathbb{z}$ and
each view-specific representation $\mathbb{h}$. Finally, we compute the
instance contrastive loss as the following:

$\mathcal{L}_{ins}=\underbrace{\sum_{i}(1-\mathcal{C}_{ii})^{2}}_{\text{invariance
term}}+\lambda_{ins}\underbrace{\sum_{i}\sum_{j\neq
i}\mathcal{C}_{ij}^{2}}_{\text{redundancy reduction term}}$ (4)

where $\lambda_{ins}$ indexes a positive constant trade-off factor. According
to the original setting [30], we set $\lambda_{ins}=5\times 10^{-3}$ in our
experiments.

####  III-C2 The class-level contrastive loss

We require that the view-common representation $\mathbb{z}$ and view-specific
representations $\mathbb{h}^{v}$ be consistent in terms of the soft label
assignment. Therefore, we define the class-level contrastive loss as
following:

$\mathcal{L}_{cls}(\mathbb{z},\mathbb{h})=-\sum_{k}Q_{(k)}(\mathbb{z})\log
P_{(k)}(\mathbb{z},\mathbb{h})$ (5)

where $k$ denotes the number of soft label, and $p(\cdot,\cdot)$ is defined as
the following:

$P_{(k)}(\mathbb{z},\mathbb{h})=\frac{\exp(\mathbb{z}^{T}g_{(k)}(\mathbb{h})/\tau)}{\sum_{i\neq
k}\exp(\mathbb{z}^{T}g_{(i)}(\mathbb{h})/\tau)}$ (6)

where $g(\cdot)$ denotes the soft label assignment network, with output $k$
dimension vector. For simplicity, we employ a MLP to build $g(\cdot)$. $\tau$
is a temperature parameter [31], we set $\tau=0.07$ in our experiments.

Furthermore, we employ the Sinkhorn algorithm [32] to compute the optimal soft
label assignment of $\mathbb{z}$, the transportation plan as following:

$Q_{(k)}(\mathbb{z}):=\\{\mathbb{T}\in\mathbb{R}_{+}^{k\times
b}|\mathbb{T1}_{b}=\frac{1}{k}\mathbb{1}_{k},\mathbb{T}^{T}\mathbb{1}_{k}=\frac{1}{b}\mathbb{1}_{b}\\}$
(7)

where $\mathbb{T}$ is the transportation matrix, and $\mathbb{1}$ denotes the
identity matrix. $b$ is the size of mini-batch. According to the previous
work’s conclusion [33], we can obtain a discrete optimal solution
$\mathbb{T}^{*}$ by using a rounding procedure. Concretely, the algorithm 1
summarizes the proposed method.

# ev: $v$-th view-specific encoder.

# f: the fusion block.

# g: the soft label assignment network.

# p: projection mlp.

# l_ins: the instance-level contrastive loss, Eq.4 .

# l_cls: the class-level contrastive loss, Eq.5.

for x1,...,xv in dataloader:

# Step 1: obtain view-specific representations

h1,...,hv = e1(x1),...,ev(xv)

hs = cat(h1,...,hv) # n-by-(v*d)

# Step 2: fusion view-specific representations

z = f(hs)

# Step 3: compute the instance-level loss

L += INS(z, hs)

# Step 4: compute the class-level loss

L += CLS(z, hs)

L.backward()

update(e, f, g, p)

def INS(z, hs) # instance-level contrastive loss

hs = split(hs) # to list.

zp = p(z) # projection.

hps = [p(h) for h in hs]

subloss = 0.

# compute zp and each hp loss .

for hp in hps:

subloss += l_ins(zp, hp)

return subloss

def CLS(z, hs) # class-level contrastive loss

all = cat(z, hs) # to list.

allc = [g(h) for h in hs]

subloss = 0.

# Traverse the ‘allc‘ and get the pairwise element

for c1, c2 in pairwise(allc):

subloss += l_cls(c1, c2)

return subloss

Algorithm 1 MORI-RAN pseudocode, PyTorch-style

##  IV Experiments

Several experiments are conducted to study and evaluate the performance of
MORI-RAN on four real-world datasets. Specifically, we evaluate MORI-RAN on
the clustering and the classification task.

###  IV-A Dataset

We evaluate MORI-RAN and other competitive methods using four well-known
multi-view datasets containing image and text data. There are:

  * •

NoisyMNIST [34], which is a large-scale and widely-used benchmark dataset
consisting of 70k handwritten digit images with $28\times 28$ pixels. We use
the multi-view version provided by [4], which consists of the original digits
and the white Gaussian noise version, respectively. As most of the comparison
method cannot process such a large dataset, we only use a subset of NoisyMNIST
consisting of 10k validation images and 10k testing images.

  * •

Scene-15 [35], which contains 4,485 images with outdoor and indoor scene
environments. We use the multi-view version provided by [5], which consists of
PHOG and GIST features.

  * •

LandUse-21 [34] consists of 2100 satellite images from 21 categories with PHOG
and LBP features.

  * •

Caltech101-20 [36]. We use the multi-view version provided by [37], which
consists of 2,386 images of 20 subjects with the views of HOG and GIST
features.

We report the dataset description in Table I.

TABLE I: Dataset Description Dataset | #samples | #view | #class | #shape
---|---|---|---|---
NoisyMNIST | 20,000 | 2 | 10 | (784, 784)
Scene-15 | 4,485 | 2 | 15 | (20, 59)
LandUse-21 | 2,100 | 2 | 21 | (59, 40)
Caltech101-20 | 2,386 | 2 | 20 | (1984, 512)

###  IV-B Baseline Models

We compare the proposed method with 12 multi-view representation learning
baseline methods, including: 1) Deep Canonical Correlation Analysis (DCCA)
[3], 2) Deep Canonically Correlated Auto-encoders (DCCAE) [4], 3) Binary
Multi-view Clustering (BMVC) [38], 4) Efficient and Effective Regularized
Incomplete Multi-view Clustering (EERIMVC) [39], 5) Partial Multi-View
Clustering (PVC) [40], 6) Autoencoder in autoencoder networks (Ae2-Nets) [41],
7) Doubly Aligned Incomplete Multi-view Clustering (DAIMC) [42], 8)Incomplete
MultiModal Visual Data Grouping (IMG) [43], 9)Unified Embedding Alignment
Framework (UEAF) [44], 10)Efficient Multi-view Clustering Networks (EMC-Nets)
[12], 11) Contrastive fusion newtorks for multi-view clustering (CONAN) [11],
and 12) Incomplete Multi-view Clustering via Contrastive Prediction
(COMPLETER) [5]. Briefly, for DCCA and DCCAE, we fix the dimensionality of the
hidden representation to 10. For BMVC, we fix the length of the binary code to
128. For EERIMVC, we construct the kernel matrix using a ”Gaussian kernel” and
use the optimal $\lambda$ in the interval from $2^{-15}$ to $2^{15}$ with an
interval of $2^{3}$ [5].

###  IV-C Implementation Details

We implement MORI-RAN and other non-linear comparison methods on the PyTorch
1.10 [45] platform, running on Ubuntu 18.04 LTS utilizing a NVIDIA A100 tensor
core Graphics Processing Units (GPUs) with 40 GB memory size. We describe the
implementation details in the following:

####  IV-C1 Networks setting

For simplicity, we set the dimensionality of view-specific encoders to
$I-1024-1024-1024-O$ for all experiments, where $I$ and $O$ indicate the
dimension of the data’s input and the encoder’s output, respectively.
Concretely, we set $O=128$ for Scene-15, LandUse-21, and Caltech101-20, while
set $O=288$ for NoisyMNIST. For the fusion block, we set the dimensionality of
network to $O-O/2-O$, and leverage the skip connection to connect the input
and output vector. Moreover, we set the dimensionality of the projection
network to $O-O\times 4-O$.

####  IV-C2 Training

We use the Adam optimization technique [46] with default parameters and an
initial learning rate of $0.0001$ for both clustering and classification. For
clustering, we treat the whole dataset as the training set and validation set,
shuffling the training set during training. We train the model for 100 epochs
and set the size of the mini-batch to 256 for all experiments. Specifically,
we train the model 5 times with the different random seeds and report the
result from the lowest value of Eq. (2). For classification, we split the
dataset into a training set, a validation set, and a test set in the ratio
$8:1:1$. We first train the network for 90 epochs without labels. Then, we
leverage the method provied by[6] to generate the predicted label. Finally, we
use the test set to evaluate the classification performance of MORI-RAN.

###  IV-D Evaluation Metrics

For clustering, the performance is measured using three standard evaluation
matrices, i.e., Accuracy (ACC), Normalized Mutual Information (NMI), and
Adjusted Rand Index (ARI). For further details on these evaluation metrics,
the reader is referred to [47]. It should be noted that the validation process
of the clustering methods involves only the cases where ground truth labels
are available. For classification, ACC, Precision, and the F-Score are used.
For all measurements, a higher value indicates better performance.

###  IV-E Compared with Baseline Models

TABLE II: Clustering results on four benchmark datasets. The best and the second best values are highlighted in red and blue, respectively. | NoisyMNIST | Scene-15 | LandUse-21 | Caltech101-20
---|---|---|---|---
Method | ACC | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI | ACC | NMI | ARI
DCCA (2013) [3] | 85.53 | 89.44 | 81.87 | 36.18 | 38.92 | 20.87 | 15.51 | 23.15 | 4.43 | 41.89 | 59.14 | 33.39
DCCAE (2015) [4] | 81.60 | 84.69 | 70.87 | 36.44 | 39.78 | 21.47 | 15.62 | 24.41 | 4.42 | 44.05 | 59.12 | 34.56
BMVC (2019) [38] | 81.27 | 76.12 | 71.55 | 40.50 | 41.20 | 24.11 | 25.34 | 28.56 | 11.39 | 42.55 | 63.63 | 32.33
EERIMVC (2020) [39] | 65.47 | 57.69 | 49.54 | 39.60 | 38.99 | 22.06 | 24.92 | 29.57 | 12.24 | 43.28 | 55.04 | 30.42
PVC (2019) [40] | 41.94 | 33.90 | 22.93 | 30.83 | 31.05 | 14.98 | 25.22 | 30.45 | 11.72 | 44.91 | 62.13 | 35.77
Ae2-Nets (2019) [41] | 56.98 | 46.83 | 36.98 | 36.10 | 40.39 | 22.08 | 24.79 | 30.36 | 10.35 | 49.10 | 65.38 | 35.66
DAIMC (2019) [42] | 39.18 | 35.69 | 23.65 | 32.09 | 33.55 | 17.42 | 24.35 | 29.35 | 10.26 | 45.48 | 61.79 | 32.40
IMG (2016) [43] | - | - | - | 24.20 | 25.64 | 9.57 | 16.40 | 27.11 | 5.10 | 44.51 | 61.35 | 35.74
UEAF (2019) [44] | 67.33 | 65.37 | 55.81 | 34.37 | 36.69 | 18.52 | 23.00 | 27.05 | 8.79 | 47.40 | 57.90 | 38.98
EMC-Nets (2022) [12] | 68.71 | 70.30 | 69.17 | 31.05 | 29.87 | 16.97 | 14.81 | 25.67 | 7.66 | 43.17 | 37.11 | 29.64
CONAN (2021) [11] | 39.33 | 33.62 | 22.39 | 36.77 | 37.33 | 20.33 | 26.70 | 29.30 | 11.47 | 29.09 | 34.09 | 18.53
COMPLETER (2022) [5] | 89.08 | 88.86 | 85.47 | 41.07 | 44.68 | 24.78 | 25.63 | 31.73 | 13.05 | 68.91 | 65.36 | 75.17
MORI-RAN (Ours) | 95.22 | 89.28 | 89.83 | 43.80 | 43.81 | 25.44 | 28.60 | 33.95 | 14.22 | 65.73 | 67.19 | 54.61
TABLE III: Classification results on four benchmark datasets. | NosiyMNIST | Scene-15 | LandUse-21 | Caltech101-20
---|---|---|---|---
Method | ACC | P(recision) | F-Score | ACC | P | F-Score | ACC | P | F-Score | ACC | P | F-Score
BMVC (2019) [38] | 90.17 | 89.40 | 87.33 | 42.72 | 40.65 | 34.57 | 43.57 | 45.92 | 41.01 | 76.61 | 73.17 | 68.29
EERIMVC (2020) [39] | 81.20 | 80.98 | 81.07 | 53.71 | 54.69 | 47.45 | 37.47 | 35.49 | 33.90 | 53.59 | 48.09 | 49.44
EMC-Nets (2022) [12] | 73.19 | 71.10 | 70.84 | 45.24 | 38.69 | 39.47 | 27.66 | 25.71 | 26.03 | 50.18 | 48.68 | 49.31
CONAN (2021) [11] | 55.13 | 52.34 | 49.29 | 41.15 | 40.31 | 37.10 | 30.82 | 26.11 | 25.44 | 29.71 | 24.03 | 23.51
COMPLETER (2022) [5] | 93.17 | 90.62 | 90.30 | 75.10 | 74.00 | 73.40 | 69.67 | 70.80 | 70.40 | 91.93 | 84.20 | 83.80
MORI-RAN | 97.53 | 93.15 | 90.76 | 78.80 | 75.84 | 75.13 | 71.74 | 70.21 | 69.83 | 88.35 | 85.61 | 84.57

We evaluate our model and baseline models on four well-known datasets and
report the performance results on clustering and classification in Table II
and Table III, respectively. For clustering, the results show that MORI-RAN
can have a significant improvement compared to baseline models. It is worth
noting that the proposed method manages a performance improvement of
$6.14\%(95.22-89.08)$, $0.42\%(89.28-88.86)$, and $4.35\%(89.83-85.47)$
against the second-best method on the NosiyMNIST dataset in terms of ACC, NMI,
and ARI metrics, respectively. Moreover, our method significantly improves
over other explicit fusion methods, such as CONAN[11] and EMC-Nets[12], on all
four datasets. Nevertheless, our method shows a significant superiority over
CONAN[11], which also uses the contrastive fusion method. For classification,
the experimental results show that our proposed method performs the
classification task better on the four datasets. We also validate the
effectiveness of the explicit fusion when compared with the second best
method, COMPLETER [5].

###  IV-F Ablation Study

TABLE IV: The Ablation Study of different modules on the NosiyMNIST dataset. $\mathcal{L}_{ins}$ | $\mathcal{L}_{cls}$ | Asym. | ACC | NMI | ARI
---|---|---|---|---|---
$\checkmark$ | $\checkmark$ | $\checkmark$ | 95.22 | 89.28 | 89.83
$\checkmark$ | – | – | 88.12(-7.09) | 85.64(-3.64) | 85.64(-4.18)
$\checkmark$ | – | $\checkmark$ | 90.47(-4.75) | 87.33(-1.95) | 85.70(-4.12)
– | $\checkmark$ | – | 36.10(-59.12) | 35.92(-53.36) | 17.53(-72.3)
– | $\checkmark$ | $\checkmark$ | 32.48(-62.74) | 28.90(-60.38) | 10.14(-79.69)
$\checkmark$ | $\checkmark$ | – | 91.00(-4.21) | 86.59(-2.68) | 86.59(-3.23)

We conducted a series of ablation studies on the NoisyMNIST dataset for each
component of MORI-RAN, i.e., the instance-level contrastive module
($\mathcal{L}_{ins}$), the class-level contrastive module
($\mathcal{L}_{cls}$), and asymmetric fusion strategy (Asym.), and the results
are shown in Table IV. The experimental results show that the instance-level
contrastive module significantly affects the quality of view-common
representation $\mathbb{z}$, and the opposite of the class-level contrastive
module degrades the generalization performance of downstream tasks when used
alone. Furthermore, we found that the asymmetrical contrastive strategy is
effective for the instance-level contrastive module, but the performance is
the lowest among all combinations when it is used in combination with the
class-level contrastive module. Also, we noted that the clustering performance
is lower than the baseline of MORI-RAN when the instance-level contrastive
loss without the asymmetrical strategy. Therefore, we believe that the hybrid
contrastive fusion with asymmetrical contrastive strategy is capable of
learning robust representations.

###  IV-G Visualization

Figure 3: Visualization of representations on the NosiyMNIST dataset before
(first row) and after fusion (second row) using T-SNE [48] at epoch 1st, 10th,
20th, 40th, and 100th.

We demonstrate the change of representations before and after contrastive
fusion on the NosiyMNIST dataset, as shown in Figure3. For simplicity, we
randomly choose 1K samples projected to 2-D using T-SNE [48]. The
visualization demonstrated that the view-specific representations
$\mathbb{h}^{1}$ and $\mathbb{h}^{2}$ preserve their self-structure during
training. At the same time, the common representation $\mathbb{z}$ becomes
more separable and compact with increasing the training epoch. Therefore, we
argue that the HCM can improve the quality of view-common representation and
does not harm the view-specific representations.

##  V Discussion

Although our method achieves superior performance on four datasets, it does
not make a significant margin with the second-best method on the image-text
dataset. Therefore, we argue that MORI-RAN has some limitations in the
multimodal downstream tasks. In this study, we have used some smaller or
medium-scale datasets. However, more large-scale datasets have been released
[49, 50], and we will try to extend MORI-RAN to fit super-large datasets in
the future. It is worth noting that MORI-RAN is not a reconstruction-based
approach to learning multi-view representations, and we claim that
reconstruction is a challenging task. However, we have noticed some promising
works [51, 52] emerging recently, and we will try to balance the contrastive
approach and the reconstruction approach in future work.

##  VI Conclusion

We proposed a novel hybrid contrastive fusion method and an asymmetrical
contrastive strategy for learning robust multi-view representation. This study
introduced an additional alignment space to preserve the multi-view diversity
and extract the multi-view shared information. The ablation studies
demonstrated that the instance-level contrastive module with an asymmetrical
strategy could preserve the multi-view diversity. The class-level contrastive
module could make the view-common structure more separable and compact.
Experiments on four datasets show that the proposed method performs the
clustering and classification tasks well compared to the other 12 competitive
methods.

## Acknowledgment

This work was supported by the Fundamental Research Funds for the Central
Universities (Science and technology leading talent team project), China (No.
2021JBZD006), and the Fundamental Research Funds for the Beijing Jiaotong
University, China (No. 2021JBWZB002).

## References

  * [1] Y. Li, M. Yang, and Z. Zhang, “A survey of multi-view representation learning,” _IEEE Trans. Knowl. Data Eng._ , vol. 31, no. 10, pp. 1863–1883, 2019. [Online]. Available: https://doi.org/10.1109/TKDE.2018.2872063
  * [2] G. Chao, S. Sun, and J. Bi, “A survey on multiview clustering,” _IEEE transactions on artificial intelligence_ , vol. 2, no. 2, pp. 146–168, 2021.
  * [3] G. Andrew, R. Arora, J. Bilmes, and K. Livescu, “Deep canonical correlation analysis,” in _International conference on machine learning_. PMLR, 2013, pp. 1247–1255.
  * [4] W. Wang, R. Arora, K. Livescu, and J. Bilmes, “On deep multi-view representation learning,” in _International conference on machine learning_. PMLR, 2015, pp. 1083–1092.
  * [5] Y. Lin, Y. Gou, Z. Liu, B. Li, J. Lv, and X. Peng, “COMPLETER: incomplete multi-view clustering via contrastive prediction,” in _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_. Computer Vision Foundation / IEEE, 2021, pp. 11 174–11 183.
  * [6] Y. Lin, Y. Gou, X. Liu, J. Bai, J. Lv, and X. Peng, “Dual contrastive prediction for incomplete multi-view representation learning,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [7] F. Sun, X. Xie, J. Qian, Y. Xin, Y. Li, C. Wang, and G. Chao, “Multi-view k-proximal plane clustering,” _Applied Intelligence_ , pp. 1–15, 2022.
  * [8] X. Peng, Z. Huang, J. Lv, H. Zhu, and J. T. Zhou, “Comic: Multi-view clustering without parameter selection,” in _International conference on machine learning_. PMLR, 2019, pp. 5092–5101.
  * [9] Z. Li, Q. Wang, Z. Tao, Q. Gao, and Z. Yang, “Deep adversarial multi-view clustering network,” in _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019_. ijcai.org, 2019, pp. 2952–2958. [Online]. Available: https://doi.org/10.24963/ijcai.2019/409
  * [10] Z. Xue, G. Li, S. Wang, C. Zhang, W. Zhang, and Q. Huang, “Gomes: A group-aware multi-view fusion approach towards real-world image clustering,” in _2015 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2015, pp. 1–6.
  * [11] G. Ke, Z. Hong, Z. Zeng, Z. Liu, Y. Sun, and Y. Xie, “CONAN: contrastive fusion networks for multi-view clustering,” in _2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA, December 15-18, 2021_. IEEE, 2021, pp. 653–660. [Online]. Available: https://doi.org/10.1109/BigData52589.2021.9671851
  * [12] G. Ke, Z. Hong, W. Yu, X. Zhang, and Z. Liu, “Efficient multi-view clustering networks,” _Applied Intelligence_ , pp. 1–17, 2022.
  * [13] D. J. Trosten, S. Lokse, R. Jenssen, and M. Kampffmeyer, “Reconsidering representation alignment for multi-view clustering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1255–1265.
  * [14] J. Xu, Y. Ren, H. Tang, X. Pu, X. Zhu, M. Zeng, and L. He, “Multi-vae: Learning disentangled view-common and view-peculiar visual representations for multi-view clustering,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 9234–9243.
  * [15] R. Zhou and Y. Shen, “End-to-end adversarial-attention network for multi-modal clustering,” in _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. IEEE, 2020, pp. 14 607–14 616. [Online]. Available: https://doi.org/10.1109/CVPR42600.2020.01463
  * [16] H. Salman and J. Zhan, “Semi-supervised learning and feature fusion for multi-view data clustering,” in _IEEE International Conference on Big Data, Big Data 2020, Atlanta, GA, USA, December 10-13, 2020_. IEEE, 2020, pp. 645–650. [Online]. Available: https://doi.org/10.1109/BigData50022.2020.9378412
  * [17] M. Abavisani and V. M. Patel, “Deep multimodal subspace clustering networks,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 12, no. 6, pp. 1601–1614, 2018.
  * [18] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06)_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [19] X. Chen and K. He, “Exploring simple siamese representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 750–15 758.
  * [20] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 9729–9738.
  * [21] T. Chen, S. Kornblith, M. Norouzi, and G. E. Hinton, “A simple framework for contrastive learning of visual representations,” in _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 119\. PMLR, 2020, pp. 1597–1607. [Online]. Available: http://proceedings.mlr.press/v119/chen20j.html
  * [22] F. Yan and K. Mikolajczyk, “Deep correlation for matching images and text,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3441–3450.
  * [23] M. M. Bronstein, A. M. Bronstein, F. Michel, and N. Paragios, “Data fusion through cross-modality metric learning using similarity-sensitive hashing,” in _2010 IEEE computer society conference on computer vision and pattern recognition_. IEEE, 2010, pp. 3594–3601.
  * [24] Z. Lyu, M. Yang, and H. Li, “Multi-view group representation learning for location-aware group recommendation,” _Information Sciences_ , vol. 580, pp. 495–509, 2021.
  * [25] L. Du, P. Zhou, L. Shi, H. Wang, M. Fan, W. Wang, and Y.-D. Shen, “Robust multiple kernel k-means using l21-norm,” in _Twenty-fourth international joint conference on artificial intelligence_ , 2015.
  * [26] X. Zhao, N. Evans, and J.-L. Dugelay, “A subspace co-training framework for multi-view clustering,” _Pattern Recognition Letters_ , vol. 41, pp. 73–82, 2014.
  * [27] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [28] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9912–9924, 2020.
  * [29] Y. Tian, D. Krishnan, and P. Isola, “Contrastive multiview coding,” in _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI_ , ser. Lecture Notes in Computer Science, vol. 12356. Springer, 2020, pp. 776–794. [Online]. Available: https://doi.org/10.1007/978-3-030-58621-8\\_45
  * [30] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow twins: Self-supervised learning via redundancy reduction,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 12 310–12 320.
  * [31] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via non-parametric instance discrimination,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3733–3742.
  * [32] M. Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” _Advances in neural information processing systems_ , vol. 26, 2013.
  * [33] Y. M. Asano, C. Rupprecht, and A. Vedaldi, “Self-labelling via simultaneous clustering and representation learning,” _arXiv preprint arXiv:1911.05371_ , 2019.
  * [34] Y. Yang and S. Newsam, “Bag-of-visual-words and spatial extensions for land-use classification,” in _Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems_ , 2010, pp. 270–279.
  * [35] L. Fei-Fei and P. Perona, “A bayesian hierarchical model for learning natural scene categories,” in _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 2. IEEE, 2005, pp. 524–531.
  * [36] L. Fei-Fei, R. Fergus, and P. Perona, “Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,” in _2004 conference on computer vision and pattern recognition workshop_. IEEE, 2004, pp. 178–178.
  * [37] Y. Li, F. Nie, H. Huang, and J. Huang, “Large-scale multi-view spectral clustering via bipartite graph,” in _Twenty-ninth AAAI conference on artificial intelligence_ , 2015.
  * [38] Z. Zhang, L. Liu, F. Shen, H. T. Shen, and L. Shao, “Binary multi-view clustering,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 41, no. 7, pp. 1774–1782, 2018.
  * [39] X. Liu, M. Li, C. Tang, J. Xia, J. Xiong, L. Liu, M. Kloft, and E. Zhu, “Efficient and effective regularized incomplete multi-view clustering,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 8, pp. 2634–2646, 2020.
  * [40] R. Li, C. Zhang, H. Fu, X. Peng, T. Zhou, and Q. Hu, “Reciprocal multi-layer subspace learning for multi-view clustering,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 8172–8180.
  * [41] C. Zhang, Y. Liu, and H. Fu, “Ae2-nets: Autoencoder in autoencoder networks,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 2577–2585.
  * [42] M. Hu and S. Chen, “Doubly aligned incomplete multi-view clustering,” _arXiv preprint arXiv:1903.02785_ , 2019.
  * [43] H. Zhao, H. Liu, and Y. Fu, “Incomplete multi-modal visual data grouping.” in _IJCAI_ , 2016, pp. 2392–2398.
  * [44] J. Wen, Z. Zhang, Y. Xu, B. Zhang, L. Fei, and H. Liu, “Unified embedding alignment with missing views inferring for incomplete multi-view clustering,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 5393–5400.
  * [45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, pp. 8026–8037, 2019.
  * [46] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [47] A. Kumar, P. Rai, and H. Daume, “Co-regularized multi-view spectral clustering,” _Advances in neural information processing systems_ , vol. 24, pp. 1413–1421, 2011.
  * [48] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _Journal of machine learning research_ , vol. 9, no. 11, 2008.
  * [49] B. Wang, F. Lv, T. Yao, Y. Yuan, J. Ma, Y. Luo, and H. Liang, “Chiqa: A large scale image-based real-world question answering dataset for multi-modal understanding,” _arXiv preprint arXiv:2208.03030_ , 2022.
  * [50] J. Gu, X. Meng, G. Lu, L. Hou, M. Niu, H. Xu, X. Liang, W. Zhang, X. Jiang, and C. Xu, “Wukong: 100 million large-scale chinese cross-modal pre-training dataset and a foundation framework,” _arXiv preprint arXiv:2202.06767_ , 2022\.
  * [51] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, “Masked autoencoders are scalable vision learners,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 16 000–16 009.
  * [52] C. Wei, H. Fan, S. Xie, C.-Y. Wu, A. Yuille, and C. Feichtenhofer, “Masked feature prediction for self-supervised visual pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 14 668–14 678.