# Alleviating Mode Collapse in GAN via Pluggable Diversity Penalty Module

Sen Pei1, Richard Yi Da Xu2, Shiming Xiang1, Gaofeng Meng1

###### Abstract

The vanilla GAN suffers from mode collapse deeply. This problem usually
manifests as that the images generated by generators tend to have high
similarity amongst them, even though their corresponding latent vectors have
been very different. In this paper, a Pluggable Diversity Penalty Module
(PDPM) is proposed to address this issue. The motivation behind our framework
is to enforce the generator to generate images with distinct features if their
corresponding latent vectors are different. To this end, PDPM extracts the
feature maps of fake images via discriminator first, and then, the normalized
Gram matrix is used to measure the similarity of these feature maps. In latent
space, the similarity of latent vector pairs is given in the same way for
consistency. With these similarity relationships among latent vectors and
their corresponding image features, PDPM penalizes the generator if two latent
vectors with low similarity are mapped to fake images with similar features.
This will reduce the chance of mode collapse in GAN significantly. Further,
the proposed PDPM has been extensively compared with some mainstream methods
such as ALI, DCGAN, Unrolled GAN, WGAN_GP, VEEGAN, PacGAN, BourGAN, StarGAN
and MSGAN on several public datasets, both visual and quantitative results
show that PDPM achieves SOTA performance in image generation, image data
augmentation, domain translation and other tasks.

## Introduction

In the past few years, GAN (Goodfellow et al. 2014) has been widely used in
image generation, image inpainting, style transfer and super-resolution
reconstruction, etc. However, with the great progresses in GAN, an essential
problem has always been with us, that’s mode collapse. This phenomenon heavily
harms the diversity and quality of images generated by generator. In this
paper, we mainly focus on mode collapse alleviation and aim to generate data
in high diversity based on the available, and further, apply the augmentated
data into downstream tasks for better performance.

Figure 1: Domain translation with PDPM. Both StarGAN and PDPM are trained 10
epochs. PDPM can transfer the facial expressions of attribute images to other
faces while the vanilla StarGAN can not capture the change of eyes, this
indicates PDPM converges much faster. Besides, it is clear to see that PDPM
generates accurate attention masks of changes compared with the first column
while StarGAN captures much more background.

In general, mode collapse usually manifests as that the trained generator can
only generate images in some specific classes which really harms the data
diversity. Currently, to the best of our knowledge, there are two main ways to
alleviate mode collapse, modifying the architecture (or training method) of
GANs or refining the loss function. The main drawback of the former is its
poor generalization performance since it is effective just for some specific
networks, for example, in Unrolled GAN (Metz et al. 2016), the generator has
to consider both its current state and the state of discriminator after $K$
iterations which is hard to apply to other models. By contrast, the latter
method usually has better generalization ability, but it is difficult to
design an universal module, for example, in DRAGAN (Kodali et al. 2017) and
MSGAN (Mao et al. 2019), new penalty terms are introduced for improving data
diversity, but in our experiments (see Figure 8), we notice these methods may
generate some noisy pixels which harms the image quality. Besides, using
multiple GANs can alleviate this problem to some extent, but due to its high
cost, this method is rarely adopted in practice. Up to this day, most
approaches of mode collapse alleviation start with the original data space
while few methods deal with this problem via features of the fake images.

Moreover, in our experiments (see Figure 6), we notice an abnormal phenomenon
that sometimes very different latent vectors may be mapped to similar images
which is the essential characteristic of mode collapse. Besides, in
traditional GANs, the images generated by generator are more like the
combination of several images, and this usually leads to low image resolution
and quality. In brief, the observations stated above are appearances of mode
collapse, and they indicatie the necessity of addressing this issue.

Figure 2: The proposed PDPM. In the framework above, $f_{1}$ and $f_{2}$ are
feature maps extracted from the discriminator, $z_{1}$ and $z_{1}$ are latent
vectors while $g_{1}$ and $g_{2}$ are their corresponding fake images.
$S(\cdot)$ indicates the similarity measurement function. The key idea of PDPM
is that the similarity relationship of fake images’ features should be
consistent with their corresponding latent vectors.

To alleviate the effects of mode collapse while avoiding the drawbacks of
previous methods, a novel pluggable diversity penalty module is proposed in
this paper, hereinafter, PDPM. Figure 2 shows the pipeline of our framework.
Concretely, the more difference between latent vectors the more different
their corresponding fake images should be, i.e., if two latent vectors are
different, then PDPM enforces generator to generate two images with different
features. Unlike current mainstream methods, PDPM performs constraints in
feature space which is more robust than that in data space. In latent space,
the similarity among latent vectors is given using Gram matrix. However, in
data space, each image usually has a great amount of pixels which are
uncessary for distinguishing, and in fact, (Zhou et al. 2016) find that the
feature representations can better describe an image than pixels. Thus, PDPM
calculates the similarity of images via their corresponding feature maps.
Besides, nonlinear mapping is performed for normalizing the similarity values.
The key idea of PDPM is that the similarity of feature pairs should be
consistent with the similarity of their corresponding latent vector pairs.
This paper has the following contributions:

  * •

A novel block named PDPM is proposed to alleviate mode collapse in GAN. PDPM
has better generalization ability compared with most current methods, it can
be used in almost all GANs as a pluggable attachment. Besides, PDPM performs
constraints in feature space which is more robust and has better pixel value
stability;

  * •

PDPM has great transfer ability and low computation cost. It can be used in
image generation, image data augmentation, domain translation and so on, this
indicates PDPM is not sensitive to tasks. Besides, PDPM is almost parameters-
free which has only one balance coefficient;

  * •

Compared with other complex methods, PDPM is effective yet easy to perform.
The results in Figure 7 on 2D synthetic dataset show that PDPM can help GAN
capture much more modes effectively, and Figure 1 also suggests PDPM has good
performance in domain translation. In image data augmentation, PDPM introduces
a markable accuracy improvement on ResNet. In image generation, PDPM
outperforms MSGAN, WGAN_GP, WGAN_GP_MS and some other SOTA architectures both
visually and quantitatively (IS and FID).

## Related Work

Mode Collapse Reduction For improving data diversity and stable training,
researchers have done a lot of work. In Unrolled GAN (Metz et al. 2016), Metz
_et al._ define the generator objective with respect to an unrolled
optimization of the discriminator. The generator has to consider both its
current state and the state of discriminator after $K$ iterations. This can
lead to a better solution, but it is hard to apply to other models. In Energy-
based GAN (Zhao, Mathieu, and LeCun 2016), Zhao _et al._ use entropy to
measure the diversity of images generated by generator while maintaining low
energy state. In VEEGAN (Srivastava et al. 2017), Srivastava _et al._
introduce a variational principle for estimating implicit probability
distributions which can help avoide mode collapse. Further, in PacGAN (Lin et
al. 2018), Lin _et al._ let the discriminator make decisions based on multiple
samples from the same class which can penalize generator with mode collapse.
In BourGAN (Xiao, Zhong, and Zheng 2018), Xiao _et al._ treat modes as a
geometric structure of data distribution in a metric space which also leads to
a better genertor. Recently, in MSGAN (Mao et al. 2019), Mao _et al._ modify
the objective function for encouraging the generators to explore more minor
modes in data space. By contrast, our proposed PDPM captures modes in feature
space which is more robust than MSGAN.

Data Augmentation Learning Image data augmentation has been proven to be
effective in practice. In (Krizhevsky, Sutskever, and Hinton 2012), data
augmentation is used to reduce overfitting. Also in (Shorten and Khoshgoftaar
2019), Shorten _et al._ find that even simple techniques such as cropping,
rotating and flipping can have markable effects on reducing overfitting.
Currently, as (Perez and Wang 2017), image data augmentation mainly has three
branches which are traditional transformations, generative methods and
learning the augmentation. The former method has been well studied while the
latter has very high computation cost like NAS in (Zoph et al. 2020). In
generative models, GAN is the representative, but it is rarely used due to the
limited diversity of generated data caused by mode collapse. That’s part of
PDPM’s motivation.

Convergence and Stability of GANs The stability of training process and
convergence speed are vital for GANs. In (Goodfellow et al. 2014), the vanilla
GAN is proposed for generating high-quality images, but at that time, it is
not an easy task to train GAN stably due to the imbalance between generator
and discriminator. Further, in (Arjovsky, Chintala, and Bottou 2017),
wasserstein distance is used to measure the similarity between distributions
instead of KL-divergence, this reduces the difficulty of GAN training greatly,
and then, in (Gulrajani et al. 2017), the gradient penalty term is proposed to
enforce the Lipschiz constraint instead of using weight clipping as WGAN.
These difficulties in training GANs suggest refining loss function is not a
trivial task, and indicate the necessity of evaluating convergence and
stability of GANs.

Feature Representations of CNN A deep convolutional layer can extract the
feature of an input image accurately. In (Zeiler and Fergus 2014), deconvnet
is used to visualize the features that a fully trained model has learned.
Furthermore, in (Zhou et al. 2016), Zhou _et al._ demonstrate that the
convolutional neural networks are able to localize the discriminative regions
of image. Based on this finding, we use the features extracted from
discriminator to represent the images instead of using images directly. And in
(Selvaraju et al. 2017), the proposed Grad-CAM method also supports the
results in (Zhou et al. 2016). Figure 18 in Appendix shows some Grad-CAM
results on CelebA (Liu et al. 2015) with our trained discriminator, and these
results show PDPM can capture image features accurately.

## Motivation

To alleviate mode collapse, generator must capture much more modes of the
available data. As in Figure 3, $z$ represents latent vectors while $f(z)$
represents the features of corresponding fake image. In feature space, mode
collapse manifests as that only a part of modes can be captured by the
generator like the vanilla GAN group in Figure 3. And as a result, the data
generated by generator will gather in some specific classes or some typical
features. Inspired by this, PDPM lets the generator to generate images with
more discrete features first, and then, with the development of discriminator,
PDPM’s penalty term will make the latent vectors distribute around centers of
feature modes mainly. The case that different latent vectors clustered around
the similar feature by generator will result in higher penalty loss. In brief,
PDPM makes the features of fake images much more discrete first (PDPM starts
in Figure 3), and then with the help of discriminator and regularization term,
to assign these latent vectors to different mode centers (PDPM ends in Figure
3). In next section, the pipeline of PDPM and its mechanism will be given in
detail.

Figure 3: Illustration of mode collapse in feature space. The feature modes
distribution is the implicit real distribution of data, and the vanilla GAN
indicates the modes captured by generator without PDPM.

## Pluggable Diversity Penalty Module

As introduced in previous sections, PDPM penalizes the generator if two
different latent vectors are mapped to images with similar features. In this
section, $G(\cdot)$ and $D(\cdot)$ are used to indicate generator and
discriminator, $p_{z}$ and $p_{r}$ are distributions of latent vectors and the
real data. Besides, fake images indicate the images generated by generator,
and if not specified, $f$ is used to represent the features of fake images
extracted from discriminator.

### Measurement of similarity

Suppose $p_{z}(z)$ is the distribution of latent vectors which follows a
standard normal distribution, a batch vectors $\\{z_{1},z_{2},...,z_{m}\\}$
are randomly sampled from $p_{z}(z)$, and then, the normalized Gram matrix can
be shown as:

$G_{z}^{*}(i,j)=\frac{z_{i}^{T}z_{j}}{||z_{i}||_{2}\cdot||z_{j}||_{2}}$ (1)

where $\|\cdot\|_{2}$ represents l2-norm. It is reasonable to suppose that
$z_{i}$ and $z_{j}$ are independent identically distributed (_i.i.d._), and in
fact, $G_{z}^{*}$ still follows a gaussian distribution which can be derived
from the following claim:
_$f(x)$ and $g(x)$ are Gaussian PDFs with means $\mu_{f}$ and $\mu_{g}$ and
standard deviations $\sigma_{f}$ and $\sigma_{g}$, then the product of $p(x)$
and $g(x)$ follows a scaled Gaussian distribution with
$\mu=\frac{\mu_{f}\sigma_{g}^{2}+\mu_{g}\sigma_{f}^{2}}{\sigma_{f}^{2}+\sigma_{g}^{2}}$
and
$\sigma=\sqrt{\frac{\sigma_{f}^{2}\sigma_{g}^{2}}{\sigma_{f}^{2}+\sigma_{g}^{2}}}$.
The scale factor is
$s=\frac{\sigma_{f}^{2}\sigma_{g}^{2}}{\sqrt{\sigma_{f}^{2}+\sigma_{g}^{2}}}\mathbf{exp}\left[-\frac{(x-\mu)^{2}}{2\sigma^{2}}\right]$._

Likewise, the similarity of feature pairs can be got as follows:

$G_{f}^{*}(i,j)=\frac{f_{i}^{T}f_{j}}{||f_{i}||_{2}\cdot||f_{j}||_{2}}$ (2)

where $f_{i}$ represents flattened feature map of the _i-th_ fake image
extracted from discriminator. Since the values in Eq (1) and Eq (2) can be
zero or negative, performing division directly doesn’t make sense, and thus,
the _sigmoid_ function is used to scale them. The scale factor is denoted by
$s$ and Eqs (1) and (2) can be revised as:

$G_{z}(i,j)=\sigma(s\frac{z_{i}^{T}z_{j}}{||z_{i}||_{2}\cdot||z_{j}||_{2}})$
(3)

$G_{f}(i,j)=\sigma(s\frac{f_{i}^{T}f_{j}}{||f_{i}||_{2}\cdot||f_{j}||_{2}})$
(4)

### Loss function

For alleviating mode collapse, the diversity penalty module should have the
following attributes:

  * •

if two latent vectors are similar, their corresponding fake images are
unlikely to be very different.

  * •

if two latent vectors are different, their corresponding images have to be
different likewise, which means the corresponding feature maps exist much
difference.

Obviously, the diversity penalty module should pay much attention to the
second situation which often results in mode collapse. Through these
observations, the diversity penalty module is designed as follows:

$DP(z)=\frac{1}{m^{2}}\sum_{i=1}^{m}\sum_{j=1}^{m}\frac{G_{f}(i,j)}{G_{z}(i,j)}$
(5)

where _m_ represents the batch size, and $DP(z)$ has to be minimized when
training GANs. Taking the vanilla GAN for example, $D(\cdot)$ is trained to
maximize the probability of assigning the correct label to both real images
and fake images, also, $G(\cdot)$ is trained simultaneously to get high score
from $D(\cdot)$. Thus, the basic loss function of GAN can be formulated as
follows:

$\displaystyle\max_{G}L_{G}(z)=\mathbb{E}_{z\sim p_{z}}D(G(z))$ (6)
$\displaystyle\min_{D}L_{D}(z,x)=\mathbb{E}_{z\sim
p_{z}}D(G(z))-\mathbb{E}_{x\sim p_{r}}D(x)$ (7)
$\displaystyle\max_{G}\min_{D}\,\mathbb{E}_{z\sim
p_{z}}D(G(z))-\mathbb{E}_{x\sim p_{r}}D(x)$ (8)

To perform diversity penalty, we just need to add diversity penalty loss to
generator. The loss function of GAN with PDPM can be formulated as follows:

$\displaystyle\max_{G}L_{G}(z)=\mathbb{E}_{z\sim
p_{z}}D(G(z))-\lambda\mathbb{E}_{z\sim p_{z}}DP(z)$ (9)
$\displaystyle\min_{D}L_{D}(z,x)=\mathbb{E}_{z\sim
p_{z}}D(G(z))-\mathbb{E}_{x\sim p_{r}}D(x)$ (10)
$\displaystyle\max_{G}\min_{D}\,\mathbb{E}_{z\sim
p_{z}}D(G(z))-\mathbb{E}_{x\sim p_{r}}D(x)-\lambda\mathbb{E}_{z\sim
p_{z}}DP(z)$ (11)

where $\lambda$ is a balance coefficient of diversity penalty term. The loss
function of discriminator remains unchanged. According to Eqs (9), (10) and
(11), the training pipeline can be summarized in Algorithm 1.

### Mechanism Explanations

When training GANs, the discriminator is usually trained $k$ times while
generator is trained only once, and that means discriminator usually converges
better than generator. At the begining of training, PDPM enforces generator to
generate fake images with discrete features, and this makes it possible for
generator to capture more feature modes like PDPM starts in Figure 3. At that
time, the discriminator is not well trained, and it dose not penalize the
generator severely. Then, as the discriminator is trained better and better,
it will enfoece the generator to map the latent vectors around peaks of the
feature distribution like PDPM ends in Figure 3, and the case that latent
vectors are mapped to the saddle of feature distribution will get low score
from discriminator. Thus, when PDPM converges, most of the latent vectors will
be mapped to the surroundings of feature modes while little vectors scatter
around untypical feature centers.

Algorithm 1 GAN with PDPM training via mini-batch Adam

1: for total training _epochs_ do

2: for  _k_ times do

3: Sample a batch data from $p_{z}$ : $\\{z_{1},z_{2},...,z_{m}\\}$;

4: Sample a batch data from $p_{r}$ : $\\{x_{1},x_{2},...,x_{m}\\}$;

5: Update discriminator :

6:
$\theta_{d}\leftarrow\;\theta_{d}-\nabla_{\theta_{d}}\frac{1}{m}\sum_{i=1}^{m}L_{D}(z_{i},x_{i})$

7: end for

8: Sample a batch data from $p_{z}$ : $\\{z_{1},z_{2},...,z_{m}\\}$;

9: Update generator :

10:
$\theta_{g}\leftarrow\;\theta_{g}-\nabla_{\theta_{g}}\frac{1}{m}\sum_{i=1}^{m}L_{G}(z_{i})$

11: end for

## Experiments

In this section, PDPM is evaluated from several different views. First, in
Basic Attribute Evaluation of PDPM part, the feasibility analysis of
similarity measurement and the convergence performance of PDPM will be talked
in detail, and next, in Ablation Study part, both visual and quantitative
comparisons between PDPM and other typical architectures such as ALI (Dumoulin
et al. 2016), Unrolled GAN (Metz et al. 2016), VEEGAN (Srivastava et al.
2017), PacGAN (Lin et al. 2018) and BourGAN (Xiao, Zhong, and Zheng 2018) on
2D Synthetic Datasets will be given, which indicates the efficiency of PDPM.
Further, in Multi Task Applications, PDPM is applied into domain translation,
image generation, image data augmentation and other tasks, and the results
show PDPM outperforms most mainstream GANs such as DCGAN (Radford, Metz, and
Chintala 2015), WGAN_GP (Gulrajani et al. 2017) and MSGAN (Mao et al. 2019).

### Datasets

The datasets used in our experiments are MNIST, Fashion-MNIST, CIFAR-10,
CelebA and 2D Synthetic Datasets. For the first three datasets, only training
set is used while no changes are made in testing set. In some tables, M, F-M
and C10 are used to represent MNIST, Fashion-MNIST and CIFAR-10 respectively.

### Training Details

Unless specified, $\mathbf{Adam}$ optimizer with $\beta_{1}$=0.5 and
$\beta_{2}$=0.9 is used for training GANs, and $\mathbf{SGD}$ optimizer with
weight decay(1e-4) and momentum(0.9) is used for training ResNet. The initial
learning rates are 1e-4 and 1e-3 for GANs and ResNet respectively. The
traditional data augmentation methods contain RandomHorizontalFlip,
RandomVerticalFlip, RandomResizedCrop, RandomRotation and RandomColorJitter.

### Basic Attribute Evaluation of PDPM

Similarity measurement must has two basic characteristics:

  * •

The similarity value should be higher within classes than between classes.

  * •

Visually similar images should be close in feature space.

Picking Fashion-MNIST images as samples, the similarity values among different
categories are calculated via their corresponding feature maps extracted from
discriminator. Results shown in Figure 4 tell that the similarity value is
higher within classes than between classes.

Figure 4: Similarity Analysis on Fashion-MNIST. For avoiding occasionality, 5k
images are sampled per class. The similarity is computed via Eq (4).

Further, similar operation is performed within one specific class on Fashion-
MNIST to verify the second character stated above. Results are attached in
Appendix Figure 19, they confirm that visually similar images are also similar
in feature space and vice versa. These statistical results verify the
reasonability of our similarity measurement.

Besides, for GANs, whether it can converge stably or not is vital, and thus,
the evaluation of PDPM on MNIST, Fashion-MNIST, CIFAR-10 and CelebA is made,
respectively. Architectures of GAN are contained in Appendix Table 6. Figure 5
gives the convergence results of domain translation on CelebA, the detailed
results of other datasets are attached in Appendix Figure 10. In domain
translation, StarGAN (Choi et al. 2018) is set as baseline, two groups with
PDPM are set for comparison. From the results shown in Figure 5, it is clear
to see that PDPM can accelerate the convergence of generator significantly.

Figure 5: Loss of generator. The balance coefficient $\lambda$ in PDPM is set
to 1e-3 in StarGAN_PDPM 1 and 1e-4 in StarGAN_PDPM 2.

This acceleration is achieved because PDPM can capture accurate feature
representations which are vital in facial expression transfer. Besides, in
Figure 1, the first column indicates the original images and their
corresponding facial masks, the following columns are results of facial
expression transfer and their corresponding attention masks. These attention
masks should capture the changes between the image after transformation and
the original. It can be seen that PDPM can generate much clearer facial
attention masks with less background which can bring better and smoother
detail changes compared with the vanilla StarGAN. Besides, StarGAN in Figure 1
can not transfer the changes of eyes to new face image, this indicates that
when training with same epochs, the vanilla StarGAN converges much worse than
PDPM.

### Ablation Study

In this part, the effects of PDPM are stated in detail. First, in Evaluation
on Basic Datasets, both visual and quantitative results of mode collapse and
mode collapse alleviation are given on MNIST, Fashion-MNSIT and CIFAR-10,
further, in Evaluation on 2D Synthetic Datasets, the comparison between some
typical GANs with and without PDPM are given for precise comparison.

#### Evaluation on Basic Datasets

(a) WGAN_GP without PDPM on MNIST

(b) WGAN_GP with PDPM on MNIST

Figure 6: Alleviation of mode collapse via PDPM. (a) WGAN_GP without PDPM. (b)
WGAN_GP with PDPM $\lambda$=5. The value above each image pair indicates the
similarity value of their latent vectors. It can be found that in GAN without
PDPM, latent vectors with low similarity value can be mapped to similar images
while PDPM not.

In vanilla GANs,the latent vectors even with very low similarity may be mapped
to similar images, but with PDPM, this phenomenon is alleviated since this
situation will result in higher loss. That is, PDPM makes similar fake images
have corresponding latent vectors with higher similarity. Using the method
shown in Appendix Figure 17, the similar images and their corresponding latent
vectors can be got simultaneously with our trained generator. With these fake
images and their latent vectors, the similarity value can be calculated via
Eqs (3) and (4). Part of these results are shown in Figure 6, others are
attached in Appendix. These results indicate that PDPM prevents the generator
from mapping latent vectors with low similarity to similar fake images.

Table 1: Statistic results of diversity penalty module. Dataset | WGAN_GP | PDPM $\lambda$=5 | PDPM $\lambda$=10
---|---|---|---
M // FM | 1 | 0.68 // 0.65 | 0.78 // 0.82 | 0.82 // 0.84
2 | 0.67 // 0.63 | 0.66 // 0.77 | 0.65 // 0.85
3 | 0.64 // 0.63 | 0.77 // 0.82 | 0.77 // 0.89
4 | 0.69 // 0.61 | 0.80 // 0.83 | 0.78 // 0.81
5 | 0.64 // 0.66 | 0.78 // 0.84 | 0.77 // 0.81
6 | 0.68 // 0.64 | 0.78 // 0.83 | 0.78 // 0.81
7 | 0.64 // 0.63 | 0.75 // 0.84 | 0.77 // 0.86
8 | 0.61 // 0.66 | 0.74 // 0.86 | 0.75 // 0.87
9 | 0.67 // 0.64 | 0.80 // 0.79 | 0.80 // 0.83
10 | 0.64 // 0.65 | 0.76 // 0.84 | 0.75 // 0.82

Further, to avoid occasionality, 5k similar fake image pairs per class are
generated by the generator with and without PDPM, and Eq (3) is used for
calculating similarity between latent vectors. In Table 1, the value indicates
the similarity of latent vector pairs whose corresponding fake images are
similar under MSE metrics. It can be seen that PDPM reduces the chance of two
different latent vectors mapped to similar fake images.

In GANs, IS (Barratt and Sharma 2018; Szegedy et al. 2016) and FID (Heusel et
al. 2017) are commonly accepted metrics used for evaluationg the quality and
diversity of fake images. On the datasets stated above, 5k fake images per
class are generated using the generator with and without PDPM for calculating
IS and FID. The parameter $n_{splits}$ of IS is set to 10. Table 2 shows the
details. Greater IS value and lower FID value are signs of high quality and
diversity of generated data.

Table 2: Quantitative results of IS and FID. Dataset | WGAN_GP | PDPM $\lambda$=5 | PDPM $\lambda$=10
---|---|---|---
M |  $\uparrow$IS | 2.18$\pm$.003 | 2.19$\pm$.005 | 2.31$\pm$.005
$\downarrow$FID | 7.36$\pm$.012 | 6.43$\pm$.009 | 5.88$\pm$.011
FM |  $\uparrow$IS | 4.28$\pm$.004 | 4.38$\pm$.006 | 4.36$\pm$.005
$\downarrow$FID | 15.68$\pm$.007 | 15.97$\pm$.013 | 15.72$\pm$.011
C10 |  $\uparrow$IS | 7.35$\pm$.007 | 7.52$\pm$.005 | 7.83$\pm$.007
$\downarrow$FID | 29.84$\pm$.017 | 28.45$\pm$.015 | 29.03$\pm$.013
CelebA |  $\uparrow$IS | 2.78$\pm$.002 | 2.91$\pm$.005 | 2.94$\pm$.002
$\downarrow$FID | 33.48$\pm$.002 | 25.45$\pm$.015 | 24.86$\pm$.002

#### Evaluation on 2D Synthetic Datasets

On synthetic dataset, the quantitative evaluation results of mode collapse can
be got accurately, because the distribution of data and its modes are known.
As prior works, GANs with and without PDPM are evaluated on 2D Ring and 2D
Grid. 2D Ring dataset contains eight 2D Gaussian distributions whose centers
locate on a ring equally. 2D Grid contains twenty-five 2D Gaussian
distributions whose centers locate on the meshgrid of a square. For
comparison, PDPM is applied into the vanilla GAN, Unrolled GAN and BourGAN.
The number of modes captured by generator and the percentage of points
generated by generator in high-quality (h-q) are used as metrics. As in
(Srivastava et al. 2017), a sample is counted as high quality, if it is within
three standard deviations of the nearest mode, and the number of modes
captured by generator is the number of Gaussian centers which are nearest to
at least one high quality sample.

Table 3: Quantitative results on 2D Synthetic Dataset. | 2D Ring | 2D Grid
---|---|---
| modes | h-q | modes | h-q
GAN | 1.0 | $\times$ | 17.7 | 82.3
ALI | 2.8 | 0.13 | 12.8 | 1.6
Unrolled GAN | 7.6 | 87.97 | 14.9 | 4.89
VEEGAN | 8.0 | 86.77 | 24.4 | 77.16
PacGAN | 7.8 | 98.21 | 24.3 | 79.46
BourGAN | 8.0 | 99.76 | 25.0 | 95.91
GAN_PDPM | 2.0 | $\times$ | 21.3 | 80.8
Unrolled_PDPM | 8.0 | 99.36 | 21.7 | 75.21
BourGAN_PDPM | 8.0 | 99.89 | 25.0 | 95.99
Figure 7: Visual results on Synthetic Dataset. From the first two columns, it
can be seen that PDPM help the vanilla GAN capture more modes, especially in
2D Grid, the GAN with PDPM captures four more modes than its vanilla
counterpart.

From visual results in Figure 7 and quantitative results in Table 3, it can be
seen that GAN with PDPM captures more modes of the data distribution, and the
vanilla GAN group with PDPM outperforms the ALI and Unrolled GAN on 2D Grid
Dataset while closer to VEEGAN and PacGAN. Besides, from BourGAN and
BourGAN_PDPM in Figure 7, it is clear to see that the group with PDPM
converges to the mode centers better than its vanilla counterpart. The results
of Unrolled GAN, VEEGAN and PacGAN are from (Xiao, Zhong, and Zheng 2018), no
official codes of VEEGAN and PacGAN are found until we finish this part, thus,
PDPM is not applied into these GANs.

### Multi Task Applications

In this part, PDPM is applied into image data augmentation, image generation
and domain translation, results in these tasks all suggest that GANs with PDPM
outperform their vanilla counterpart.

#### Image Generation on CelebA

The GANs are split into two groups which are DCGAN series with {DCGAN,
DCGAN_MS, DCGAN_PDPM} and WGAN_GP series with {WGAN_GP, WGAN_GP_MS,
WGAN_GP_PDPM}. Here MS represents the mode seeking regularization propsed in
MSGAN. The coefficient $\lambda_{ms}$ is set to 1, and the penalty coefficient
$\lambda$ of PDPM shown in Eq (11) is set to 10. $\mathbf{Adam}$ with
$\beta_{1}$=0.5 and $\beta_{2}$=0.9 is used as optimizer, and the learning
rate is set to 1e-4. All GANs are trained with a batch size of 128 and 100
epochs in total. The details of architectures are attached in Appendix Table
7. Figure 8 shows the results of linear interpolation in latent space, and
Table 4 gives the quantitative results of IS and FID. In Figure 8, MS group
generates many noisy pixels, and the transition between images is not smooth
since the man with glasses only appears in last two images. By contrast, PDPM
can interpolate between two images without noisy pixels, and the transition is
much more smoother.

Figure 8: Linear interpolation in latent space. From (1) to (6) are WGAN_GP,
WGAN_GP_MS, WGAN_GP_PDPM, DCGAN, DCGAN_MS and DCGAN_PDPM. The MS group in blue
box generates some noisy pixels while PDPM group in purple box not.

It can be seen that PDPM also gets higher IS value and lower FID value
compared with DCGAN, WGAN_GP and MSGAN from quantitative results shown in
Table 4.

Table 4: IS and FID results on CelebA. | DCGAN | DCGAN_MS | DCGAN_PDPM
---|---|---|---
$\uparrow$IS | 2.113$\pm$ 0.014 | 2.360$\pm$0.006 | 2.379$\pm$0.013
$\downarrow$FID | 24.23$\pm$0.150 | 23.51$\pm$0.090 | 21.76$\pm$0.110
| WGAN_GP | WGAN_GP_MS | WGAN_GP_PDPM
$\uparrow$IS | 2.775$\pm$0.018 | 2.927$\pm$0.016 | 2.941$\pm$0.021
$\downarrow$FID | 33.48$\pm$0.011 | 24.86$\pm$0.020 | 24.18$\pm$0.031

#### Image Data Augmentation with PDPM

GANs with PDPM are used for augmentating data on MNIST, Fashion-MNIST and
CIFAR-10. The fake images are served as auxiliary training set. ResNet20
proposed in (He et al. 2016) is adopted as classification net. $\mathbf{SGD}$
optimizer is used with learning rate decay. Results of accuracy on testing set
are shown in Table 5, training details are attached in Appendix Figure 9.

Table 5: Testing Accuracy on Several Datasets.

Testing Acc | MNIST | Fashion-MNIST | CIFAR-10
---|---|---|---
Baseline | 0.9897 | 0.9257 | $\times$
DA | $\times$ | $\times$ | 0.9172
WGAN_GP | 0.9951 | 0.9394 | 0.9184
WGAN_GP_MS | 0.9961 | 0.9430 | 0.9200
PDPM_1 $\lambda=5$ | 0.9975 | 0.9465 | 0.9239
PDPM_2 $\lambda=10$ | 0.9969 | 0.9527 | 0.9212

DA : Traditional Data Augmentation

Compared with WGAN_GP, PDPM gains improvements of 0.24%, 1.33% and 0.55% on
MNIST, Fashion-MNIST and CIFAR-10 respectively. More details about the
training process refer to Appendix Figure 9.

## Conclusion

In this paper, a pluggable block called diversity penalty module (PDPM) has
been proposed to alleviate mode collapse in GAN. This penalty term is used to
enforce the similarity between feature pairs to be consistent with that
between latent vector pairs. The advantage of our proposed method is its
generalization ability, it almost can be combined with all GANs in different
architectures and vision tasks.

## References

  * Arjovsky, Chintala, and Bottou (2017) Arjovsky, M.; Chintala, S.; and Bottou, L. 2017.  Wasserstein gan.  _arXiv preprint arXiv:1701.07875_.
  * Barratt and Sharma (2018) Barratt, S.; and Sharma, R. 2018.  A note on the inception score.  _arXiv preprint arXiv:1801.01973_.
  * Choi et al. (2018) Choi, Y.; Choi, M.; Kim, M.; Ha, J.-W.; Kim, S.; and Choo, J. 2018.  Stargan: Unified generative adversarial networks for multi-domain image-to-image translation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 8789–8797.
  * Dumoulin et al. (2016) Dumoulin, V.; Belghazi, I.; Poole, B.; Mastropietro, O.; Lamb, A.; Arjovsky, M.; and Courville, A. 2016.  Adversarially learned inference.  _arXiv preprint arXiv:1606.00704_.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  In _Advances in neural information processing systems_ , 2672–2680.
  * Gulrajani et al. (2017) Gulrajani, I.; Ahmed, F.; Arjovsky, M.; Dumoulin, V.; and Courville, A. C. 2017\.  Improved training of wasserstein gans.  In _Advances in neural information processing systems_ , 5767–5777.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  In _Advances in neural information processing systems_ , 6626–6637.
  * Kodali et al. (2017) Kodali, N.; Abernethy, J.; Hays, J.; and Kira, Z. 2017.  On convergence and stability of gans.  _arXiv preprint arXiv:1705.07215_.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Krizhevsky, Sutskever, and Hinton (2012) Krizhevsky, A.; Sutskever, I.; and Hinton, G. E. 2012.  Imagenet classification with deep convolutional neural networks.  In _Advances in neural information processing systems_ , 1097–1105.
  * LeCun (1998) LeCun, Y. 1998.  The MNIST database of handwritten digits.  _http://yann. lecun. com/exdb/mnist/_.
  * Lin et al. (2018) Lin, Z.; Khetan, A.; Fanti, G.; and Oh, S. 2018.  Pacgan: The power of two samples in generative adversarial networks.  _Advances in neural information processing systems_.
  * Liu et al. (2015) Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015.  Deep Learning Face Attributes in the Wild.  In _Proceedings of International Conference on Computer Vision (ICCV)_.
  * Mao et al. (2019) Mao, Q.; Lee, H.-Y.; Tseng, H.-Y.; Ma, S.; and Yang, M.-H. 2019.  Mode seeking generative adversarial networks for diverse image synthesis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 1429–1437.
  * Metz et al. (2016) Metz, L.; Poole, B.; Pfau, D.; and Sohl-Dickstein, J. 2016.  Unrolled generative adversarial networks.  _arXiv preprint arXiv:1611.02163_.
  * Perez and Wang (2017) Perez, L.; and Wang, J. 2017.  The effectiveness of data augmentation in image classification using deep learning.  _arXiv preprint arXiv:1712.04621_.
  * Radford, Metz, and Chintala (2015) Radford, A.; Metz, L.; and Chintala, S. 2015.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Shorten and Khoshgoftaar (2019) Shorten, C.; and Khoshgoftaar, T. M. 2019.  A survey on image data augmentation for deep learning.  _Journal of Big Data_ , 6(1): 60.
  * Srivastava et al. (2017) Srivastava, A.; Valkov, L.; Russell, C.; Gutmann, M. U.; and Sutton, C. 2017.  Veegan: Reducing mode collapse in gans using implicit variational learning.  _arXiv preprint arXiv:1705.07761_.
  * Szegedy et al. (2016) Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016.  Rethinking the inception architecture for computer vision.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2818–2826.
  * Xiao, Zhong, and Zheng (2018) Xiao, C.; Zhong, P.; and Zheng, C. 2018.  Bourgan: Generative networks with metric embeddings.  _arXiv preprint arXiv:1805.07674_.
  * Xiao, Rasul, and Vollgraf (2017) Xiao, H.; Rasul, K.; and Vollgraf, R. 2017.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_.
  * Zeiler and Fergus (2014) Zeiler, M. D.; and Fergus, R. 2014.  Visualizing and understanding convolutional networks.  In _European conference on computer vision_ , 818–833. Springer.
  * Zhao, Mathieu, and LeCun (2016) Zhao, J.; Mathieu, M.; and LeCun, Y. 2016.  Energy-based generative adversarial network.  _arXiv preprint arXiv:1609.03126_.
  * Zhou et al. (2016) Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016.  Learning deep features for discriminative localization.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2921–2929.
  * Zoph et al. (2020) Zoph, B.; Cubuk, E. D.; Ghiasi, G.; Lin, T.-Y.; Shlens, J.; and Le, Q. V. 2020.  Learning data augmentation strategies for object detection.  In _European Conference on Computer Vision_ , 566–583. Springer.

## Appendix

(a) Testing Acc on MNIST

(b) Testing Acc on Fashion-MNIST

(c) Testing Acc on CIFAR-10

Figure 9: Image Classification using GAN-based Data Augmentation on Several Datasets. Table 6: Architectures of GAN in Keras-like style on MNIST / Fashion-MNIST / CIFAR-10. Names | MNIST / Fashion-MNIST | CIFAR-10
---|---|---
Generator | Layer_1 | Input Size |  Dense BN ReLU Reshape | 100x1 |  Dense ReLU Reshape | 128x1
Output Size | 7x7x24 | 4x4x512
Layer_2 | Input Size |  Conv2DTranspose BN ReLU | 7x7x24 |  Conv2DTranspose ReLU | 4x4x512
Output Size | 14x14x12 | 8x8x256
Layer_3 | Input Size |  Conv2DTranspose BN ReLU | 14x14x12 |  Conv2DTranspose ReLU | 8x8x256
Output Size | 14x14x6 | 16x16x128
Layer_4 | Input Size |  Conv2DTranspose Tanh | 14x14x6 |  Conv2DTranspose Tanh | 16x16x128
Output Size | 28x28x1 | 32x32x3
Trainable Params | 124477 | 5162755
Discriminator | Layer_1 | Input Size |  Conv2D LeakyReLU | 28x28x1 |  Conv2D LeakyReLU | 32x32x3
Output Size | 14x14x6 | 16x16x128
Layer_2 | Input Size |  Conv2D BN LeakyReLU | 14x14x6 |  Conv2D LeakyReLU | 16x16x128
Output Size | 7x7x12 | 8x8x256
Layer_3 | Input Size |  Conv2D LeakyReLU | 7x7x12 |  Conv2D LeakyReLU | 8x8x256
Output Size | 4x4x24 | 4x4x512
Layer_4 | Input Size |  Flatten Dense | 4x4x512 |  Flatten Dense | 4x4x512
Output Size | 1 | 1
Trainable Params | 9649 | 4114689
Table 7: Architectures of GAN in Keras-like style on CelebA. Names | DCGAN Series | WGAN_GP Series
---|---|---
Generator | Layer_1 | Input Size |  Dense Reshape BN ReLU | 100x1 |  Dense ReLU Reshape | 100x1
Output Size | 4x4x1024 | 4x4x1024
Layer_2 | Input Size |  Conv2DTranspose BN ReLU | 4x4x1024 |  Conv2DTranspose ReLU | 4x4x1024
Output Size | 8x8x512 | 8x8x512
Layer_3 | Input Size |  Conv2DTranspose BN ReLU | 8x8x512 |  Conv2DTranspose ReLU | 8x8x512
Output Size | 16x16x256 | 16x16x256
Layer_4 | Input Size |  Conv2DTranspose BN ReLU | 16x16x256 |  Conv2DTranspose ReLU | 16x16x256
Output Size | 32x32x128 | 32x32x128
Layer_5 | Input Size |  Conv2DTranspose Tanh | 32x32x128 |  Conv2DTranspose Tanh | 32x32x128
Output Size | 64x64x3 | 64x64x3
Trainable Params | 12679171 | 12675331
Discriminator | Layer_1 | Input Size |  Conv2D LeakyReLU | 64x64x3 |  Conv2D LeakyReLU | 64x64x3
Output Size | 32x32x128 | 32x32x128
Layer_2 | Input Size |  Conv2D BN LeakyReLU | 32x32x128 |  Conv2D LeakyReLU | 32x32x128
Output Size | 16x16x256 | 16x16x256
Layer_3 | Input Size |  Conv2D BN LeakyReLU | 16x16x256 |  Conv2D LeakyReLU | 16x16x256
Output Size | 8x8x512 | 8x8x512
Layer_4 | Input Size |  Conv2D BN LeakyReLU | 8x8x512 |  Conv2D LeakyReLU | 8x8x512
Output Size | 4x4x1024 | 4x4x1024
Layer_5 | Input Size |  Flatten Dense Sigmoid | 4x4x1024 |  Flatten Dense | 4x4x1024
Output Size | 8x8x512 | 8x8x512
| Output Size |  | 1 |  | 1
| Trainable Params | 11038081 | 11034497

(a) Convergence on MNIST

(b) Convergence on Fashion-MNIST

(c) Convergence on CIFAR-10

Figure 10: Convergence on Several Datasets. In images above, the lines
indicate the averaging loss of 10 generators, the band in light color marks
out the boundaries of all losses. Since the loss of discriminator is almost
the same in each groups (WGAN_GP, Ours_1 and Ours_2), we just give the results
on generator here.

(a) DPM on CelebA

(b) DPM on Fashion-MNIST

Figure 11: Alleviation of mode collapse via DPM on several datasets.

(a) WGAN_GP

(b) WGAN_GP_DP with $\lambda_{2}=5$

(c) WGAN_GP_DP with $\lambda_{2}=10$

Figure 12: Generated Images on MNIST.

(a) WGAN_GP

(b) WGAN_GP_DP with $\lambda_{2}=5$

(c) WGAN_GP_DP with $\lambda_{2}=10$

Figure 13: Generated Images on Fashion-MNIST.

(a) WGAN_GP

(b) WGAN_GP_DP with $\lambda_{2}=5$

(c) WGAN_GP_DP with $\lambda_{2}=10$

Figure 14: Generated Images on CIFAR-10.

(a) DCGAN

(b) DCGAN_MS

(c) DCGAN_DP with $\lambda_{2}=10$

(d) WGAN_GP

(e) WGAN_GP_MS

(f) WGAN_GP_DP with $\lambda_{2}=10$

Figure 15: Generated Images on CelebA.

(a) DCGAN

(b) DCGAN_MS

(c) DCGAN_DP

(d) WGAN_GP

(e) WGAN_GP_MS

(f) WGAN_GP_DP

Figure 16: Training loss of generator and discriminator on CelebA. Diversity
penalty coefficient $\lambda$ in WGAN_GP_DP and DCGAN_DP is set to 10.
$\lambda_{ms}$ in WGAN_MS and DCGAN_MS is set to 1.

(a) Our Detecting Framework

(b) Some mode collapse samples on MNIST

(c) Alleviation of mode collapse on CelebA

Figure 17: Reduction of data diversity detected in our experiments. We use the
adversarial learning method to get two similar images under the MSE metrics.
Given random noise $z_{1}$ and its corresponding generated image img1, we
optimize random noise $z_{2}$ using back propagation to minimize the
difference between its corresponding image img2 and img1. Our results show
that $z_{2}$ may be very different from $z_{1}$, even though their
corresponding generated images are similar. (a) WGAN_GP (Gulrajani et al.
2017) on MNIST (LeCun 1998): some examples of very different noise vectors
which are mapped to similar images. The value above each image pair indicates
the similarity of their corresponding noise vectors. (b) WGAN_GP (Gulrajani et
al. 2017) with diversity penalty on CelebA (Liu et al. 2015): results show
that the similar images also have corresponding latent vectors with higher
similarity compared with (b). Calculation of similarity is shown in formula
(3). Figure 18: Grad-CAM samples on CelebA. The discriminator of GAN with PDPM
can generate facial mask accurately. The architecture of discriminator is
shown in Appendix Table 7. Figure 19: Analysis within a Specific Class on
Fashion-MNIST. The result supports that even within a specific class, visually
similar images remain close in feature space.


# Cryptography approach for Secure Outsourced Data Storage in Cloud
Environment

Rishabh Gupta
Department of Computer Science and Engineering
The University of Aizu, Aizuwakamatsu, Japan
<EMAIL_ADDRESS>
Deepika Saxena
Department of Computer Science and Engineering
The University of Aizu, Aizuwakamatsu, Japan
<EMAIL_ADDRESS>
Ashutosh Kumar Singh
Department of Computer Applications
National Institute of Technology
Kurukshetra, India
<EMAIL_ADDRESS>

###### Abstract

A large amount of data and applications are migrated by researchers,
stakeholders, academia, and business organizations to the cloud environment
due to its large variety of services, which involve the least maintenance
cost, maximum flexibility, and on-demand service for storage, computation, and
data distribution intentions. Despite the various characteristics the cloud
environment supports, it also faces many challenges. However, data users may
not completely trust a cloud environment that is engaged by a third party.
Every cloud user always has a prime concern, i.e., security. Numerous methods
have been designed to solve the issue of data security during data storage,
calculation, and sharing across stakeholders and users. Nevertheless, there is
a lack of existing methods that tackle the issue of the security of data when
it is stored in a cloud environment. This article presents a precise security
method that has handled the security of data while it is being shared and
stored in the cloud. These methods have been utilized to lessen security
assaults and prevent unauthorized parties from accessing the actual data. The
article is concluded with some limitations and recommendations for the future
in terms of secure data retention and distribution.

_Keywords_ Cloud Computing $\cdot$ Data Security $\cdot$ Cloud Storage $\cdot$
Privacy Preservation $\cdot$ Cryptography

##  1 Introduction

With its fast data processing and storage capacity, cloud computing is
essential for all applications that need high processing costs, such as
machine learning for classification [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,
13, 14, 15, 16, 17, 18]. Cloud computing offers open, on-demand, scalable, and
easy-to-use computing services [19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29,
30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42]. The cloud server requires
a huge quantity of data for computation and sharing among users [43, 44, 45,
46, 47, 48, 49, 50, 51, 52, 53, 54]. These data are obtained from different
data owners. Every data owner has its data containing sensitive data such as
personal images, social media data, and medical records [55, 56, 57, 58, 59,
60, 61, 62]. Such sensitive data is transmitted to the cloud service provider
for keeping and processing purposes [63, 64, 65, 66, 67, 68]. The data owners
lose their rights, and also do not know about who is accessing outsourced data
[69, 70]. But, after outsourced data, different organizations or any adversary
can access the data since the cloud service provider holds all data on its
servers and it is the third party [71, 72]. The data owner may not trust the
cloud server [73, 74]. Cloud computing suffers from various security problems,
which are widely discussed topics [75]. In fact, cloud computing offers many
tools but also has critical security concerns. Therefore, to preserve data
privacy from others, owners first convert their data before passing data to
another entity. Although the data is converted using existing approaches, such
as differential privacy [76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87]. To
protect the actual information of the owners, the existing approach produces
the noise using a probability-based distribution function. The generated noise
is added to the actual information and acquires the noise-added data which is
stored on the cloud server and shared across all the entities. But this
approach does not trade-off between accuracy and privacy. To resolve the above
challenge, a cryptography approach for secure outsourced data storage in the
cloud environment is proposed for secure computation and distribution among
entities. The proposed scheme adopts a cryptography approach to encrypt the
owners’ data. Data is secured by encryption and decryption processes. So that
the adversary is unable to understand the actual data of each owner. In this
proposed model, it is considered that the data owners are not communicating
with each other. The cloud server is honest-but-curious (non-colluding),
following the protocol strictly. The data in encrypted form is forwarded to
the cloud server that stores it. Cloud server performs computation on the
received data and shares it among users whenever they send the request for the
owners’ data. Users decrypt the data that are obtained from the cloud service
provider and acquire the actual information. Therefore, the proposed model
stores and shares the data in such a way that doesn’t reveal the data owners’
data.

##  2 Related Work

To assess the suitability of fog computing, a three-tier architecture is
presented in [88], which transmits the data from terminal nodes to the cloud
through the fog nodes. On the basis of this model, the authors described the
mathematical disparity between the traditional cloud computing paradigm and
the fog computing paradigm for different renewable and non-renewable energy
resources and related costs. A case study was conducted and the result
revealed that fog computing outperforms cloud computing in the sense of the
Internet of Things ( IoT), with a large number of latency-sensitive
applications. The proposed architecture reduced latency, energy consumption,
and carbon dioxide emissions but suffered from limited data sharing.

In the current storage schema, the data of the users are stored entirely on
cloud servers. By doing so, users lose their right to data protection and face
the possibility of privacy leakage. To preserve the privacy of data, a fog
computing-driven three-layer storage framework was devised by Wang et al.
[89]. The suggested system has the potential to make the most of cloud storage
while still preserving data privacy. A distribution method based on risk
mitigation is used to spread the components among various cloud storage
platforms. In addition, the Hash-Solomon Code algorithm was also designed to
divide data into different sections. Each component can be put on the cloud,
fog, and local machines and reconstructed from plaintext. The privacy
protection of sensitive data is tackled by fog computing-based data collection
methods with exceptional performance. However, user revocation was not
considered.

The Server-aided Network Topology (SNT) system and the Fully-connected Network
Topology (FNT) system, both of which are based on connections to SNT and FNT
servers, were introduced by Phong and Phuong [90] for stochastic gradient
descent (SGD) protection. The SGD or its derivatives can be used by several
machine learning trainers in these systems over the combined dataset without
sharing the local dataset of each trainer. The clients communicate the locally
derived weights for the model to the server, which then aggregates the
incoming weights and transmits them back to the clients. Applying privacy-
preserving weight transfer to network training in the encrypted domain can
offer robustness against model extraction attempts. Such an approach’s benefit
is that it withstands updated loss and doesn’t need frequent synchronization.
The experiments were conducted using various datasets, and results showed that
their system outperformed the existing system in terms of learning accuracy.
These systems are also effective in terms of calculation & communication. The
developed systems, which utilized weight parameters rather than gradient
parameters, had accuracy close to SGD.

A privacy-preserving price e-negotiation (3PEN) protocol is introduced in [91]
that protects the prices of the seller and buyer. A secure multi-party
computation (SMC) technique was employed for the input data encryption. The
oracle and qubit comparator was applied to acquire the final state, depicting
the two-party’s price results. It presents the results of the prices of the
two parties’ comparisons, and counting is done to total the number of products
that satisfy the trading requirements (i.e., the number of times the buyer’s
offer exceeds the seller’s asking price by a predetermined amount). This
prevents any confidential information about Alice’s prices from being
discovered. The suggested protocol offers higher security and significantly
lowers the likelihood of an external eavesdropper attack than the traditional
channel while increasing computing costs.

Table 1: A capsulization of cryptography-based models Model/Scheme /Framework |  Workflow |  Implementation |  Outcomes |  Drawbacks & Future Scope
---|---|---|---|---
A fog computing-based network model for data service [88] |  • The network model of fog computing was constructed to exchange traffic pattern • A case study was performed with traffic generated from the 100 highest-populated cities |  • The experimental results performed range from 2 to 10 terminal nodes • The ratio of the total bytes transmitted to the fog computing is [0.05, 0.75] |  • Reduced 50% service latency due to fog computing • Less computational and storage overhead |  • This work can be extended by considering the heterogeneous devices
A cloud storage scheme for data privacy [89] |  • The data was encoded using hash transformation • The distribution proportion was computed to divide the data into small parts |  • The experiments were performed based on ‘one more block’ principle • Audio (.MP3, 84.2 MB), Picture (.NEF, 24.3 MB), and Video (.RMVB, 615 MB) were used for experiments |  • It reduced the lower servers’ storage pressure • Safety analysis proves that this framework is secure, feasible, and efficient |  • A fraction of data gets exposed to each outsourcing cloud server
A secure sharing network weights system driven on multilayered neural network [90] |  • The security of input data was performed via symmetric encryption • SGD computed the gradient parameters |  • Breast Cancer, Skin/non-Skin, and MNIST datasets were adopted to evaluate this framework • The experimental results were achieved in the python programming language |  • Maintain the accuracy of deep learning at its highest level • According to security analysis, the proposed framework is more computationally efficient |  • The privacy of the output is not being considered • Low efficiency and lack of privacy concerns are caused by updated weight
A secure multi-party computation-based e-negotiation protocol for price protection [91] |  • The two-party’s prices were acquired by using a qubit comparator • The corresponding state was prepared with the assistance of Oracle operations |  • Not Available |  • The proposed model is more secure according to privacy analysis • Less communication cost |  • Only one-to-one communication (between seller and buyer) • This work can be extended by using multi-party price e-negotiation
OU cryptosystem-based privacy-preserving protocol for data classification [92] |  • The data was encrypted using OU encryption algorithm • Users acquired the secure classification results by applying an outsourced classification protocol |  • The MIRACL library was used for simulations • The Naive Bayes classifier was trained using titanic, thyroid, breast cancer, and heart data sets |  • Increased efficiency due to OU cryptosystem • The computation cost is modulus $n$, which is less than the cost of the Paillier cryptosystem |  • A large number of calculations makes the system complex • The system’s efficiency can be enhanced by applying the optimization mechanisms

Due to resource limitations, classifier owners employ outsourced
classification services to move their classifiers to remote servers, where
users request services of classification. To protect the classifier, Chai et
al. [92] devised an outsourced classification protection scheme having less
computational and communication overhead. The authors utilized a substitutive
OU cryptosystem to mitigate the bandwidth consumption. The proposed scheme
resists the substitution-then-comparison (STC) attack, ensuring that users can
accurately receive the classification outcome without disclosing the
classifier’s private, with limited data sharing. Table I presents a thumbnail
of relevant models based on the classical mechanism containing potential
details.

##  3 System Model

The system model consists of three entities, such as Data Owners ($DO$), Users
($U$), and Cloud Service Provider ($CSP$).

Figure 1: Schematic representation of cryptography-driven model

Fig. 1 presents the building block of the cryptography technique, which
consists of symmetric and asymmetric approaches. In symmetric approach, each
data owner $DO_{1}$, $DO_{2}$, $\dots$, $DO_{n}$ and user $U_{1}$, $U_{2}$,
$\dots$, $U_{m}$ has secret keys $S_{\mathcal{K}1}$, $S_{\mathcal{K}2}$,
$\dots$, $S_{\mathcal{K}n}$. $DO_{1}$, $DO_{2}$, $\dots$, $DO_{n}$ encrypts
data $D_{1}$, $D_{2}$, $\dots$, $D_{n}$ with their secret keys
$S_{\mathcal{K}1}$, $S_{\mathcal{K}2}$, $\dots$, $S_{\mathcal{K}n}$,
respectively, and obtained encrypted data $D_{1}^{E}$, $D_{2}^{E}$, $\dots$,
$D_{n}^{E}$. The produced data are passed to the users $U_{1}$, $U_{2}$,
$\dots$, $U_{m}$ through $CSP$ for utilization purposes. $U_{1}$, $U_{2}$,
$\dots$, $U_{m}$ decrypt the obtained encrypted data $D_{1}^{E}$, $D_{2}^{E}$,
$\dots$, $D_{n}^{E}$ with their secret keys $S_{\mathcal{K}1}$,
$S_{\mathcal{K}2}$, $\dots$, $S_{\mathcal{K}n}$ and acquire the plain
documents $D_{1}$, $D_{2}$, $\dots$, $D_{n}$. Similarly, in asymmetric
approach, each data owner $DO_{1}$, $DO_{2}$, $\dots$, $DO_{n}$ has data
$D_{1}$, $D_{2}$, $\dots$, $D_{n}$, and public keys $P_{\mathcal{B}1}$,
$P_{\mathcal{B}2}$, $\dots$, $P_{\mathcal{B}n}$. $DO_{1}$, $DO_{2}$, $\dots$,
$DO_{n}$ encrypts data $D_{1}$, $D_{2}$, $\dots$, $D_{n}$ with their
$P_{\mathcal{B}1}$, $P_{\mathcal{B}2}$, $\dots$, $P_{\mathcal{B}n}$,
respectively, and acquired encrypted data $D_{1}^{E}$, $D_{2}^{E}$, $\dots$,
$D_{n}^{E}$. The acquired data are transferred to the users $U_{1}$, $U_{2}$,
$\dots$, $U_{m}$ through $CSP$ for various purposes. $U_{1}$, $U_{2}$,
$\dots$, $U_{m}$ decrypt the acquired encrypted data $D_{1}^{E}$, $D_{2}^{E}$,
$\dots$, $D_{n}^{E}$ with their private keys $P_{\mathcal{V}1}$,
$P_{\mathcal{V}2}$, $\dots$, $P_{\mathcal{V}m}$ and obtain the plain
informations $D_{1}$, $D_{2}$, $\dots$, $D_{n}$.

##  4 Data Encryption and Decryption

Let data $D_{1}$, $D_{2}$, $\dots$, $D_{n}$ $\in$ $\mathbb{D}$ that are
protected by encryption and decryption process of symmetric and asymmetric
cryptography techniques using sets of public keys ($P_{\mathcal{B}}$), secret
keys ($S_{\mathcal{K}}$), and private keys ($P_{\mathcal{V}}$). The symmetric
cryptography technique maps $\Omega_{E}$ : $D_{i}$ $\times$ $S_{\mathcal{K}}$
$\rightarrow$ $D_{i}^{E}$, and $\Lambda_{D}$; $D_{i}^{E}$ $\times$
$S_{\mathcal{K}}$ $\rightarrow$ $D_{i}$, such that
$\Lambda_{D}$$\Omega_{E}$($\mathcal{D}_{i}$, $S_{k}$) = $\mathcal{D}_{i}$;
whereas $\Omega_{E}$ and $\Lambda_{D}$; are the encryption and decryption
operations. While the asymmetric cryptography technique maps $\Omega_{E}$ :
$D_{i}$ $\times$ $P_{\mathcal{B}}$ $\rightarrow$ $D_{i}^{E}$, and
$\Lambda_{D}$; $D_{i}^{E}$ $\times$ $P_{\mathcal{V}}$ $\rightarrow$ $D_{i}$,
such that $\Lambda_{D}$$\Omega_{E}$($\mathcal{D}_{i}$, $\mathcal{P}_{b}$) =
$\mathcal{D}_{i}$; $\forall$ $\mathcal{D}_{i}$ $\in$ $D_{i}$,
$\mathcal{S}_{k}$ $\in$ $S_{\mathcal{K}}$, $\mathcal{P}_{b}$ $\in$
$P_{\mathcal{B}}$, $\mathcal{P}_{v}$ $\in$ $P_{\mathcal{V}}$ where
$\mathcal{D}_{i}^{E}$ $\in$ $D_{i}^{E}$.

The symmetric cryptography technique ($\Omega_{E}$, $\Lambda_{D}$, $D_{i}$,
$D_{i}^{E}$, $S_{\mathcal{K}}$) comprises three operations: Key Generation,
Encryption, and Decryption, which are defined as:

  1. 1.

Key Generation ($K_{G}^{SY}$ ($\mathcal{CG}$)): This function generates the
secret key ($\mathcal{S}_{k}$) using Eq. (1), which is utilized to encrypt and
decrypt the data.

$\mathcal{S}_{k}=K_{G}^{SY}(\mathcal{CG})\forall\mathcal{S}_{k}\in
S_{\mathcal{K}}$ (1)

  2. 2.

Encryption ($\Omega_{E}$): This function ($\Omega_{E}$ : $\mathcal{D}_{i}$
$\times$ $\mathcal{S}_{k}$ $\rightarrow$ $\mathcal{D}_{i}^{E}$) performs the
encryption task on the actual data $\mathcal{D}_{i}$ using the secret key
($\mathcal{S}_{k}$) and procures the encrypted data $\mathcal{D}_{i}^{E}$, as
given in Eq. (2).

$\mathcal{D}_{i}^{E}=\Omega_{E}(\mathcal{D}_{i},\mathcal{S}_{k})\forall\quad\mathcal{D}_{i}\in
D_{i}\wedge\mathcal{S}_{k}\in S_{\mathcal{K}}\wedge\mathcal{D}_{i}^{E}\in
D_{i}^{E}$ (2)

  3. 3.

Decryption ($\Lambda_{D}$): This function ($\Lambda_{D}$;
$\mathcal{D}_{i}^{E}$ $\times$ $\mathcal{S}_{k}$ $\rightarrow$
$\mathcal{D}_{i}$) takes the encrypted data $\mathcal{D}_{i}^{E}$ and
$\mathcal{S}_{k}$ as input and provides the actual data $\mathcal{D}_{i}$ as
an output using Eq. (3).

$\mathcal{D}_{i}=\Lambda_{D}(\mathcal{D}_{i}^{E},\mathcal{S}_{k})\forall\mathcal{D}_{i}^{E}\in
D_{i}^{E}\wedge\mathcal{S}_{k}\in S_{\mathcal{K}}\wedge\mathcal{D}_{i}\in
D_{i}$ (3)

The asymmetric cryptography technique ($\Omega_{E}$, $\Lambda_{D}$, $D_{i}$,
$D_{i}^{E}$, $P_{\mathcal{B}}$, $P_{\mathcal{V}}$) contains three operations
that are described as:

  1. 1.

Key Generation ($K_{G}^{AS}$ ($\mathcal{CG}$)): This operation generates the
keys $\mathcal{P}_{b}$, and $\mathcal{P}_{v}$ for encryption and decryption,
respectively, using Eq. (4).

$\mathcal{P}_{b},\mathcal{P}_{V}=K_{G}^{AS}(\mathcal{CG})\forall\mathcal{P}_{b}\in
P_{\mathcal{B}}\wedge\mathcal{P}_{v}\in P_{\mathcal{V}}$ (4)

  2. 2.

Encryption ($\Omega_{E}$): This function ($\Omega_{E}$ : $\mathcal{D}_{i}$
$\times$ $\mathcal{P}_{b}$ $\rightarrow$ $\mathcal{D}_{i}^{E}$) encrypt the
actual information $\mathcal{D}_{i}$ with $\mathcal{P}_{b}$ keys and provides
the encrypted data $\mathcal{D}_{i}^{E}$ by applying Eq. (5).

$\mathcal{D}_{i}^{E}=\Omega_{E}(\mathcal{D}_{i},\mathcal{P}_{b})\forall\quad\mathcal{D}_{i}\in
D_{i}\wedge\mathcal{P}_{b}\in P_{\mathcal{B}}\wedge\mathcal{D}_{i}^{E}\in
D_{i}^{E}$ (5)

  3. 3.

Decryption ($\Lambda_{D}$): This function ($\Lambda_{D}$;
$\mathcal{D}_{i}^{E}$ $\times$ $\mathcal{P}_{v}$ $\rightarrow$
$\mathcal{D}_{i}$) takes the encrypted data $\mathcal{D}_{i}^{E}$ and
$\mathcal{P}_{v}$ as input and provides the actual data $\mathcal{D}_{i}$ as
an output using Eq. (6).

$\mathcal{D}_{i}=\Lambda_{D}(\mathcal{D}_{i}^{E},\mathcal{P}_{v})\forall\mathcal{D}_{i}^{E}\in
D_{i}^{E}\wedge\mathcal{P}_{v}\in P_{\mathcal{V}}\wedge\mathcal{D}_{i}\in
D_{i}$ (6)

##  5 Conclusion

Data security is an especially challenging issue in the cloud computing
environment for storage and information sharing. This work focuses on the
security of data that has been outsourced by various data owners. It provides
a succinct and concise overview of the methods used for outsourcing data
security. The system model, data encryption, and decryption processes are
discussed, followed by a comparison of various existing works. A comprehensive
data security method that can secure the data and effectively use it for
machine learning is required in light of the aforementioned evaluation and new
obstacles. This work can be extended by designing a more efficient privacy-
preserving mechanism to secure the data for various owners.

## References

  * [1] Hui Wang.  Privacy-preserving data sharing in cloud computing.  Journal of Computer Science and Technology, 25:401–414, 2010.
  * [2] Mazhar Ali, Revathi Dhamotharan, Eraj Khan, Samee U Khan, Athanasios V Vasilakos, Keqin Li, and Albert Y Zomaya.  Sedasc: secure data sharing in clouds.  IEEE Systems Journal, 11(2):395–404, 2015.
  * [3] Smitha Sundareswaran, Anna Squicciarini, and Dan Lin.  Ensuring distributed accountability for data sharing in the cloud.  IEEE transactions on dependable and secure computing, 9(4):556–568, 2012.
  * [4] Danan Thilakanathan, Shiping Chen, Surya Nepal, and Rafael A Calvo.  Secure data sharing in the cloud.  Security, privacy and trust in cloud systems, pages 45–72, 2014\.
  * [5] Xin Dong, Jiadi Yu, Yuan Luo, Yingying Chen, Guangtao Xue, and Minglu Li.  Achieving an effective, scalable and privacy-preserving data sharing service in cloud computing.  Computers & security, 42:151–164, 2014.
  * [6] Nisha Chaudhary, Vidushi Gupta, Kanav Sandhir, Rishabh Gupta, Sakshi Chhabra, and Ashutosh Kumar Singh.  Privacy preserving ensemble learning classification model for mental healthcare.  In 2022 Seventh International Conference on Parallel, Distributed and Grid Computing (PDGC), pages 513–518. IEEE, 2022.
  * [7] Rishabh Gupta and Ashutosh Kumar Singh.  A privacy-preserving outsourced data model in cloud environment.  arXiv preprint arXiv:2211.13542, 2022.
  * [8] Jian Shen, Huijie Yang, Pandi Vijayakumar, and Neeraj Kumar.  A privacy-preserving and untraceable group data sharing scheme in cloud computing.  IEEE Transactions on Dependable and Secure Computing, 19(4):2198–2210, 2021.
  * [9] Rishabh Gupta, Ishu Gupta, Ashutosh Kumar Singh, Deepika Saxena, and Chung-Nan Lee.  An iot-centric data protection method for preserving security and privacy in cloud.  IEEE Systems Journal, 2022.
  * [10] Rishabh Gupta and Ashutosh Kumar Singh.  A differential privacy-based secure data sharing model in cloud environment.  In 2022 IEEE 6th Conference on Information and Communication Technology (CICT), pages 1–6. IEEE, 2022.
  * [11] Rishabh Gupta, Deepika Saxena, Ishu Gupta, and Ashutosh Kumar Singh.  Differential and triphase adaptive learning-based privacy-preserving model for medical data in cloud environment.  IEEE Networking Letters, 4(4):217–221, 2022.
  * [12] Rishabh Gupta, Ishu Gupta, Deepika Saxena, and Ashutosh Kumar Singh.  A differential approach and deep neural network based data privacy-preserving model in cloud environment.  Journal of Ambient Intelligence and Humanized Computing, pages 1–16, 2022.
  * [13] Mu Yang, Andrea Margheri, Runshan Hu, and Vladimiro Sassone.  Differentially private data sharing in a cloud federation with blockchain.  IEEE Cloud Computing, 5(6):69–79, 2018.
  * [14] Rishabh Gupta and Ashutosh Kumar Singh.  A differential approach for data and classification service-based privacy-preserving machine learning model in cloud environment.  New Generation Computing, 40(3):737–764, 2022.
  * [15] Ashutosh Kumar Singh and Rishabh Gupta.  A privacy-preserving model based on differential approach for sensitive data in cloud environment.  Multimedia Tools and Applications, 81(23):33127–33150, 2022.
  * [16] Rishabh Gupta, Deepika Saxena, Ishu Gupta, Aaisha Makkar, and Ashutosh Kumar Singh.  Quantum machine learning driven malicious user prediction for cloud network communications.  IEEE Networking Letters, 4(4):174–178, 2022.
  * [17] Rishabh Gupta and Ashutosh Kumar Singh.  Privacy-preserving cloud data model based on differential approach.  In 2022 Second International Conference on Power, Control and Computing Technologies (ICPC2T), pages 1–6. IEEE, 2022.
  * [18] Rishabh Gupta and Ashutosh Kumar Singh.  Differential and access policy based privacy-preserving model in cloud environment.  Journal of Web Engineering, pages 609–632, 2022.
  * [19] Jatinder Kumar, Rishabh Gupta, Deepika Saxena, and Ashutosh Kumar Singh.  Power consumption forecast model using ensemble learning for smart grid.  The Journal of Supercomputing, pages 1–22, 2023.
  * [20] Ashutosh Kumar Singh, Sakshi Chhabra, Rishabh Gupta, and Deepika Saxena.  A reliable client detection system during load balancing for multi-tenant cloud environment.  SN Computer Science, 4(1):86, 2022.
  * [21] Smruti Rekha Swain, Deepika Saxena, Jatinder Kumar, Ashutosh Kumar Singh, and Chung-Nan Lee.  An ai-driven intelligent traffic management model for 6g cloud radio access networks.  IEEE Wireless Communications Letters, 2023.
  * [22] Ashutosh Kumar Singh, Smruti Rekha Swain, Deepika Saxena, and Chung-Nan Lee.  A bio-inspired virtual machine placement toward sustainable cloud resource management.  IEEE Systems Journal, 2023.
  * [23] Deepika Saxena, Jitendra Kumar, Ashutosh Kumar Singh, and Stefan Schmid.  Performance analysis of machine learning centered workload prediction models for cloud.  IEEE Transactions on Parallel and Distributed Systems, 34(4):1313–1330, 2023.
  * [24] Deepika Saxena, Ashutosh Kumar Singh, Chung-Nan Lee, and Rajkumar Buyya.  A sustainable and secure load management model for green cloud data centres.  Scientific Reports, 13(1):491, 2023.
  * [25] Deepika Saxena and Ashutosh Kumar Singh.  A high availability management model based on vm significance ranking and resource estimation for cloud applications.  IEEE Transactions on Services Computing, 2022.
  * [26] Ashutosh Kumar Singh and Deepika Saxena.  A cryptography and machine learning based authentication for secure data-sharing in federated cloud services environment.  Journal of Applied Security Research, 17(3):385–412, 2022.
  * [27] Deepika Saxena, Ishu Gupta, Ashutosh Kumar Singh, and Chung-Nan Lee.  A fault tolerant elastic resource management framework toward high availability of cloud services.  IEEE Transactions on Network and Service Management, 19(3):3048–3061, 2022.
  * [28] Deepika Saxena and Ashutosh Kumar Singh.  Ofp-tm: an online vm failure prediction and tolerance model towards high availability of cloud computing environments.  The Journal of Supercomputing, 78(6):8003–8024, 2022.
  * [29] Deepika Saxena and Ashutosh Kumar Singh.  An intelligent traffic entropy learning-based load management model for cloud networks.  IEEE Networking Letters, 4(2):59–63, 2022.
  * [30] Deepika Saxena and Ashutosh Kumar Singh.  Vm failure prediction based intelligent resource management model for cloud environments.  In 2022 Second International Conference on Power, Control and Computing Technologies (ICPC2T), pages 1–6. IEEE, 2022.
  * [31] Deepika Saxena, Ishu Gupta, Jitendra Kumar, Ashutosh Kumar Singh, and Xiaoqing Wen.  A secure and multiobjective virtual machine placement framework for cloud data center.  IEEE Systems Journal, 16(2):3163–3174, 2021.
  * [32] Deepika Saxena and Ashutosh Kumar Singh.  Osc-mc: Online secure communication model for cloud environment.  IEEE Communications Letters, 25(9):2844–2848, 2021.
  * [33] Ashutosh Kumar Singh, Deepika Saxena, Jitendra Kumar, and Vrinda Gupta.  A quantum approach towards the adaptive prediction of cloud workloads.  IEEE Transactions on Parallel and Distributed Systems, 32(12):2893–2905, 2021.
  * [34] Deepika Saxena, Ashutosh Kumar Singh, and Rajkumar Buyya.  Op-mlb: An online vm prediction-based multi-objective load balancing framework for resource management at cloud data center.  IEEE Transactions on Cloud Computing, 10(4):2804–2816, 2021.
  * [35] Deepika Saxena and Ashutosh Kumar Singh.  A proactive autoscaling and energy-efficient vm allocation framework using online multi-resource neural network for cloud data center.  Neurocomputing, 426:248–264, 2021.
  * [36] Deepika Saxena and AK Singh.  Security embedded dynamic resource allocation model for cloud data centre.  Electronics Letters, 56(20):1062–1065, 2020.
  * [37] Jitendra Kumar and Ashutosh Kumar Singh.  Performance evaluation of metaheuristics algorithms for workload prediction in cloud environment.  Applied Soft Computing, 113:107895, 2021.
  * [38] Jitendra Kumar and Ashutosh Kumar Singh.  Decomposition based cloud resource demand prediction using extreme learning machines.  Journal of Network and Systems Management, 28:1775–1793, 2020.
  * [39] Jitendra Kumar, Deepika Saxena, Ashutosh Kumar Singh, and Anand Mohan.  Biphase adaptive learning-based neural network model for cloud datacenter workload forecasting.  Soft Computing, 24:14593–14610, 2020.
  * [40] Jitendra Kumar and Ashutosh Kumar Singh.  Adaptive learning based prediction framework for cloud datacenter networks’ workload anticipation.  Journal of Information Science & Engineering, 36(5), 2020.
  * [41] Jatinder Kumar and Ashutosh Kumar Singh.  Security and privacy-preservation of iot data in cloud-fog computing environment.  arXiv preprint arXiv:2212.00321, 2022.
  * [42] Ashutosh Kumar Singh and Jatinder Kumar.  A privacy-preserving multidimensional data aggregation scheme with secure query processing for smart grid.  The Journal of Supercomputing, 79(4):3750–3770, 2023.
  * [43] Ashutosh Kumar Singh and Ishu Gupta.  Online information leaker identification scheme for secure data sharing.  Multimedia Tools and Applications, 79:31165–31182, 2020.
  * [44] Ishu Gupta and Ashutosh Kumar Singh.  Seli: statistical evaluation based leaker identification stochastic scheme for secure data sharing.  IET Communications, 14(20):3607–3618, 2020.
  * [45] Ishu Gupta and Ashutosh Kumar Singh.  Dynamic threshold based information leaker identification scheme.  Information Processing Letters, 147:69–73, 2019.
  * [46] Niharika Singh and Ashutosh Kumar Singh.  Sql-injection vulnerabilities resolving using valid security tool in cloud.  Pertanika Journal of Science & Technology, 27(1), 2019.
  * [47] Niharika Singh and Ashutosh Kumar Singh.  Data privacy protection mechanisms in cloud.  Data Science and Engineering, 3(1):24–39, 2018.
  * [48] Ashutosh Kumar Singh and Jitendra Kumar.  Secure and energy aware load balancing framework for cloud data centre networks.  Electronics Letters, 55(9):540–541, 2019.
  * [49] Jitendra Kumar, Rimsha Goomer, and Ashutosh Kumar Singh.  Long short term memory recurrent neural network (lstm-rnn) based workload forecasting model for cloud datacenters.  Procedia Computer Science, 125:676–682, 2018.
  * [50] Sakshi Chhabra and Ashutosh Kumar Singh.  A probabilistic model for finding an optimal host framework and load distribution in cloud environment.  Procedia Computer Science, 125:683–690, 2018.
  * [51] Aaisha Makkar, Tae Woo Kim, Ashutosh Kumar Singh, Jungho Kang, and Jong Hyuk Park.  Secureiiot environment: Federated learning empowered approach for securing iiot from data breach.  IEEE Transactions on Industrial Informatics, 18(9):6406–6414, 2022\.
  * [52] Ashutosh Kumar Singh, Smruti Rekha Swain, and Chung Nan Lee.  A metaheuristic virtual machine placement framework toward power efficiency of sustainable cloud environment.  Soft Computing, 27(7):3817–3828, 2023.
  * [53] Archana Yadav, Shivam Kushwaha, Jyoti Gupta, Deepika Saxena, and Ashutosh Kumar Singh.  A survey of the workload forecasting methods in cloud computing.  In Proceedings of 3rd International Conference on Machine Learning, Advances in Computing, Renewable Energy and Communication: MARC 2021, pages 539–547. Springer, 2022.
  * [54] Ashutosh Kumar Singh and Jatinder Kumar.  A secure and privacy-preserving data aggregation and classification model for smart grid.  Multimedia Tools and Applications, pages 1–19, 2023.
  * [55] Utkarsh Saini, Prachee Atmapoojya, Rohit Patidar, Rishabh Gupta, Sakshi Chhabra, and Ashutosh Kumar Singh.  Data privacy preservation model using noise concept in cloud.  International Journal of Engineering Research & Technology, 11(3):326–329, 2022.
  * [56] Anjali Tripathi, Upasana Singh, Garima Bansal, Rishabh Gupta, and Ashutosh Kumar Singh.  A review on emotion detection and classification using speech.  In Proceedings of the international conference on innovative computing & communications (ICICC), 2020.
  * [57] Aman Singh Chauhan, Dikshika Rani, Akash Kumar, Rishabh Gupta, and Ashutosh Kumar Singh.  A survey on privacy-preserving outsourced data on cloud with multiple data providers.  In Proceedings of the International Conference on Innovative Computing & Communications (ICICC), 2020.
  * [58] Anjali Tripathi, Upasana Singh, Garima Bansal, Rishabh Gupta, and Ashutosh Kumar Singh.  Hedcm: Human emotions detection and classification model from speech using cnn.  In Workshop on Advances in Computational Intelligence at ISIC, 2021\.
  * [59] Deepika Deepika, Rajnesh Malik, Saurabh Kumar, Rishabh Gupta, and Ashutosh Kumar Singh.  A review on data privacy using attribute-based encryption.  In Proceedings of the International Conference on Innovative Computing & Communications (ICICC), 2020.
  * [60] Prachee Atmapoojya, Utkarsh Saini, Rohit Patidar, Rishabh Gupta, and Ashutosh Kumar Singh.  Differential privacy based preserving data on cloud environment.  International Journal of Engineering Research & Technology, 10(5):780–785, 2021.
  * [61] Badal Pardhan, Bhagwan Singh, Abhishek Bhoria, Rishabh Gupta, and Ashutosh Kumar Singh.  A comparative study on cipher text policy attribute based encryption schemes.  International Journal of Engineering Research & Technology, 10(5):786–790, 2021.
  * [62] Dikshika Rani, Akash Kumar, Aman Singh Chauhan, Rishabh Gupta, and Ashutosh Kumar Singh.  Privacy-preserving deep learning on cloud with multiple data providers.  International Research Journal of Modernization in Engineering Technology and Science, 3(2):46–54, 2021.
  * [63] Rishabh Gupta and Ashutosh Kumar Singh.  A privacy-preserving model for cloud data storage through fog computing.  International Journal of Computer Aided Engineering and Technology, 17(3):348–359, 2022.
  * [64] Deepika Saxena, Rishabh Gupta, and Ashutosh Kumar Singh.  A survey and comparative study on multi-cloud architectures: emerging issues and challenges for cloud federation.  arXiv preprint arXiv:2108.12831, 2021.
  * [65] Rishabh Gupta, Deepika Saxena, and Ashutosh Kumar Singh.  Data security and privacy in cloud computing: concepts and emerging trends.  arXiv preprint arXiv:2108.09508, 2021.
  * [66] Sakshi Chhabra and Ashutosh Kumar Singh.  A comprehensive vision on cloud computing environment: Emerging challenges and future research directions.  arXiv preprint arXiv:2207.07955, 2022.
  * [67] Sakshi Chhabra and Ashutosh Kumar Singh.  Dynamic data leakage detection model based approach for mapreduce computational security in cloud.  In 2016 Fifth International Conference on Eco-friendly Computing and Communication Systems (ICECCS), pages 13–19. IEEE, 2016.
  * [68] Smruti Rekha Swain, Ashutosh Kumar Singh, and Chung Nan Lee.  Efficient resource management in cloud environment.  arXiv preprint arXiv:2207.12085, 2022.
  * [69] Ishu Gupta, Rishabh Gupta, Ashutosh Kumar Singh, and Rajkumar Buyya.  Mlpam: A machine learning and probabilistic analysis based model for preserving security and privacy in cloud environment.  IEEE Systems Journal, 15(3):4248–4259, 2020.
  * [70] Niharika Singh, Jitendra Kumar, Ashutosh Kumar Singh, and Anand Mohan.  Privacy-preserving multi-keyword hybrid search over encrypted data in cloud.  Journal of Ambient Intelligence and Humanized Computing, pages 1–14, 2022.
  * [71] Sakshi Chhabra and Ashutosh Kumar Singh.  Secure and energy efficient dynamic hierarchical load balancing framework for cloud data centers.  Multimedia Tools and Applications, pages 1–14, 2023.
  * [72] Sakshi Chhabra and Ashutosh Kumar Singh.  Security enhancement in cloud environment using secure secret key sharing.  Journal of Communications Software and Systems, 16(4):296–307, 2020\.
  * [73] Ehab Zaghloul, Kai Zhou, and Jian Ren.  P-mod: Secure privilege-based multilevel organizational data-sharing in cloud computing.  IEEE Transactions on Big Data, 6(4):804–815, 2019.
  * [74] Jianfei Sun, Guowen Xu, Tianwei Zhang, Hu Xiong, Hongwei Li, and Robert Deng.  Share your data carefree: An efficient, scalable and privacy-preserving data sharing service in cloud computing.  IEEE Transactions on Cloud Computing, 2021.
  * [75] Mazhar Ali, Samee U Khan, and Athanasios V Vasilakos.  Security in cloud computing: Opportunities and challenges.  Information sciences, 305:357–383, 2015.
  * [76] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor.  Federated learning with differential privacy: Algorithms and performance analysis.  IEEE Transactions on Information Forensics and Security, 15:3454–3469, 2020.
  * [77] Maoguo Gong, Jialun Feng, and Yu Xie.  Privacy-enhanced multi-party deep learning.  Neural Networks, 121:484–496, 2020.
  * [78] Mahawaga Arachchige Pathum Chamikara, Peter Bertok, Ibrahim Khalil, Dongxi Liu, and Seyit Camtepe.  Privacy preserving face recognition utilizing differential privacy.  Computers & Security, 97:101951, 2020.
  * [79] Weibei Fan, Jing He, Mengjiao Guo, Peng Li, Zhijie Han, and Ruchuan Wang.  Privacy preserving classification on local differential privacy in data centers.  Journal of Parallel and Distributed Computing, 135:70–82, 2020\.
  * [80] Xiang Wu, Yongting Zhang, Minyu Shi, Pei Li, Ruirui Li, and Neal N Xiong.  An adaptive federated learning scheme with differential privacy preserving.  Future Generation Computer Systems, 127:362–372, 2022.
  * [81] Yufeng Wang, Minjie Huang, Qun Jin, and Jianhua Ma.  Dp3: A differential privacy-based privacy-preserving indoor localization mechanism.  IEEE Communications Letters, 22(12):2547–2550, 2018.
  * [82] Chuan Ma, Long Yuan, Li Han, Ming Ding, Raghav Bhaskar, and Jun Li.  Data level privacy preserving: A stochastic perturbation approach based on differential privacy.  IEEE Transactions on Knowledge and Data Engineering, 2021.
  * [83] Miao Du, Kun Wang, Zhuoqun Xia, and Yan Zhang.  Differential privacy preserving of training model in wireless big data with edge computing.  IEEE transactions on big data, 6(2):283–295, 2018.
  * [84] Lin Zhang, Yan Liu, Ruchuan Wang, Xiong Fu, and Qiaomin Lin.  Efficient privacy-preserving classification construction model with differential privacy technology.  Journal of Systems Engineering and Electronics, 28(1):170–178, 2017\.
  * [85] Ting Bao, Lei Xu, Liehuang Zhu, Lihong Wang, Ruiguang Li, and Tielei Li.  Privacy-preserving collaborative filtering algorithm based on local differential privacy.  China Communications, 18(11):42–60, 2021.
  * [86] Hyunbum Kim, Jalel Ben-Othman, and Lynda Mokdad.  Udipp: A framework for differential privacy preserving movements of unmanned aerial vehicles in smart cities.  IEEE Transactions on Vehicular Technology, 68(4):3933–3943, 2019\.
  * [87] Ningbo Wu, Changgen Peng, and Kun Niu.  A privacy-preserving game model for local differential privacy by using information-theoretic approach.  IEEE Access, 8:216741–216751, 2020.
  * [88] Subhadeep Sarkar, Subarna Chatterjee, and Sudip Misra.  Assessment of the suitability of fog computing in the context of internet of things.  IEEE Transactions on Cloud Computing, 6(1):46–59, 2015.
  * [89] Tian Wang, Jiyuan Zhou, Xinlei Chen, Guojun Wang, Anfeng Liu, and Yang Liu.  A three-layer privacy preserving cloud storage scheme based on computational intelligence in fog computing.  IEEE Transactions on Emerging Topics in Computational Intelligence, 2(1):3–12, 2018.
  * [90] Tran Thi Phuong et al.  Privacy-preserving deep learning via weight transmission.  IEEE Transactions on Information Forensics and Security, 14(11):3003–3015, 2019.
  * [91] Wen-Jie Liu, Chun-Tang Li, Yu Zheng, Yong Xu, and Yin-Song Xu.  Quantum privacy-preserving price e-negotiation.  International Journal of Theoretical Physics, 58(10):3259–3270, 2019.
  * [92] Y. Chai, Y. Zhan, B. Wang, Y. Ping, and Z. Zhang.  Improvement on a privacy-preserving outsourced classification protocol over encrypted data.  Wireless Networks, 26(6):4363–4374, 2020.