#  AutoDropout: Learning Dropout Patterns to Regularize Deep Networks

Hieu Pham1,2 and Quoc V. Le1

###### Abstract

Neural networks are often over-parameterized and hence benefit from aggressive
regularization. Conventional regularization methods, such as Dropout
(Srivastava et al. 2014) or weight decay, do not leverage the structures of
the network’s inputs and hidden states. As a result, these methods are less
effective than recent methods that leverage the structures, such as
SpatialDropout (Tompson et al. 2020) and DropBlock (Ghiasi, Lin, and Le 2018),
which randomly drop the values at certain contiguous areas in the hidden
states and setting them to zero. Although the locations of dropout areas are
random, the patterns of SpatialDropout and DropBlock are manually designed and
fixed. Here we propose _AutoDropout_ , which automates the process of
designing dropout patterns. In our method, a controller learns to generate a
dropout pattern at every channel and layer of a target network, such as a
ConvNet or a Transformer. The target network is then trained with the dropout
pattern, and its resulting validation performance is used as a signal for the
controller to learn from. We show that this method works well for both image
recognition on CIFAR-10 and ImageNet, as well as language modeling on Penn
Treebank and WikiText-2. The learned dropout patterns also transfers to
different tasks and datasets, such as from language model on Penn Treebank to
Engligh-French translation on WMT 2014. Our code will be available.111Code
repository: https://github.com/google-research/google-
research/tree/master/auto˙dropout.

## Introduction

Modern neural networks are often over-parameterized (Nakkiran et al. 2020) and
thus require proper regularization to avoid overfitting. A common
regularization method is Dropout (Srivastava et al. 2014), which randomly
selects neurons from some intermediate layers of a network and replaces the
values of these neurons with zero. In other words, we drop these neurons out
of the current step of training. More recent studies show that imposing
certain structures to the dropped neurons can lead to significant improvements
over dropout neurons uniformly at random (Huang et al. 2016; Tompson et al.
2020; Ghiasi, Lin, and Le 2018; Gal and Ghahramani 2016b; Zoph et al. 2018;
Zaremba, Sutskever, and Vinyals 2014; Vaswani et al. 2017). In practice,
however, the dropout patterns are adapted to become different for different
applications.

For example, in the text domain, Zaremba, Sutskever, and Vinyals (2014)
suggest that, for a multi-layered LSTM (Hochreiter and Schmidhuber 1997), it
is better to only drop the neurons in the vertical connections than to drop
the neurons everywhere. Gal and Ghahramani (2016b) later propose Variational
Dropout, where they drop neurons everywhere in the network but share a dropout
pattern along the temporal dimension. Both methods, however, are not used in
the recent Transformer architecture, which only uses vanilla Dropout. The
differences in how LSTM and Transformer implement Dropout suggest that dropout
patterns need to be tailored to different model architectures in NLP.

In the image domain, vanilla Dropout is often only applied to the fully-
connected layers within a ConvNet (He et al. 2016; Zagoruyko and Komodakis
2016; Han, Jiwhan, and Kim 2017; Hu et al. 2018; Szegedy et al. 2016). Other
convolutional layers often require the dropout neurons to have particular
structures. For example, stochastic depth (Huang et al. 2016) drops the whole
residual branch in residual networks, and DropPath (Zoph et al. 2018) drops a
whole branch in multi-branched convolutional cells. Ghiasi, Lin, and Le (2018)
propose DropBlock which drop contiguous squares of neurons in the
convolutional layers. While DropBlock works well on ResNet-50 and AmoebaNet
(Real et al. 2018), it is not proven to be successful in more recent
architectures such as EfficientNet (Tan, Pang, and Le 2020) and EfficientDet
(Tan, Pang, and Le 2020; Zoph et al. 2019). Again, the differences in the way
ConvNet architectures use dropout patterns suggest that they also need to be
specialized to architectures.

By studying the dropout patterns from previous works, we observe that these
patterns are difficult to design and need to be specialized for each model
architecture, task, and domain. In this work, we address this difficulty by
learning a specialized pattern for each model architecture, task, and domain.
To this end, we propose _AutoDropout_ which automates the process of designing
specialized dropout patterns. The main contribution of AutoDropout is a novel
search space of structured dropout patterns. In the search space we design,
one can find a suitable for each model architecture and task. Our search space
generalizes many existing dropout patterns (Srivastava et al. 2014; Gal and
Ghahramani 2016a, b; Huang et al. 2016; Ghiasi, Lin, and Le 2018). For
example, Figure 1 shows a dropout pattern from our search space. The pattern
is generated by tiling a contiguous area and then transforming it
geometrically. The resulting pattern is applied to a convolutional output
channel, which is a common building block of image recognition models.

Figure 1: An example dropout pattern from our search space applied to an
convolutional output channel. Left: the input image. Middle: DropBlock sets
contiguous square blocks in the channel to zero. Right: a dropout pattern in
the search space of AutoDropout. More patterns in our noise space are
described in Section Methods.

Our implementation of AutoDropout has a controller that is trained by
reinforcement learning (RL). The reward for the RL is the validation
performance of the dropout pattern on a target network on a dataset of
interest. We design a distributed RL-based search algorithm, which allows us
to maximally leverage all machines available on an arbitrary cluster of
computational nodes.222We will release the datasets consisting of the dropout
patterns that our search algorithm has sampled and run. Like the similar
datasets collected from benchmarking various model architectures (Ying et al.
2019; Dong and Yang 2019).

Our experiments show that AutoDropout can find dropout patterns that
significantly improve commonly-used ConvNet and Transformer architectures. On
ImageNet, AutoDropout improves the top-1 accuracy of ResNet-50 from 76.5% to
78.7%, and EfficientNet-B7 from 84.1% to 84.7%. In the semi-supervised setting
with CIFAR-10-4000, AutoDropout also improves the accuracy of Wide-ResNet-28-2
from 94.9% to 95.8%. For language modeling, AutoDropout reduces the perplexity
of Transformer-XL (Dai et al. 2019) on Penn Treebank from 56.0 to 54.9.

Additionally, when transferred to German-to-English translation on the IWSLT
14 dataset, the dropout pattern found by AutoDropout improves Transformer’s
BLEU score from 34.4 to 35.8, which is a new state-of-the-art on this dataset.
On English-to-French translation with WMT 2014, the transferred dropout
pattern also yields an improvement of 1.9 BLEU scores over the Transformer
model with vanilla Dropout.

Although the search cost of AutoDropout can be high, a simple use case of
AutoDropout is to drop our found patterns into existing pipelines in the same
way that AutoAugment policies (Cubuk et al. 2019a) were used to improve state-
of-the-art models.

#### Related works.

Our work has the same philosophy with existing neural architecture search and
AutoAugment lines of research (Pham et al. 2018; Liu, Simonyan, and Yang 2019;
Zoph and Le 2017; Zoph et al. 2018; Bello et al. 2017b; Cubuk et al. 2019a;
Park et al. 2019; Lim et al. 2019; Tan and Le 2019; Real et al. 2018;
Mirhoseini et al. 2017; Bello et al. 2017a; Cubuk et al. 2019b; Real et al.
2017; Cai, Zhu, and Han 2019; Cai et al. 2019). We create a search space
comprising the possible decisions and then use RL to search for the best
decision.

More specifically, AutoDropout can also be viewed as data augmentation in the
networks’ hidden states. We generalize the successful approaches of searching
for data augmentation (Park et al. 2019; Cubuk et al. 2019a, b; Lim et al.
2019) and apply them to the hidden states of ConvNets and Transformer
networks. Unlike data augmentations, which are domain-specific, our dropout
patterns for the hidden states have the same design philosophy on ConvNets for
image recognition models and Transformer for text understanding models. CutMix
(Yun et al. 2019) and ManifoldMixup (Verma et al. 2019a) also apply successful
data augmentation techniques such as CutOut (DeVries and Taylor 2017) and
Mixup (Zhang et al. 2018) into the hidden states. Implicit Semantic Data
Augmentation (ISDA; Wang et al. (2019)) approximate a Gaussian distribution of
ConvNets’ hidden states using the moving averages of their mean and standard
deviations to generate more training examples.

## Methods

#### Representing dropout patterns.

We represent the dropout patterns in our search space using elementwise
multiplicative masks as adopted by many previous works (Srivastava et al.
2014; Gal and Ghahramani 2016a, b; Huang et al. 2016; Zoph et al. 2018;
Ghiasi, Lin, and Le 2018; Vaswani et al. 2017). To bridge the gap between
training, when the mask is used, and inference, when the mask is not used, we
scale the values of the non-drop neurons properly during training.
Specifically, to apply a dropout pattern to a layer $h$ of a neural network,
we randomly generate a binary mask $m$ of the same shape with $h$. We then
scale the values in the mask $m$, and replace $h$ with:

$\displaystyle\text{Drop}(h,m)\mathrel{\stackrel{{\scriptstyle\makebox[0.0pt]{\mbox{\scalebox{0.5}{$\triangle$}}}}}{{=}}}h\otimes\left(\frac{\text{Size}(m)}{\text{Sum}(m)}\cdot
m\right)$ (1)

#### Dimensional notations.

In modern deep learning frameworks (Abadi et al. 2016; Paszke et al. 2019),
intermediate layers are represented as high dimensional tensors. We denote the
general shape of a tensor as $(N,d_{1},d_{2},...,d_{k},C)$, where $N$ is the
batch dimension, $C$ is the feature dimension, and $d_{1}$, $d_{2}$, …,
$d_{k}$ are the spatiotemporal dimensions. For instance, a layer in a typical
ConvNet has a shape of $(N,H,W,C)$ where $H$ and $W$ are the layer’s height
and width; while a Transformer layer has the output of shape $(N,T,C)$ where
$T$ is the temporal dimension which represents the number of tokens.

Our method is general and works well for both ConvNets and Transformers where
the spatiotemporal dimensions are different from each other. In the following,
we will first discuss the search space for ConvNets, and then discuss how we
generalize it to Transformers.

### Search Space for Dropout Patterns in ConvNets

Figure 2: Example of the basic patterns in our search space. A dropout
pattern, represented in black and gray, is applied to a grid of while cells
representing the tensors. The neuron corresponding to the gray cells are
retained.

#### Basic patterns.

The basic pattern in our search space is a contiguous rectangle. The rectangle
is then tiled to produce a dropout pattern. For ConvNets, the hyper-parameters
that define the basic rectangle are two sizes height and width. The hyper-
parameters that define the tiling are the stride and the number of repeats.
Figure 2 shows an example. For $C$ channels, we can either sample $C$
independent dropout patterns, or we can sample only one dropout pattern and
then share it along the feature dimension.

#### Geometric transformations.

In addition to tiling the rectangles, we introduce two geometric
transformations into our search space: rotating about the spatial center, and
shearing along each spatial dimension. When the transformations result in
fractional coordinates, we round them to the nearest integers.

#### Where to apply the dropout pattern.

Once we have a dropout pattern, there is a decision about where we should
apply it to. Here, we apply the dropout pattern to the output of batch
normalization layers because we empirically observe that applying the pattern
elsewhere in the network often leads to unstable training during our search
process. If there is a residual connection in the ConvNet to regularize, then
there is a choice of whether we should apply the dropout pattern to the
residual branch as well. We leave this decision to the controller. Appendix
Details on the Search Spaces for ConvNets visualizes where the noise masks are
applied in some network architectures in our experiments in Figure 7, and
specifies more details about our ConvNet search space.

### Controller Model and Search Algorithms

Figure 3: Our controller is a Transformer network. The network generates the
tokens to describe the configurations of the dropout pattern. The tokens are
generated like words in a language model. For every layer in a ConvNet, a
group of 8 tokens need to be made to create a dropout pattern. These 8 tokens
are generated sequentially. In the figure above, size, stride, and repeat
indicate the size and the tiling of the pattern; rotate, shear_x, and shear_y
specify the geometric transformations of the pattern; share_c is a binary
deciding whether a pattern is applied to all $C$ channels; and residual is a
binary deciding whether the pattern is applied to the residual branch as well.
If we need $L$ dropout patterns, the controller will generate $8L$ decisions.

#### Model Architecture and Search Algorithm.

We parameterize our controller with a Transformer network, as illustrated in
Figure 3. We train the parameters $\theta$ of our controller using the
REINFORCE algorithm with a standard moving average baseline (Williams 1992).
That is, we optimize $\theta$ to minimize the objective via Monte Carlo
gradient estimation:

$\displaystyle J(\theta)$ $\displaystyle=\mathbb{E}_{r\sim
P({\mathbf{r}};\theta)}[\text{Perf}(r)]$ (2)
$\displaystyle\widehat{\nabla}_{\theta}J$
$\displaystyle=\frac{1}{M}\sum_{i=1}^{M}(\text{Perf}(r_{i})-b)\cdot\nabla_{\theta}\log{P(r_{i};\theta)}$

Here, $b$ is the moving average baseline, $M$ is the empirical batch size
which we set to 16, and $\text{Perf}(r)$ is measured by training a target
network with a dropout pattern $r$ on a designated proxy task’s validation
set. We find it important to tailor the proxy task according to the actual
downstream task. We discuss our proxy tasks in detailed in Experiments.

#### Improving Parallelism.

Previous works in architecture search and data augmentation search (Zoph and
Le 2017; Zoph et al. 2018; Bello et al. 2017b; Cubuk et al. 2019a; Park et al.
2019; Tan and Le 2019) typically wait for minibatches of $M$ dropout patterns
to finish training before making every update on $\theta$. Since each child
model can take significantly long to train, and is subjected to multiple
failures, such as jobs being descheduled on a shared cluster, waiting for $M$
dropout patterns to finish can cause an unnecessary bottleneck.

To alleviate this bottleneck, we propose a modification. Specifically, in a
shared environment, the number of available machines will vary over time.
Sometimes, the number of machines will be lower than $M$. In this case, we
will have to use this low number of machines to slowly compute the rewards for
$M$ configurations. However, sometimes the number of machines will be much
higher than $M$. In such case, we want to generate many more than $M$ jobs to
take advantage of the available resources. But even in such case, for training
stability, we only use a minibatch of $M$ configurations, causing the other
trained configurations to have stale gradient. To adjust for the staleness of
their gradients, we need to reweigh the gradient properly as explained later.

Our implementation maintains two queues: a queue $q_{\text{unfinished}}$ of
unfinished jobs and a queue $q_{\text{finished}}$ of finished jobs. Whenever
the $q_{\text{unfinished}}$ contains less than its capacity $C$, the
controller generates
$n=C-\mathopen{}\left|q_{\text{unfinished}}\mathopen{}\right|$ new dropout
patterns $r_{1}$, $r_{2}$, …, $r_{n}$ and fills up $q_{\text{unfinished}}$
with the pairs $(r_{i},P(r_{i};\theta_{i}))$, where $\theta_{i}$ is the value
of the controller’s parameters at the time $r_{i}$ is sampled.

On the other hand, whenever a dropout pattern $r$ finishes training, the
controller dequeues $(r,\text{Perf}(r))$ from $q_{\text{unfinished}}$ and
moves it into $q_{\text{finished}}$. Whenever the capacity
$\mathopen{}\left|q_{\text{finished}}\mathopen{}\right|$ reaches $M$, $M$
configurations along with their accuracy are dequeued from
$q_{\text{finished}}$ to perform an update on $\theta$. The caveat of this
approach is that due to many dropout patterns being executed in parallel, the
controller parameter $\theta$ when we update the controller with a
configuration $r_{i}$ can be different from the $\theta_{i}$ when $r_{i}$ was
generated. To account for this difference, we resort to importance sampling,
which allows us to write the training objective $J(\theta)$ as follows:

$\displaystyle\nabla_{\theta}J(\theta)=\nabla_{\theta}\mathbb{E}_{r\sim
P({\mathbf{r}};\theta)}[\text{Perf}(r)]$ (3)
$\displaystyle\approx\frac{1}{M}\sum_{i=1}^{M}\text{Perf}(r_{i})\cdot\frac{P(r_{i};\theta)}{P(r_{i};\theta_{i})}\cdot\nabla_{\theta}\log{P(r_{i};\theta)}$

Implementing this update rule simply requires scaling the gradient
$\nabla_{\theta}\log{P(r_{i};\theta)}$ by the ratio of the two probabilities
as shown in Equation 3. In our design, the only training bottleneck is the
number of workers that can be run in parallel. In practice, distributed search
procedures like ours typically run on a shared cluster, where the number of
available workers varies instantly. Our design obviates the need to reserve
all $C$ workers throughout the search procedure and allows us to use a large
value of $C$ to achieve better parallelism when more workers are available.

### Search Space for Dropout Patterns in Transformers

Figure 4: An example of our controller generating ad dropout pattern for a
self-attention operation. Top: the controller’s outputs. The tokens have the
following meanings: size=2 and stride1 means that the dropout pattern affects
two consecutive tokens, then skips two token, then affects the next two
consecutive tokens, and so on; share_t=True means that every block of two
consecutive tokens that the dropout pattern affects shares the same dropout
mask; share_c=False means that each of the $C$ feature dimensions of the
$(N,T,C)$ tensor has its own independent mask. Bottom: The dropout pattern
that the controller’s outputs realize on the self-attention operation. The
values in the red cells are set to zero, while the values in the green are
kept intact.

#### Basic patterns.

Intermediate layers in Transformer models typically have three dimensions
$(N,T,C)$, where $N$ and $C$ are the batch dimension and the channel
dimension, similar to those of ConvNets, and $T$ is the number of tokens, such
as words or sub-word units. The dropout pattern for this dimension $T$ is
realized by generating four hyper-parameters: size, stride, share_t, and
share_c. size indicates how many tokens does a pattern affects; stride
indicates the number of tokens to be skipped by the pattern; share_t is a
binary deciding whether all the tokens covered by size are set to zero using
the same noise mask or independent noise masks; and share_c is a binary
deciding whether a the dropout pattern shared along the channel dimension $C$.
Once the values of size, stride, share_t, and share_c are decided, at each
training step, we sample the starting position to apply the resulting dropout
pattern. We repeat the pattern until the end of the sequence, following size
and stride. Figure 4 provides an illustration of a dropout pattern that our
controller samples from our search space, and how the pattern is applied to a
sequence of words.

Many successful regularization patterns for text processing models are
included in our basic patterns. For instance, WordDropout (Sennrich, Haddow,
and Birch 2016) can be realized from our patterns by setting share_c=True,
while Variational Dropout (Gal and Ghahramani 2016b) can be realized by
setting share_t=True and setting $size$ to the $T$, number of tokens in the
sequence.

#### Where to apply the dropout pattern.

Unlike the case for image recognition models, we find that the dropout
patterns in our search space can be flexibly applied at multiple sub-layers
within a Transformer layer (e.g., on the query, key, value, softmax, output
projection, and residual). As a result, we apply one independent dropout
pattern to each of them. Figure 8 in our Appendix Details on the Search Spaces
for Transformer specifies all the possible places to apply the dropout
patterns in our Transformer model. We will use this pattern at all Transformer
layers in the Transformer network. In our implementation, size is overloaded,
and if it has the value of zero, the dropout pattern is not applied at the
corresponding.

## Experiments

In the following sections, we will apply AutoDropout to both ConvNets and
Transformers. For ConvNets, we first consider Supervised Image Classification
and then we consider Semi-supervised Image Classification. For Transformer, we
consider Language Model and Machine Translation applications. Finally, we
compare our search method against random search.

### Supervised Image Classification with ConvNets

We first evaluate AutoDropout on two standard benchmarks for image
classification: CIFAR-10 (Krizhevsky 2009) and ImageNet (Russakovsky et al.
2015). For CIFAR-10, we use Wide ResNet 28-10 (WRN-28-10; Zagoruyko and
Komodakis (2016)) because it is a common baseline on this dataset. For
ImageNet, we consider ResNet-50 (He et al. 2016) because it is a common
architecture for ImageNet. We also consider EfficientNet (Tan and Le 2019)
since it is closer to the state-of-the-art than ResNet. For each benchmark and
model, we first use AutoDropout to search for a good dropout pattern on a
proxy task, and then scale up the best found pattern to apply to the final
task.

#### Search Configurations and Proxy Tasks.

We refer readers to Appendix Hyper-parameters of Experiments for detailed
training information of our controller. Here, we focus on the proxy tasks that
we design to reduce the AutoDropout’s search time. We scale down the final
architecture and reduce the amount of data for the final task as follows.

For CIFAR-10, we search with a WRN-28-2 on the entire dataset, reserving 10%
of the original training set for validation. For ImageNet, we scale down
ResNet-50 and EfficientNet-B0 so that each of their layers has half the number
of channels as the original models. We use 80,000 examples for training and
5,000 examples for validation. The controller’s reward is the accuracy of the
dropout pattern on the validation set. We train each dropout pattern on
CIFAR-10 for 32,000 steps, and train each pattern on ImageNet for 16,000
steps. Under these settings, each dropout pattern trains in approximately 40
minutes on both datasets. Our search explores 16,384 patterns for each task.

#### Baselines.

For WRN-28-10 and ResNet-50, we compare AutoDropout against DropBlock (Ghiasi,
Lin, and Le 2018), since DropBlock has been well-tuned for these models. For
EfficientNet, we compare AutoDropout with Stochastic Depth (Huang et al. 2016)
since it is the default noise-based regularization scheme of this
architecture. We implement these baselines in our environment for fair
comparison. Note that large EfficientNet models, such as B3, B5, B7 in our
experiments, enlarge the spatial dimensions of the input images. For these
models, we proportionally scale up the sizes and strides of the masks found by
AutoDropout on these models. Training details of all models are in our
Appendix Hyper-parameters of Experiments.

Regularization Methods | CIFAR-10 | ImageNet
---|---|---
WRN-28-10 | ResNet-50 | EfficientNet-B0 | EfficientNet-B3 | EfficientNet-B5 | EfficientNet-B7
None | 96.1 $\pm$ 0.12 | 76.5 / 93.4 | 76.2 / 92.9 | $-$ | $-$ | $-$
DropBlock (Ghiasi, Lin, and Le 2018) | 96.2 $\pm$ 0.07 | 78.3 / 94.3 | 76.3 / 92.8 | $-$ | $-$ | $-$
Stochastic Depth (Huang et al. 2016) | 96.2 $\pm$ 0.07 | 77.5 / 93.7 | 76.8 / 93.1 | 80.2 / 95.0 | 82.5 / 96.2 | 84.1 / 96.9
AutoDropout | 96.8 $\pm$ 0.09 | 78.7 / 94.3 | 77.5 / 93.8 | 80.9 / 95.6 | 83.1 / 96.5 | 84.7 / 97.1

Table 1: Performance of AutoDropout and the baselines on supervised image
classification (higher is better). This is a control experiment and all models
are implemented by us.

#### Results.

Figure 1 reports the results of our control experiments on ResNets and
EfficientNet. From Table 1, it can be seen that AutoDropout outperforms
DropBlock by 0.6% accuracy on CIFAR-10 with WRN-28-10, which corresponds to a
16% error reduction. Notably, on CIFAR-10 with WRN-28-10, DropBlock does not
yield significant improvements compared to not using regularization at all,
suggesting that the intuition on blocking contiguous regions is not
sufficient. On ImageNet, AutoDropout improves the top-1 accuracy of ResNet-50
on ImageNet by 0.4% compared to DropBlock. AutoDropout improves the accuracy
of all EfficientNet models by a margin of 0.7% on average. This is larger than
the improvement of 0.5% that DropBlock delivers on AmoebaNet (Ghiasi, Lin, and
Le 2018; Real et al. 2018), even though EfficientNet baselines have higher
accuracy than AmoebaNet.

#### Pushing the limits of ResNets.

In the above experiments, we wanted to perform fair comparisons against other
baselines, and did not combine AutoDropout with any data augmentation methods.
Here, we aim to push the limits of WRN-28-10 and ResNet-50 by combining
AutoDropout and other data augmentation methods. As such, we apply the pattern
found by AutoDropout on CIFAR-10 with RandAugment (Cubuk et al. 2019b) to
WRN-28-10 and achieve 97.9% accuracy. We also apply the pattern found by
AutoDropout on ImageNet with RandAugment and achieve 80.3% top-1 accuracy with
ResNet-50 on ImageNet. These results outperform existing state-of-the-art
results on these datasets with the same model architectures, as presented in
Table 2.

Table 2 also shows that AutoDropout is the only method that improves the
performance on both CIFAR-10 with WRN-28-10 and ImageNet with ResNet-50. Among
other baselines, Manifold Mixup (Verma et al. 2019a) improves the CIFAR-10
accuracy but has a weak accuracy on ImageNet. Meanwhile, CutMix (Yun et al.
2019) achieves good accuracy on ImageNet but worsens CIFAR-10 accuracy. These
observations suggest that regularization methods that are validated for a
certain architecture and dataset might not deliver as strong performance for
another architecture and dataset, necessitating automated designing procedures
like AutoDropout.

Methods | CIFAR-10 | ImageNet
---|---|---
(WRN-28-10) | (ResNet-50)
Stochastic Depth (2016) | 96.2 $\pm$ 0.07† | 77.5 / 93.7
DropPath (2017) | 95.4 | 77.1 / 93.5
Manifold Mixup (2019a) | 97.5 $\pm$ 0.02 | 77.5 / 93.8
Mixup (2018) | 97.1 $\pm$ 0.08† | 77.9 / 93.9
CutMix (2019) | 96.7 $\pm$ 0.05† | 78.6 / 94.1
MoEx (2020) | 96.7 $\pm$ 0.03 | 79.1 / 94.3
CutMix+RandAugment (2019b) | 97.0 $\pm$ 0.06† | 78.3 / 94.2†
CutMix+FixRes (2019) | n/a | 79.8 / 94.9
AutoDropout+RandAugment | 97.9 $\pm$ 0.06 | 80.3 / 95.1

Table 2: Performance of AutoDropout and representative baselines on supervised
image classification (higher is better). $(\dagger)$ denotes our
implementation. CutMix+FixRes is not applicable for CIFAR-10 since we keep the
image resolution at 32x32 for CIFAR-10.

Methods | CIFAR-10-4K | ImageNet-10%
---|---|---
(WRN-28-2) | (ResNet-50)
LGA+VAT (2019) | 87.9 $\pm$ 0.19 | $-$
ICT (2019b) | 92.7 $\pm$ 0.02 | $-$
MixMatch (2019) | 93.8 $\pm$ 0.06 | $-$
ReMixMatch (2020) | 94.9 $\pm$ 0.04 | $-$
LLP (2020) | $-$ |  $-$ / 88.5
SimCLR (2020) | $-$ | 69.3 / 89.0
FixMatch (2020) | 95.7 $\pm$ 0.05 | 71.5 / 89.1
UDA (2019a) | 94.9 $\pm$ 0.12 | 68.8 / 88.8
UDA+AutoDropout | 95.8 $\pm$ 0.04 | 72.9 / 91.4

Table 3: Performance of AutoDropout and representative baselines on semi-
supervised image classification (higher is better).

#### Qualitative analysis of good dropout patterns.

AutoDropout finds several patterns that are unexpected. For example, the best
noise pattern found for ResNet-50 on ImageNet, which is visualized in Figure
10 in our Appendix Visualization of Good Dropout Patterns, only injects noise
into the first and the last bottleneck convolutional blocks. These two blocks
also have different noise patterns. This behavior is different from DropBlock
(Ghiasi, Lin, and Le 2018), where a fixed and predefined mask of size 7x7 is
applied at every layer. Additionally, rotation is applied in the first block,
but not in the last block, suggesting that AutoDropout finds that rotational
invariance should be enforced at the first block, where most low-level feature
extracting happens, rather than in the last block, where most features have
become more abstract. To validate the decisions of AutoDropout, we vary the
locations where the dropout patterns are applied and observe about 1% drop in
top-1 accuracy, which is significant for ResNet-50 on ImageNet.

### Semi-supervised Image Classification with ConvNets

#### Experiment Settings.

We now consider two typical benchmarks for semi-supervised image
classification: CIFAR-10 with 4,000 labeled examples and ImageNet with 10%
labeled examples. Since our search procedure of AutoDropout on ImageNet, as
described in the previous section, uses a subset of images in ImageNet-10%, we
simply take the same dropout patterns found in that setting. We make sure that
ImageNet-10% contain the 80,000 images that we perform the search on. On
CIFAR-10, we repeat our AutoDropout search with 3,600 training examples and
400 validation examples.

#### Baselines and Results.

We apply AutoDropout into Unsupervised Data Augmentation (UDA; Xie et al.
(2019a)), since UDA has a simple implementation. As shown in Table 3, the
dropout patterns found by AutoDropout improves UDA by 0.9% on CIFAR-10 and
4.1% Top-1 accuracy on ImageNet. Here we compare against recent representative
strong baselines and skip earlier works such as (Tarvainen and Valpola 2017;
Miyato et al. 2018; Lee 2013).

### Language Model and Machine Translation

Methods | PennTreebank | WikiText-2 | IWSLT-14 DeEn | WMT-14 EnFr
---|---|---|---|---
ValidPPL ($\downarrow$) | TestPPL ($\downarrow$) | ValidPPL ($\downarrow$) | TestPPL ($\downarrow$) | BLEU ($\uparrow$) | BLEU ($\uparrow$)
Transformer / Transformer-XL | 59.1 | 56.0 | 63.9 | 61.4 | 34.4 | 38.1
+AutoDropout | 58.1 | 54.9 | 62.7 | 59.9 | 35.8 | 40.0

Table 4: Performance of Transformer and Transformer-XL models trained with
default regularization techniques vs. trained with AutoDropout. For PTB and
WikiText-2, we report the model’s perplexity (lower is better $\downarrow$).
For IWSLT-14-DeEn and WMT-14-EnFr, we report BLEU scores (higher is better
$\uparrow$).

In this section, we first apply AutoDropout to regularize Transformer-XL (Dai
et al. 2019) the task of language model on the Penn Treebank dataset (PTB;
Marcus et al. (1994)). PTB is a small dataset, with about 929K training
tokens, 73K validation tokens, and 82K test tokens, on a vocabulary size of
10K. The small size of PTB makes it a suitable testbed for AutoDropout.

#### Search Configuration.

We use the search space for Transformer models as described in the Section
Search Space for Dropout Patterns inTransformers. Every dropout pattern that
our controller sampled is employed to regularize a training process of
Transformer-XL (Dai et al. 2019). We use the same model size as specified by
Dai et al. (2019). We train every configuration from scratch for 160,000
steps, using a batch size of 16 and a segment length of 70. We use the cosine
learning rate schedule so that each trial converges to a reasonable
perplexity. On 4 TPU v2 chips, each of our runs takes about 40 minutes. The
performance of a configuration $r$ is computed by
$\text{Perf}(r)=80/\text{ValidPPL}(r)$.

#### Results.

Our results for Transformer models are reported in Table 4. Once again, hyper-
parameters for each experiment are reported in our Appendix Hyper-parameters
of Experiments. First, we take the dropout pattern that achieves the lowest
perplexity on PTB and train for 300,000 steps. We compare our results with
Variational Dropout, which is originally used by Transformer-XL (Dai et al.
2019). Under this setting, AutoDropout outperforms Variational Dropout by 1.1
perplexity, which is a significant improvement on this dataset.

#### Transfer learning results.

To test the transferability of the found pattern, we also transfer it to three
other tasks: 1) language modeling on WikiText-2 (Merity et al. 2017), 2)
German-English translation on the IWSLT-14 dataset, and 3) English-French
translation on the WMT-14 dataset. On Wiki-Text-2, we compare AutoDropout’s
dropout pattern against Variational Dropout because we find that it works
better than vanilla Dropout on this task. On translation tasks, we compare
AutoDropout’s dropout pattern against the vanilla Dropout configurations that
are typically applied in Transformer models (Vaswani et al. 2017).

#### Qualitative analysis of the AutoDropout’s dropout pattern.

For Transformer models, AutoDropout assigns different sizes and strides at
different sub-layers in a Transformer layer. For instance, in our best dropout
pattern, visualized in Figure 11 in our Appendix, AutoDropout learns that the
pattern for the multi-head attention layer is similar to Variational Dropout
(Gal and Ghahramani 2016b), but the pattern for the positional feed-forward
layer follows word dropout (Sennrich, Haddow, and Birch 2016). To validate
that such decision is beneficial, we try to apply Variational Dropout in all
layers of Transformer-XL and got the resulting validation perplexity of 59.8,
which is 1.7 point higher than the configuration found by AutoDropout.

### Comparison with Random Search

Figure 5: Best-so-far performances of the first 512 dropout patterns sampled
by AutoDropout and by random search. Top: Accuracy on CIFAR-10 (higher is
better); Bottom: ValidPPL on PennTreebank (lower is better).

Recent works on neural architecture search (Li and Talwalkar 2019) show that
random search is a strong search baseline. Here we perform a controlled
experiment to verify the advantage of AutoDropout’s search process over random
search. To this end, we sample 512 uniformly random patterns from the search
space for WRN-28-2 on CIFAR-10 and another 512 uniformly random patterns from
the search space for Transformer-XL on PTB. We train each of these patterns to
convergence, and compare the results against training the first 512 patterns
suggested by AutoDropout under the same settings. In Figure 5, we plot the
best-so-far performances of both methods, and observe substantial differences
between AutoDropout and random search. Specifically, on CIFAR-10, the best
patterns found by AutoDropout is more than 0.2% accuracy above that of Random
Search. Recall that from Table 1, we know that the standard deviation of
CIFAR-10 accuracy in our code base is less than 0.1%. This means that
AutoDropout is more than 2x standard deviations away from random search and
makes the difference significant. On PTB, the difference between AutoDropout
and Random Search is more than 3 validation perplexity points, which is also
significant for the dataset. We thus conclude that when searching for
structured noise to regularize deep networks, RL search exhibits significant
advantage compared to Random Search.

## Conclusion and Future Directions

We proposed AutoDropout, an algorithm to automatically design dropout patterns
to regularize neural networks. Our algorithm successfully found dropout
patterns that improve the performance of various ConvNets for image
classification, as well as Transformer models for language modeling and
machine translation. Currently, a weakness of AutoDropout is that the method
is computationally expensive. Therefore, a potential future direction is to
develop more efficient search approaches, similar to the developments on
architecture search (Pham et al. 2018; Liu, Simonyan, and Yang 2019; Cai, Zhu,
and Han 2019; Liu et al. 2017) and data augmentation search (Lim et al. 2019;
Wang et al. 2019; Cubuk et al. 2019b).

Although the search cost of AutoDropout can be high, a simple use case of
AutoDropout is to reuse our found patterns in the same way that AutoAugment
policies (Cubuk et al. 2019a) were used to improve state-of-the-art models. To
date, the method of reusing the found AutoAugment (Cubuk et al. 2019a) and
RandAugment (Cubuk et al. 2019b) policies has benefitied many state-of-the-art
models on CIFAR-10 and ImageNet (e.g., Tan and Le (2019); Xie et al. (2019b);
Ridnik et al. (2020); Foret et al. (2020)).

## References

  * Abadi et al. (2016) Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; Kudlur, M.; Levenberg, J.; Monga, R.; Moore, S.; G, Derek . Murrayand Steiner, B.; Tucker, P.; Vasudevan, V.; Warden, P.; Wicke, M.; Yu, Y.; and Zheng, X. 2016.  TensorFlow: A system for large-scale machine learning.  _Arxiv 1605.08695_ .
  * Bello et al. (2017a) Bello, I.; Pham, H.; Le, Q. V.; Norouzi, M.; and Bengio, S. 2017a.  Neural Combinatorial Optimization with Reinforcement Learning.  In _International Conference on Learning Representations Workshop_.
  * Bello et al. (2017b) Bello, I.; Zoph, B.; Vasudevan, V.; and Le, Q. V. 2017b.  Neural optimizer search with reinforcement learning.  In _International Conference on Machine Learning_.
  * Berthelot et al. (2020) Berthelot, D.; Carlini, N.; Cubuk, E. D.; Kurakin, A.; Sohn, K.; Zhang, H.; and Raffel, C. 2020.  ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring.  In _International Conference on Learning Representations_.
  * Berthelot et al. (2019) Berthelot, D.; Carlini, N.; Goodfellow, I.; Papernot, N.; Oliver, A.; and Raffel, C. 2019.  MixMatch: A Holistic Approach to Semi-Supervised Learning.  In _Advances in Neural Information Processing Systems_.
  * Cai et al. (2019) Cai, H.; Lin, J.; Lin, Y.; Liu, Z.; Wang, K.; Wang, T.; and Zhu, Ligeng Han, S. 2019\.  AutoML for Architecting Efficient and Specialized Neural Networks.  In _IEEE Micro_.
  * Cai, Zhu, and Han (2019) Cai, H.; Zhu, L.; and Han, S. 2019.  ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware.  In _International Conference on Learning Representations_.
  * Chen et al. (2020) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020.  A Simple Framework for Contrastive Learning of Visual Representations.  In _International Conference on Machine Learning_.
  * Cubuk et al. (2019a) Cubuk, E. D.; Zoph, B.; Mane, D.; Vasudevan, V.; and Le, Q. V. 2019a.  AutoAugment: Learning Augmentation Policies from Data.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Cubuk et al. (2019b) Cubuk, E. D.; Zoph, B.; Shlens, J.; and Le, Q. V. 2019b.  RandAugment: Practical data augmentation with no separate search.  _Arxiv 1909.13719_ .
  * Dai et al. (2019) Dai, Z.; Yang, Z.; Yang, Y.; Carbonell, J.; Le, Q. V.; and Salakhutdinov, R. 2019\.  Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.  In _Annual Meeting of the Association for Computational Linguistics_.
  * DeVries and Taylor (2017) DeVries, T.; and Taylor, G. W. 2017.  Improved Regularization of Convolutional Neural Networks with Cutout.  _Arxiv, 1708.04552_ .
  * Dong and Yang (2019) Dong, X.; and Yang, Y. 2019.  NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search.  In _International Conference on Learning Representations_.
  * Foret et al. (2020) Foret, P.; Kleiner, A.; Mobahi, H.; and Neyshabur, B. 2020.  Sharpness-Aware Minimization for Efficiently Improving Generalization.  _Arxiv, 2010.01412_ .
  * Gal and Ghahramani (2016a) Gal, Y.; and Ghahramani, Z. 2016a.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.  In _International Conference on Machine Learning_.
  * Gal and Ghahramani (2016b) Gal, Y.; and Ghahramani, Z. 2016b.  A Theoretically Grounded Application of Dropout in Recurrent Neural Networks.  In _Advances in Neural Information Processing Systems_.
  * Ghiasi, Lin, and Le (2018) Ghiasi, G.; Lin, T.-Y.; and Le, Q. V. 2018.  DropBlock: A regularization method for convolutional networks.  In _Advances in Neural Information Processing Systems_.
  * Han, Jiwhan, and Kim (2017) Han, D.; Jiwhan, K.; and Kim, J. 2017.  Deep Pyramidal Residual Networks.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; and Schmidhuber, J. 1997.  Long Short-term Memory.  In _Neural Computations_.
  * Hu et al. (2018) Hu, J.; Shen, L.; Albanie, S.; Sun, G.; and Wu, E. 2018.  Squeeze-and-Excitation Networks.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Huang et al. (2016) Huang, G.; Sun, Y.; Liu, Z.; Sedra, D.; and Weinberger, K. 2016.  Deep Networks with Stochastic Depth.  In _European Conference on Computer Vision_.
  * Jackson and Schulman (2019) Jackson, J.; and Schulman, J. 2019.  Semi-Supervised Learning by Label Gradient Alignment.  _Arxiv 1902.02336_ .
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. L. 2015.  Adam: A Method for Stochastic Optimization.  In _International Conference on Learning Representations_.
  * Krizhevsky (2009) Krizhevsky, A. 2009.  Learning Multiple Layers of Features from Tiny Images.  Technical report.
  * Kudo and Richardson (2018) Kudo, T.; and Richardson, J. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing.  In _Conference on Empirical Methods for Natural Language Processing_.
  * Larsson, Maire, and Shakhnarovich (2017) Larsson, G.; Maire, M.; and Shakhnarovich, G. 2017.  FractalNet: Ultra-Deep Neural Networks without Residuals.  In _International Conference on Learning Representations_.
  * Lee (2013) Lee, D.-H. 2013.  Pseudo-Label: The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks.  In _International Conference on Machine Learning Workshop_.
  * Li et al. (2020) Li, B. L.; Wu, F.; Lim, S.-N.; Belongie, S.; and Weinberger, K. Q. 2020.  On Feature Normalization and Data Augmentation.  _Arxiv, 2002.11102_ .
  * Li and Talwalkar (2019) Li, L.; and Talwalkar, A. 2019.  Random Search and Reproducibility for Neural Architecture Search.  In _Conference on Uncertainty in Artificial Intelligence_.
  * Lim et al. (2019) Lim, S.; Kim, I.; Kim, T.; Kim, C.; and Kim, S. 2019.  Fast AutoAugment.  In _Advances in Neural Information Processing Systems_.
  * Liu et al. (2017) Liu, C.; Zoph, B.; Shlens, J.; Hua, W.; Li, L.-J.; Fei-Fei, L.; Yuille, A.; Huang, J.; and Murphy, K. 2017.  Progressive Neural Architecture Search.  _Arxiv, 1712.00559_ .
  * Liu, Simonyan, and Yang (2019) Liu, H.; Simonyan, K.; and Yang, Y. 2019.  DARTS: Differentiable Architecture Search.  In _International Conference on Learning Representations_.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  SGDR: Stochastic Gradient Descent with Warm Restarts.  In _International Conference on Learning Representations_.
  * Marcus et al. (1994) Marcus, M.; Kim, G.; Marcinkiewicz, M. A.; MacIntyre, R.; Bies, A.; Ferguson, M.; Katz, K.; and Schasberger, B. 1994.  The Penn Treebank: Annotating Predicate Argument Structure.  In _Proceedings of the Workshop on Human Language Technology_.
  * Merity, Keskar, and Socher (2017) Merity, S.; Keskar, N. S.; and Socher, R. 2017.  Regularizing and Optimizing LSTM Language Models.  _Arxiv, 1708.02182_ .
  * Merity et al. (2017) Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2017.  Pointer Sentinel Mixture Models.  In _International Conference on Learning Representations_.
  * Mirhoseini et al. (2017) Mirhoseini, A.; Pham, H.; Le, Q. V.; Steiner, B.; Larsen, R.; Zhou, Y.; Kumar, N.; Norouzi, M.; Bengio, S.; and Dean, J. 2017.  Device Placement Optimization with Reinforcement Learning.  In _International Conference on Machine Learning_.
  * Miyato et al. (2018) Miyato, T.; Maeda, S.-i.; Ishii, S.; and Koyama, M. 2018.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_.
  * Nakkiran et al. (2020) Nakkiran, P.; Kaplun, G.; Bansal, Y.; Yang, T.; Barak, B.; and Sutskever, I. 2020\.  Deep Double Descent: Where Bigger Models and More Data Hurt.  In _International Conference on Learning Representations_.
  * Park et al. (2019) Park, D. S.; Chan, W.; Zhang, Y.; Chiu, C.-C.; Zoph, B.; Cubuk, E. D.; and Le, Q. V. 2019.  SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition.  In _Interspeech_.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; and Chintala, S. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems_.
  * Pham et al. (2018) Pham, H.; Guan, M. Y.; Barret, Z.; Le, Q. V.; and Dean, J. 2018.  Efficient Neural Architecture Search via Parameter Sharing.  In _International Conference on Machine Learning_.
  * Real et al. (2018) Real, E.; Aggarwal, A.; Huang, Y.; and Le, Q. V. 2018.  Regularized Evolution for Image Classifier Architecture Search.  _Arxiv, 1802.01548_ .
  * Real et al. (2017) Real, E.; Moore, S.; Selle, A.; Saxena, S.; Leon, Y. S.; Tan, J.; Le, Q.; and Kurakin, A. 2017.  Large-Scale Evolution of Image Classifiers.  In _International Conference on Machine Learning_.
  * Ridnik et al. (2020) Ridnik, T.; Lawen, H.; Noy, A.; Baruch, E. B.; Sharir, G.; and Friedman, I. 2020\.  TResNet: High Performance GPU-Dedicated Architecture.  _Arxiv, 2003.13630_ .
  * Russakovsky et al. (2015) Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision_ .
  * Sennrich, Haddow, and Birch (2016) Sennrich, R.; Haddow, B.; and Birch, A. 2016.  Edinburgh neural machine translation systems for wmt 16.  In _Workshop on Machine Translation_.
  * Sohn et al. (2020) Sohn, K.; Berthelot, D.; Li, Chun-Liang Zhang, Z.; Carlini, N.; Cubuk, E. D.; Kurakin, A.; Zhang, H.; and Raffel, C. 2020.  FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014.  Dropout: A Simple Way to Prevent Neural Networks from Overfitting.  In _JMLR_.
  * Szegedy et al. (2016) Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; and Wojna, Z. 2016.  Rethinking the Inception Architecture for Computer Vision.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Tan and Le (2019) Tan, M.; and Le, Q. V. 2019.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks.  In _International Conference on Machine Learning_.
  * Tan, Pang, and Le (2020) Tan, M.; Pang, R.; and Le, Q. V. 2020.  EfficientDet: Scalable and Efficient Object Detection.  In _IEEE Conference on Computer Vision and Pattern Recognition_.
  * Tarvainen and Valpola (2017) Tarvainen, A.; and Valpola, H. 2017.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _Advances in Neural Information Processing Systems_.
  * Tompson et al. (2020) Tompson, J.; Goroshin, R.; Jain, A.; LeCun, Y.; and Bregler, C. 2020.  Efficient Object Localization Using Convolutional Networks.  In _International Conference on Learning Representations_.
  * Touvron et al. (2019) Touvron, H.; Vedaldi, A.; Douze, M.; and Jegou, H. 2019.  Fixing the train-test resolution discrepancy.  In _Advances in Neural Information Processing Systems_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017.  Attention is All You Need.  In _Advances in Neural Information Processing Systems_.
  * Verma et al. (2019a) Verma, V.; Lamb, A.; Beckham, C.; Najafi, A.; Mitliagkas, I.; Courville, A.; Lopez-Paz, D.; and Bengio, Y. 2019a.  Manifold Mixup: Better Representations by Interpolating Hidden States.  In _International Conference on Learning Representations_.
  * Verma et al. (2019b) Verma, V.; Lamb, A.; Kannala, J.; Bengio, Y.; and Lopez-Paz, D. 2019b.  Interpolation Consistency Training for Semi-Supervised Learning.  In _International Joint Conference on Artificial Intelligence_.
  * Wang et al. (2019) Wang, Y.; Pan, X.; Song, S.; Zhang, H.; Wu, C.; and Huang, G. 2019.  Implicit Semantic Data Augmentation for Deep Networks Authors.  In _Advances in Neural Information Processing Systems_.
  * Williams (1992) Williams, R. J. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ .
  * Xie et al. (2019a) Xie, Q.; Dai, Z.; Hovy, E.; Luong, M.-T.; and Le, Q. V. 2019a.  Unsupervised Data Augmentation For Consistency Training.  _Arxiv, 1904.12848_ .
  * Xie et al. (2019b) Xie, Q.; Luong, M.-T.; Hovy, E.; and Le, Q. V. 2019b.  Self-training with Noisy Student improves ImageNet classification.  _Arxiv 1911.04252_ .
  * Ying et al. (2019) Ying, C.; Klein, A.; Real, E.; Christiansen, E.; Murphy, K.; and Hutter, F. 2019\.  NAS-Bench-101: Towards Reproducible Neural Architecture Search.  In _International Conference on Machine Learning_.
  * Yun et al. (2019) Yun, S.; Han, D.; Oh, S. J.; Chun, S.; Choe, J.; and Yoo, Y. 2019.  CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.  In _International Conference on Computer Vision_.
  * Zagoruyko and Komodakis (2016) Zagoruyko, S.; and Komodakis, N. 2016.  Wide Residual Networks.  In _British Machine Vision Conference_.
  * Zaremba, Sutskever, and Vinyals (2014) Zaremba, W.; Sutskever, I.; and Vinyals, O. 2014.  Recurrent Neural Network Regularization.  _Arxiv, 1409.2329_ .
  * Zhang et al. (2018) Zhang, H.; Cisse, M.; Dauphin, Y. N.; and Lopez-Paz, D. 2018.  mixup: Beyond Empirical Risk Minimization.  In _International Conference on Learning Representations_.
  * Zhuang et al. (2020) Zhuang, C.; Ding, X.; Murli, D.; and Yamins, D. 2020.  Local Label Propagation for Large-Scale Semi-Supervised Learning.  _Arxiv, 1905.11581_ .
  * Zoph et al. (2019) Zoph, B.; Cubuk, Ekin D.and Ghiasi, G.; Lin, T.-Y.; Shlens, J.; and Le, Q. V. 2019\.  Learning Data Augmentation Strategies for Object Detection.  _Arxiv 1906.11172_ .
  * Zoph and Le (2017) Zoph, B.; and Le, Q. V. 2017.  Neural Architecture Search with Reinforcement Learning.  In _International Conference on Learning Representations_.
  * Zoph et al. (2018) Zoph, B.; Vasudevan, V.; Shlens, J.; and Le, Q. V. 2018.  Learning Transferable Architectures for Scalable Image Recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition_.

Appendix

AutoDropout: Learning Dropout Patterns to Regularize Deep Networks

##  Appendix A Details on the Search Spaces for ConvNets

#### Details on generating dropout pattern.

In our search space for ConvNets, each dropout pattern is generated by its
hyper-parameters: size, stride, repeat, share_c, residual, rotate, shear_x,
and shear_y, in that order. The available values for the operations are
summarized in Table 5.

Parameters | Semantics | Available Values
---|---|---
size | Size of the pattern |  $\\{k\times\lfloor d/5\rfloor:k=0,1,...,4\\}$ ($d$ is the tensor’s size)
stride | How many cells to skip when tiling | $1,2,4,8,16$
repeat | How many times to repeat the pattern | $1,2,3,...,32$
share_c | Share a pattern across channels | True, False
residual | Apply a pattern to the residual branch | True, False
rotate | Pattern’s max rotating degree | $0,15,30,45,60,75$
shear_x | Pattern’s max shearing rate | $0.,0.05,0.1,...,0.55$
shear_y | Pattern’s max shearing rate | $0.,0.05,0.1,...,0.55$
Table 5: Semantics of the hyper-parameters that specify a ConvNet dropout
pattern and their available values in our search space.

We use the same dropout pattern for layers with same spatial size. For
instance, in a ResNet-50, there are 4 bottleneck convolutions groups: having
has 3 blocks, 4 blocks, 6 blocks, and 3 blocks respectively. The spatial
dimensions of these blocks are 56x56, 28x28, 14x14, and 7x7, decreasing by a
factor of 2 after each block due to strided convolutions or spatial reduction
pooling. To reduce the number of decisions that our controller has to make,
within each of these 4 groups, the dropout patterns are kept the same (but the
actual samples of the dropout masks are still random at training time). Figure
10 in Appendix shows this sharing scheme in a pattern found by AutoDropout.

Figure 6: Example noise patterns from our search space for image understanding
models. Here, we visualize the noise applied on the RGB channels of an image
for illustration purposes. (a): The original image. (b-c): Three different
patterns, one for each of the RGB channels of an image, and the resulting
image after applying the patterns. (d-e): Three channels RGB share the same
pattern (which leads to the color black), and the resulting image by applying
this shared pattern. Image best viewed in color.

#### Example geometric transformations.

The geometric transformations, namely rotating and shearing along each
dimensions, are implemented using projective transformations. Figure 6 shows
the effects of these transformations on some example dropout patterns. In this
figure, we consider 3 RGB channels and visualize the patterns as they apply to
the image. In our search space, the masks are applied to intermediate layers
with many more channels.

Figure 7: For ConvNets, we apply the dropout patterns immediate after the
batch normalization layers. Shown are the examples ConvNet blocks in our
experiments: (a) post-activation ResNet; (b) pre-activation ResNet; (c) Mobile
Inverse Convolutional cell (MBConv; (Tan and Le 2019)).

#### Where to apply the dropout patterns.

Figure 7 specifies where we apply the dropout patterns for ConvNets. In
general, we apply the dropout pattern after each batch normalization layer. If
a convolutional block has a residual branch, which sometimes has a 1x1
convolution followed by batch normalization, then we also apply a dropout
pattern after the normalization as well.

##  Appendix B Details on the Search Spaces for Transformer

Parameters | Semantics | Available Values
---|---|---
size | How many consecutive tokens to affect | $0,10,20,30,40,50,60,70$
stride | How many consecutive tokens to skip | $0,5,10,15,20$
share_t | Share a mask across the tokens to affect | True, False
share_c | Share a pattern across channels | True, False
Table 6: Semantics of the hyper-parameters that specify a Transformer dropout
pattern and their available values in our search space.

#### Details on generating dropout patterns.

In our search space for Transformer, each dropout pattern is generated by its
hyper-parameters: size, stride, share_t, and share_c, in that order. The
available values for the operations are summarized in Table 6.

We allow our controller to generate different patterns at different steps in a
Transformer layer. Specifically, Figure 8 shows where the dropout patterns
could be applied, in a self-attention operation and in a positional feed-
forward operation. If a self-attention operation uses multi-head attention,
then we use the same dropout pattern across all heads. However, within each
head, the position to apply the dropout pattern is randomly sampled at
training time. Similarly, in a typical Transformer network, where multiple
Transformer layers are stacked above each other, we use the same dropout
pattern to all layers, but the actual dropout mask are generated randomly and
independently at each layer.

Figure 8: We apply the noise at various nodes inside a multi-head self-
attention operation. Left: Noise in a two-headed attention operation. Right:
Noise in a positional feed-forward operation.

##  Appendix C Illustration for the Controller Parallel Training Algorithm

We provide an illustrative working example of the distributed reinforcement
learning search algorithm for controller, as described in Section Controller
Model and Search Algorithms. Figure 9 visualizes the example.

Our search process runs on a shared cluster. The controller has a queue
$q_{\text{unfinished}}$ which stores the generated dropout patterns waiting to
be executed, as the workers become available. In our example, suppose that thw
cluster has only two available workers. The controller sequentially dequeues
$(r_{i},P(r_{i},\theta_{i}))$ and $(r_{j},P(r_{j},\theta_{j}))$ and sends them
to the two workers to train. When one of these dropout patterns finishes, say
the $i^{\text{th}}$ on $\text{Worker}_{1}$ finishes, the controller sends
$(r_{i},\text{Perf}(r_{i}))$ into $q_{\text{finished}}$ and sends
$(r_{k},P(r_{k},\theta_{k}))$ into the now-available $\text{Worker}_{1}$ to
train. Later, after the $j^{\text{th}}$ dropout pattern and the
$k^{\text{th}}$ dropout pattern both finish, and the controller has finished
sending their results to $q_{\text{finished}}$, then $q_{\text{finished}}$ has
$M=3$ finished configurations, where $M$ is the minibatch size for the
controller updates. The controller now computes the gradients corresponding to
the $i^{\text{th}}$, $j^{\text{th}}$, and $k^{\text{th}}$ dropout patterns,
scales them according to the probability ratios as specified in Equation 3,
and averages the resulting gradients to update its parameters.

If during the controller’s training, more workers become available, then more
dropout configurations from $q_{\text{unfinished}}$ can be sent to the
available workers to train to enhance the model’s better parallelism. This is
a significant advantage compared to previous AutoML search algorithms, which
always requires $M$ workers to be available, or the search process has to stay
idle waiting for the minibatches to finish.

Figure 9: Illustration of our controller. From left tor right order with $M=3$
and $C\gg 3$. (1) The controller generates up to $C$ regularization rules and
stores these rules in $q_{\text{spawn}}$, along with their sample
probabilities. (2) A fixed pool of workers dequeue the rules from
$q_{\text{spawn}}$ and train a model with these rules. In this case, $2$
workers dequeue and train $3$ rules $r_{i}$, $r_{j}$, $r_{k}$ to produce
$\text{Perf}(r_{i})$, $\text{Perf}(r_{j})$, $\text{Perf}(r_{k})$. (3) When $M$
rules have their Perf’s measured, their corresponding gradients are computed
using importance sampling as in Equation 3, and then are averaged to update
the controller at its instantaneous parameter $\theta_{\text{update}}$. If we
select $C$ sufficiently large, the only bottleneck of this procedure is the
number of available workers.

##  Appendix D Hyper-parameters of Experiments

#### Controller.

We use a small Transformer model to parameterize our controller. Specifically,
our Transformer architecture has 4 layers, with the hidden size of 128. Each
multi-head attention operation uses 4 heads, and each head has the hidden
dimension size of 32. The positional feed-forward has the inner dimension of
32. The controller’s parameters are initialized at a normal distribution with
zero mean and a standard deviation of 0.02. We update the controller’s
parameters using Adam (Kingma and Ba 2015) with a constant learning rate of
0.00035 and the default values $\beta_{1}=0.9$, and $\beta_{2}=0.999$. We also
use a moving average baseline with momentum $0.95$ to stabilize the update,
and an entropy regularization of $10^{-5}$ to encourage the controller’s
explorations. For each search, our controller explores 16,384 dropout patterns
in total, and updates its parameters using a batch size of 16, leading to
1,024 updates.

#### Image Recognition Models.

In order to avoid tuning the dropout rate at each layer of a ConvNet, we
specify a single dropout rate for the final convolutional layer. Previously
layers have their dropout rate linearly increased from 0 to the specified
value. During search time, we set the final value to 0.2. Once the search
finishes, we tune the final value among the list of 0.1, 0.2, …, 0.7. We find
the with out dropout pattern, the ideal final dropout rate for WRN-28-10,
ResNet-50, and EfficientNet are 0.6, 0.3, and 0.5. Apart from the layer-wise
dropout rate, we use the same values with Tan and Le (2019) for EfficientNet,
the same values with Ghiasi, Lin, and Le (2018) for ResNet-50 on ImageNet, the
same values with Xie et al. (2019a) for WRN-28-{2,10} on CIFAR-10. Note that
this means that we train ResNet-50 for 240 epochs, which is 1.5 times longer
than normally done for this architecture, but we train EfficientNet for 350
epochs, which is the same with Tan and Le (2019).

#### Language Model.

For both Penn Treebank and WikiText-2, we use the Transformer-XL architecture
(Dai et al. 2019), which has 16 layers, hidden size of 380, 10 heads each of
dimension 38, and positional feed-forward inner size of 900. For Penn
Treebank, this results in a model with 24 million parameters, while for
WikiText-2, this results in a model with 35 million parameters. We use a
dropout rate of 0.5 for the embedding layer, a dropout rate of 0.6 for the
softmax layer. We find these dropout rates from the Penn Treebank code
released by Dai et al. (2019). We use the dropout rate of 0.2 elsewhere in our
Transformer-XL model. We also use the state-value and state-difference
regularizations (Merity, Keskar, and Socher 2017), even though we do not
observe significant raise in perplexities without using them. We train with
Adam for 160K steps during AutoDropoutsearch, and 320K steps for the best
architecture that AutoDropout finds. We using a cosine-decayed learning rate
schedule (Loshchilov and Hutter 2017), starting at $3\times 10^{-4}$ and
decaying to $10^{-4}$ throughout 80% of the training process. After the
learning rate decays to $10^{-4}$, we continue the remaining 20$ of the
training process with a constant learning rate of $5\times 10^{-5}$. During
the last 20% of the training procedure, we start collecting a moving average
trail of the model’s parameters. We perform one validation evaluation every
1,000 training steps and store the best model checkpoint. In the end, we
obtain the test perplexity from the checkpoint with the lowest validation
perplexity.

#### Machine Translation.

We use the Transformer-Base architecture from Vaswani et al. (2017). We
tokenize the training, validation, and test data by SentencePiece (Kudo and
Richardson 2018), with a vocabulary size of 10,000 for the IWSLT 14 De-En
dataset, and a vocabulary size of 32,000 for the WMT 14 En-Fr dataset. After
tokenizing the data, we filter the training datasets, keeping only sentences
that have no more than 360 tokens for IWSLT 14 De-En, and keeping only
sentences that have mo nore than 200 tokens for WMT 14 En-Fr. We share the
embeddings for both the encoder and the decoder Transformer, and use the same
embedding matrix for softmax in the decoder. We train our models using Adam,
with a learning rate linearly warming up for 4,000 steps to $1.6\times
10^{-3}$, and then decreasing to 0 using the cosine schedule. We train for
15,000 steps on IWSLT 14 De-En, and 35,000 steps on WMT 14 En-Fr. We do not
use checkpoint averaging for decoding, which could potentially improve our
results.

When we transfer the dropout pattern found on Penn Treebank to our machine
translation experiments, we keep the same hyper-parameters: size, stride,
share_t, and share_c. Unlike the language model tasks, we do not use embedding
dropout or softmax dropout. We also set the dropout rate at all steps to 0.1.

##  Appendix E Visualization of Good Dropout Patterns

Figure 10: The best configuration found for ResNet-50. Left: AutoDropout
suggests that we should only apply the noise patterns at group_1 and group_4,
out of the bottleneck convolutional blocks of ResNet-50. Meanwhile, no noise
is injected to group_2 and group_3. Middle: detailed configurations for these
two groups. Right: Example masks for 8 channels of the corresponding groups,
where the values at white pixels are set to 0. Figure 11: The best dropout
pattern that AutoDropout finds for Transformer-XL on Penn Treebank. Left: the
dropout pattern in the self-attention operation. Right: the dropout pattern in
the positional feed-forward operation. Meanings of the dropout pattern’s
hyper-parameters: At each step where the controller can apply a dropout
pattern, we specify a tuple of (size, stride, share_t, share_c). size and
stride specify how many consecutive tokens are affected by the dropout
pattern, and then how many consecutive tokens are not affected by the pattern.
share_t means whether the dropout pattern uses the same mask at all size
temporal steps that it affects, and share_c) decides whether the pattern uses
the same mask across the channel dimension. A tuple of None means that the
controller decides to not apply any dropout pattern at the corresponding step.
In this case, the controller does not apply any noise pattern on the residual
branches.


# Security and Privacy Challenges in Deep Learning Models

Gopichandh Golla  Computer Science and Engineering
The State University of New York at Buffalo
Buffalo, USA
<EMAIL_ADDRESS>
###### Abstract

These days, deep learning models have achieved great success in multiple
fields, right from autonomous driving to medical diagnosis. These models have
expanded the abilities of artificial intelligence by offering great solutions
to complex problems that were very difficult to solve earlier. In spite of
their unseen success in various, it has been identified, through research
conducted, that deep learning models can be subjected to various attacks that
compromise model security and data privacy of the Deep Neural Network models.
Deep learning models can be subjected to various attacks at different stages
of their lifecycle. During the testing phase, attackers can exploit
vulnerabilities through different kinds of attacks such as Model Extraction
Attacks, Model Inversion attacks, and Adversarial attacks. Model Extraction
Attacks are aimed at reverse-engineering a trained deep learning model, with
the primary objective of revealing its architecture and parameters. Model
inversion attacks aim to compromise the privacy of the data used in the Deep
learning model. These attacks are done to compromise the confidentiality of
the model by going through the sensitive training data from the model’s
predictions. By analyzing the model’s responses, attackers aim to reconstruct
sensitive information. In this way, the model’s data privacy is compromised.
Adversarial attacks, mainly employed on computer vision models, are made to
corrupt models into confidently making incorrect predictions through malicious
testing data. These attacks introduce and change the input data in such a way
that it is not recognizable as adversary data, causing deep learning models to
mis-classify and make wrong decisions. Models can be subjected to attacks not
only evaluation phase but also during the training phase. Data Poisoning
Attacks introduce malicious examples into the training dataset, which makes
the training phase ineffective and leads to the loss of the integrity of the
deep learning model. Data privacy can be compromised by not only model
inversion attacks but also by service providers of AI models.

This Survey is done to gain an in-depth understanding of the security and
privacy problems of Deep neural nets. It is done by analyzing what kinds of
attacks are present, how each type of attack works, and its challenges and
drawbacks. This paper also discusses what are the potential evolutions of
these attacks moving forward as well.

###### Index Terms:

Adversarial Attacks Poisoning Attacks Model Extraction Attacks Model Inversion
Attacks

##  I Introduction

These days, deep learning models have achieved great success in multiple
fields, right from medical diagnosis and natural language processing to
autonomous driving and image classification. In healthcare, Deep learning
models are utilized in disease prediction and medical image analysis with
greater accuracy. In finance, they are involved in fraud detection systems.
Computer Vision models of Deep learning help self-driving cars perform in
complex environments in terms of navigation. Natural Language Processing
models are improving virtual assistants and language translation tools. The
major reason behind the success of Deep neural net models is their great
ability to learn patterns from vast datasets available. Advances in computing
power and advanced algorithm innovation have also assisted in the success of
Deep neural network models. On the flip side, integration of these advanced
models into a wide range of domains has come with its own set of security and
privacy vulnerabilities that require thorough investigation and defense
techniques.

The vulnerabilities of these Deep neural net models pose severe challenges and
threats to the integrity, reliability, and confidentiality of these systems. A
compromised DL model can lead to bad decision-making, improper working of
autonomous systems, or breaches of confidential data. These vulnerabilities
may lead to financial losses to threats to public safety[2]. For example, a
compromised Autonomous driving system may lead to accidents and loss of lives.
Face recognition systems used in biometric authentication systems can be
compromised by introducing noise to the system thus compromising the Integrity
of the systems.

This survey is conducted to analyze and understand the length and breadth of
attacks on Deep Learning systems. Attacks are categorized into 4 types, such
as Model Extraction Attacks, Model Inversion attacks, Adversarial attacks, and
Poisoning Attacks[1]. This survey discusses different techniques and their
challenges and drawbacks along with the potential evolution of the attacks in
the future.

Model Extraction and Model inversion attacks are aimed at compromising the
privacy of the DL models and exposing model parameters and training data.
Model Extraction attack is aimed at duplicating the DL model’s parameters and
architecture. Whereas, Model Inversion attacks are aimed at reproducing
confidential information by going through already available data. Adversarial
attacks( Evasion attacks) and poisoning attacks’ main purpose is to influence
prediction results either by providing falsified testing data and getting
wrong predictions with confidence or obstructing the model training during the
training phase with the help of adversarial examples. Evasion attacks are done
during the evaluation phase and are further classified into white box and
black box attacks. beforehand knowledge of the model is necessary in order to
execute white box attacks where attackers use these data to form perturbed
inputs to cause undesirable outcomes. Gradient-based techniques are the common
mode of operations employed to perform white-box attacks. Black box attacks
are more common between these 2 types as this resembles real-world scenarios
much better where the attackers don’t have prior knowledge of the models.
Queries are made to understand the model’s behavior to disturb the performance
of the models. Here, just the model’s behavior is known, but not the model’s
architecture or hyperparameters. Poisoning attacks are done during the
training phase of the model by introducing wrongful data into the trained
dataset. These kinds of attacks are under-researched and are divided into 3
subcategories, they are performance degradation, targeted poisoning, and
backdoor attacks. In consideration of the impact of adversarial threats,
extensive research has been dedicated to developing defense mechanisms against
adversarial attacks. Techniques such as adversarial training, gradient
masking, and robust optimization aim to bolster DL models’ resilience to
adversarial perturbations. Adversarial attacks are not constrained to a single
domain, they can affect a wide range of domains or architectures such as CNNs,
RNNs, etc.[3]

To summarize, This Survey is done to gain an in-depth understanding of the
security and privacy problems of Deep neural nets. It is done by analyzing
what kinds of attacks are present, how each type of attack works, and its
challenges and drawbacks. This paper also discusses what are the potential
evolutions of these attacks moving forward as well.

##  II Main Techniques

###  II-A Model Extraction Attacks

In this type of attack, the attacker doesn’t know the internal workings of the
model and the aim is to duplicate the model. The attacker queries the model
with input x to get the result y. By performing extensive querying as
mentioned before, the attacker gains enough information needed to reverse
engineer the model and be able to replicate its internal working. This
information can be further used to perform subsequent attacks like adversarial
attacks on the model. The adversary model for this attack typically operates
in a black-box manner, where the attackers have no information on how the
model works and no information regarding the architecture, parameters,
hyperparameters, and training dataset is known. The attacker can only interact
with the model through prediction APIs. Moreover, they do not have access to
data that matches the distribution of the target model’s training data. There
are also limitations on query frequency, as attackers may face restrictions or
be blocked by the API if they submit queries too frequently.

Model extraction can be done in 3 different ways, such as Equation solving,
Metamodel training, and Training substitute model. In equation solving, a
typical classification machine learning model that works on continuous
function can be solved by having enough pairs of input and output values. In
the metamodel training approach(classifier of classifiers), attackers query
the model to gain the output for a specific input. These values are used to
train a metamodel to replicate the original model, thus used later to predict
the original model’s parameters. Training substitute model is similar to the
metamodel training approach where a set of input-output value pairs are
trained on the substitute model and that model’s parameters mimic the original
model’s parameters.

###  II-B Model Inversion Attacks

Model Inversion attack is a kind of attack employed to gain information about
the training data of Neural Networks models. Model Inversion attack uses a
concept called inverse information flow. Since neural networks can remember a
lot of information, inverse information flow can be used to gain insights into
data properties, thus compromising the data privacy of Deep learning models.
Model Inversion attack can be done in both white box and black box settings.
Model Inversion attack can be done in 2 different ways, such as Membership
Inversion attack and property inversion attack. The presence of specific data
records in the training dataset can be determined using a Membership Inversion
attack, whereas Statistical property inference can be achieved through a
Property Inference Attack.

####  II-B1 Membership Inference Attack :

Membership inference attack is a kind of Model inversion attack whose main aim
is to determine the presence of specific data in the training dataset or not.
This attack can be done in both black-box and white-box setups. The working of
this kind of attack is done in 4 steps, namely, Data synthesis, Shadow model
training, attack model training, and membership determination. In the data
synthesis step, data can be generated either manually or by training generated
models. As part of shadow model training, shadow models are trained to
generate training data for the attack model. The attack model is trained using
the data generated above to determine whether a given instance of the training
dataset is part of the training dataset or not is done attack model training
step. In the membership determination step, the determination of whether the
input instance is part of the training dataset or not.

####  II-B2 Property Inference Attack :

A property Inference Attack is a kind of Model Inversion Attack where the main
goal of the attack is to determine the characteristics of the dataset. A
workflow of this kind is similar to a Membership inference attack but the end
goal is different. Here, the dataset is divided based on the presence of a
certain feature.

###  II-C Adversarial Attacks

This kind of attack is also known as an evasion attack where adversarial
examples are applied during the inference phase(testing phase), thereby
causing the model to incorrectly classify the input data. This attack can be
done on all kinds of models such as image classification, speech recognition,
text processing, and malware detection. Adversary examples used are not
recognizable by humans but can deceive the target models. Adversarial attacks
can also be done in both white-box and black-box settings. Adversarial attacks
are efficiently developed in image classification models. Here, the
distance(perturbation) between the adversarial example and the true example
should be as minimal as possible. This distance can be measured using Lp
norms. Here the attacks can be either targeted or non-targeted.

####  II-C1 White-box attacks :

In this type of attack, attackers already have intrinsic information about the
target models. perturbations required to generate adversarial examples can be
obtained by either calculating gradients or solving optimization problems. The
first attack that we discuss here is done using the Limited-memory Broyden-
Fletcher-Goldfarb-Shanno( L-BFGS) algorithm to generate adversarial
examples[4]. Similar to the above attack, CW-Attack minimizes a different
objective function to obtain perturbations[5]. Jacobian-based Saliency Map
Attack ( JSMA ) finds the perturbations by calculating the saliency maps of
gradients derived from forward propagation[6]. This saliency map is then used
to find the perturbations. L0 is used as a distance metric here. Fast Gradient
Sign Method ( FGSM ) : FGSM explores the linearity of neural net models to
drive the input data towards the gradients to obtain needed perturbations[7].
This model can be helpful in linearly related simple models. Iterative FGSM:
Unlike the traditional FGSM model, the iterative FGSM model computes the
perturbated examples step by step by applying the FGSM multiple times[8]. This
method is faster than the traditional model with higher success rates as well.

####  II-C2 Black-box attacks :

In general, black box attacks are closer to the real world as model details,
such as architecture, parameters, hyperparameters, and gradients, are not
known to attackers most of the time. Industrial and mobile deployed models are
kept confidential and are not accessible to adversaries. The following are
some of the Black-box attacks.

Transfer-based attacks[4] are developed based on the assumption of inherent
linearity of the neural nets, allowing a substitute model to approximately
predict the decision boundary of the actual model. This substitute model’s
architecture is determined based on the input format, such as images or text.
Zeroth-order optimization is a popular form of attack where attackers
approximate the gradient using only the Model’s prediction API. However, these
are computationally expensive to implement. In Limited Information attacks,
Chen et al.[9] proposed a method that approximates the gradient of a target
model using only the output scores thus minimizing the amount of information
needed. Suet et al.[10] introduced a technique where only one pixel is changed
to produce adversarial examples. In Predicted Label-based attacks, Only
predicted labels are used to produce the adversary examples due to limited
information availability. Boundary attacks use random walks to approach the
decision boundary iteratively.

###  II-D Poisoning Attacks

These kinds of attacks are employed during the training phase of the target
model by introducing malicious examples to weaken the model training. These
kinds of attacks are under-researched to date and are classified into 3
categories, namely performance degradation attacks, targeted poisoning
attacks, and backdoor attacks.

####  II-D1 Performance degradation attack:

In this kind of attack, performance degradation is achieved using malicious
examples that are developed using bi-level optimization.

####  II-D2 Targeted Poisoning Attack:

The main aim of this attack is to misclassify certain test examples during
testing by modifying training data based on changes to parameters and loss
functions.

####  II-D3 Backdoor attack:

In this attack, the main aim is to create a backdoor to the model that can be
exploited during the testing phase. Insertion of subtle triggers like patches
or watermarks into training samples and forcing neural nets to produce false
outputs.

##  III Issues and Problems

Extracting machine learning models using approaches like equation solving,
training metamodels, and training substitute models contains certain
challenges and drawbacks. Solving equations is not so complex for small
models. For larger models, equations become more complex thus leading to less
practicality of solving them. Equation solving cannot be a good choice when
quick extraction of models is necessary. The equation-solving method is
limited to classification models that deal with continuous class variables.
When it comes to training metamodels and training substitute models, large
amounts of data are required. So, querying the models to get relevant data can
be a challenging process. In addition to collecting large amounts of data, the
computation of substitute and metamodel models requires considerable amounts
of computational resources. Substitute models trained cannot perform well on
complex target models or models that are trained on diverse data.

With respect to Model Inversion attacks, the Shadow model training step can be
a time-consuming task and require a good amount of computational resources.
One of the challenges with respect to data synthesis is the step collection of
training data that closely replicates the training data can be really
difficult if there is no extensive knowledge about the dataset’s specifics is
available. The effectiveness of the models developed cannot be consistent
across all the attacks and can be ineffective at times when the type of data
is diverse or the target model is more complex. One of the main drawbacks of
model inversion attacks is the amount of resources required to train shadow
models is expensive. Strongly defended target models cannot be attacked with
this kind of attack

White box attacks contain their own set of challenges and drawbacks. L-BFGS
attack is computationally expensive and has a high misclassification rate. CW
attack also has its challenges such as tuning hyperparameters and solving the
complex objective function is computationally expensive. One of the challenges
of JSMA attack is the calculation of accurate salient maps and one of its
major drawbacks is its high reliance on the L0 norm to calculate distances
between the adversary and true examples. Since Iterative FGSM depends on
calculating gradient values iteratively, convergence is not guaranteed in all
scenarios and may be tiresome. Determining the Optimal number of iterations
required can be a challenge.

One of the challenges in black-box attacks is a limited amount of information
with respect to the target model is available. Substitute model creation is
really expensive on both data collection and model training fronts. Model APIs
block the users when they query frequently, thus posing a challenge to collect
the data required. Drawbacks of black-box attacks consist of vulnerability to
defenses mainly substitute models. Not all attacks are effective. Models that
are developed to attack a target system can be effective on only that model,
but not on other models thus limiting the model’s transferability.

For poisoning attacks, developing perturbed examples can be a challenging task
when attackers have a limited amount of training data. This kind of attack can
also require intensive resources. One of the challenging tasks in these kinds
of attacks is the creation of good adversarial examples that look like actual
samples. These kinds of attacks are limited to models to which they are aiming
to perturb, and cannot be transferred to other target models.

##  IV Future trends

Deep learning is a discipline that is always evolving, and there is a never-
ending battle between attackers and defenders over model security.

###  IV-A Model Extraction Attacks :

Attackers may create ”advanced query strategies” that query models to obtain
information with fewer queries, which is one possible progression of model
extraction attacks. To obtain more accuracy, attackers can also combine
various extraction model combinations. One evolution in model extraction
attacks is improvement in transfer learning.

###  IV-B Model Inversion Attacks :

These days, model inversion attacks on complex systems are not as effective;
nevertheless, in the future, with the development of sophisticated
methodologies, complicated models may be inverted. As security continues to
advance, attackers can leverage the most recent knowledge to circumvent
advanced data privacy strategies.

###  IV-C Adversarial Attacks :

One possible development in adversarial attacks is the ability to expand
beyond attacks on a certain target model to additional models. This guarantees
that transfer learning will improve. This development of more complicated
adversarial attacks may target multiple different models at the same time. In
addition, adversaries might create algorithms that operate in real time and
drastically reduce time overhead.

###  IV-D Poisoining Attacks :

In terms of poisoning attacks, altered instances produced during the model’s
training phase will resemble real examples even more and become more difficult
to identify during the training phase of the models. In the future, backdoor
attacks may develop into more sophisticated ones that may include more
sophisticated trigger mechanisms and stealthy activation functions. Attackers
may utilize recent advancements in the field of reinforcement learning and
develop even better backdoor triggers. Attackers may modify their poisoning
tactics in response to the target model’s defense measures.

In summary, It’s important to note that as attacks evolve, the field of AI
security will also evolve to develop better defenses and strategies to
mitigate these threats.

## References

  * [1] Y. He, G. Meng, K. Chen, X. Hu and J. He, ”Towards Security Threats of Deep Learning Systems: A Survey” in IEEE Transactions on Software Engineering, vol. 48, no. 05, pp. 1743-1770, 2022
  * [2] J. Li, F. Schmidt, and Z. Kolter, “Adversarial camera stickers: a physical camera-based attack on deep learning systems,” in Proc. International Conference on Machine Learning, 2019.
  * [3] S. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 86–94
  * [4] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
  * [5] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in Proc. IEEE Symposium on Security and Privacy, 2017.
  * [6] N. Papernot, P. D. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “the limitations of deep learning in adversarial settings,” in Proc. IEEE European Symposium on Security and Privacy, 2016.
  * [7] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  * [8] A. Kurakin, I. Goodfellow, S. Bengio et al., “Adversarial examples in the physical world,” 2016.
  * [9] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “ZOO: zeroth order optimization based blackbox attacks to deep neural networks without training substitute models,” in Proc. 10th ACM Workshop on Artificial Intelligence and Security, 2017.
  * [10] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” IEEE Transactions on Evolutionary Computation, no. 5, pp. 828–841, 2019.
  * [11] Ho Baey, Jaehee Jangy, Dahuin Jung, Hyemi Jang, Heonseok Ha, Hyungyu Lee, and Sungroh Yoon, ” Security and Privacy Issues in Deep Learning”, arXiv:1807.11655v4 [cs.CR] 10 Mar 2021