# Adversarial Rain Attack and Defensive Deraining for DNN Perception

Liming Zhai, Felix Juefei-Xu, Qing Guo, Xiaofei Xie, Lei Ma, Wei Feng,
Shengchao Qin, and Yang Liu Liming Zhai, Qing Guo and Yang Liu are with
Nanyang Technological University, 639798, Singapore.
Felix Juefei-Xu is with Alibaba Group, Sunnyvale, CA 94085, USA.
Xiaofei Xie is with Singapore Management University, 188065, Singapore.
Lei Ma is with University of Alberta, Edmonton, AB T6G 2R3, Canada and also
with Kyushu University, Fukuoka 819-0395, Japan.
Wei Feng is with the School of Computer Science and Technology, Tianjin
University, Tianjin 300305, China.
Shengchao Qin is with the School of Computing, Media and the Arts, Teesside
University, Middlesbrough TS1 3BX, United Kingdom.

###### Abstract

Rain often poses inevitable threats to deep neural network (DNN) based
perception systems, and a comprehensive investigation of the potential risks
of the rain to DNNs is of great importance. However, it is rather difficult to
collect or synthesize rainy images that can represent all rain situations that
would possibly occur in the real world. To this end, in this paper, we start
from a new perspective and propose to combine two totally different studies,
_i.e_., rainy image synthesis and adversarial attack. We first present an
adversarial rain attack, with which we could simulate various rain situations
with the guidance of deployed DNNs and reveal the potential threat factors
that can be brought by rain. In particular, we design a factor-aware rain
generation that synthesizes rain streaks according to the camera exposure
process and models the learnable rain factors for adversarial attack. With
this generator, we perform the adversarial rain attack against the image
classification and object detection. To defend the DNNs from the negative rain
effect, we also present a defensive deraining strategy, for which we design an
adversarial rain augmentation that uses mixed adversarial rain layers to
enhance deraining models for downstream DNN perception. Our large-scale
evaluation on various datasets demonstrates that our synthesized rainy images
with realistic appearances not only exhibit strong adversarial capability
against DNNs, but also boost the deraining models for defensive purposes,
building the foundation for further rain-robust perception studies.

###### Index Terms:

Adversarial rain, deraining, factor-aware rain generation, classification,
detection, data augmentation.

##  1 Introduction

Rain is condensed aqueous vapor in the form of falling drops with high speeds
and small sizes. As a common weather phenomenon, rain can bring massive
impacts not only on our human society, but also significant influences on
today’s intelligent era. Such impacts are most prominent in deep neural
network (DNN) based perception systems, _e.g_., autonomous driving, video
surveillance and unmanned aerial vehicle (UAV), which can be easily disturbed
by the inevitable rain effects, suffering from severe safety and security
issues [1, 2]. Therefore, it is of great importance and pressing to
comprehensively study how the rain affects the DNNs and how to reduce the
negative rain effect.

The rain effect in images to DNNs has something in common with adversarial
examples, which are typically generated by adding well-crafted perturbations
to benign samples under some constraints [3]. In particular, rain can also
mislead the DNNs as if by adding minor rain noises (_i.e_., some non-trivial
perturbation) to clean images. However, the reasons behind the adversarial
noise attack and rain effect are obviously different. The attack capability of
adversarial noises comes from the controllable artificial perturbations
specified by target DNNs. While the rain effect on DNNs not only depends on
the object occlusion and degraded visibility, but also is related to the rain
appearances, such as rain intensity, rain direction, and rain brightness,
_etc_. Which type of rain appearance mostly affects the DNNs, and how the rain
appearances cause the failure of DNNs? These are important problems that have
so far been relatively little explored, and directly using natural rain to
study their effects on different DNN based perception systems is also labor-
consuming and expense-costly.

Figure 1: Framework of our methodology. Top: Adversarial rain attack and
defensive deraining. Bottom left: DNN system directly processes rainy images
and leads to false result. Bottom right: DNN system with defensive deraining
provides true result. Figure 2: Illustration of clean images (left) and
adversarial rain examples (right). First row: images from NeurIPS’17 DEV
dataset with image classifier Inception v3 [4]. Second row: images from MS
COCO dataset with object detector Faster-RCNN [5].

A common counter-measure to the rain effect on DNNs is the deraining technique
[6], which removes rain streaks from captured rainy images to restore rain-
free images. The deraining task requires large quantities of rainy/clean image
pairs for training deraining models, and the rainy images are often
artificially synthesized [7, 8, 9] due to the unavailability of rainy and
clean images for the same scene. However, the real rain has various
appearances determined by weather conditions and environment lighting, and
further undergoes visual changes presented in rainy images owning to varying
camera parameters [10]. While the synthesized rain emphasizes the rain
accumulation that well evaluates the removal capability of deraining models
and often neglects the realness and diversity of rain, resulting in a large
gap between synthetic rainy scenes and real-word rainy scenes. Even existing
rain rendering methods can to some extent generate more natural rainy images,
they are mostly used for a specific scenario [11, 12, 13, 14]. Therefore, it
is challenging or even impossible to collect or synthesize the rainy images in
all kinds of rain situations that can potentially occur in the real world, and
especially the hard rain examples that easily mislead the DNNs.

In this paper, we intend to tackle the above problems altogether from a new
angle, by taking a combined perspective of two different studies, _i.e_.,
rainy image synthesis and adversarial attack, and present an adversarial rain
attack and a defensive deraining strategy, by which we could simulate various
(potentially worst-case) rain situations with the guidance of deployed DNNs to
reveal the potential threat factors of rain, and in turn we could also train
more powerful deraining models with the adversarial rain to defend the
deployed DNNs from negative rain effect. The framework of our methodology is
shown in Fig. 1. For the adversarial rain attack, we take both adversarial
capability and rain appearance quality into consideration. The adversarial
rain is made intentionally visible, which is completely opposite to the
imperceptible adversarial noises, thus the adversarial ability has to be
achieved with stronger constraints. For this purpose, we design a factor-aware
rain generation method that simulates rain streaks following the camera
exposure process. In addition, we also model several types of controllable
rain factors, including rain intensity, rain direction and rain brightness,
all of which can be learned and tuned for adversarial attack under the
constraints of rain appearance. For the defensive deraining, we propose an
adversarial rain augmentation method, which mixes multiple adversarial rain
layers to randomize and diversify the rain cases, enhancing the performance of
deraining models. Finally, the rain-augmented deraining models can be used as
a preprocessor for follow-up perception systems.

We conduct a large-scale evaluation of our adversarial rain attack and
defensive deraining. We find that, indeed, our approaches can successfully
generate adversarial rainy images with high-quality (see some examples in Fig.
2), and can benefit the training of deraining models. Furthermore, our learned
rain factors can also be used to guide the generation of real adversarial rain
examples, demonstrating the potential threats of natural rain that should be
called attention. Our main contributions are summarized as follows:

  * •

We identify the essential problem of the inevitable rain factors to the
impacts and risks of DNNs. With a comprehensive study on the relation of rain
effect and DNNs, we propose a new type of adversarial attack by using common
rainy weather rather than adversarial noises. The proposed adversarial rain
enriches the current family of adversarial examples, in providing an important
family of physical effects that can occur in the real world.

  * •

We design a novel rain generation method that can synthesize photo-realistic
rainy images directly from random noises. The synthesized rain exhibits strong
DNN attack capability against image classification and object detection,
helping to analyze the risks of rain effects to DNNs.

  * •

We present a defensive deraining strategy, in which the synthesized rain
generated from adversarial attack can augment the rainy image datasets,
diversifying the rain situations and thus improving the deraining performance
for downstream vision tasks.

  * •

Furthermore, we also propose a simulation approach to launch a real-world
adversarial rain attack, from which the real adversarial rain is obtained by
the rain factors guided by DNNs, building a bridge between digital adversarial
rain and real adversarial rain.

The rest of this paper is organized as follows. In Section II, we summarize
the related works on adversarial examples, rain rendering and rain removal
methods. In Section III, we elaborate on the design of our proposed
adversarial rain attack and defensive deraining. In Section IV, we perform
extensive experiments to verify the effectiveness of our attack and defensive
methods. Finally, we conclude with some suggestions for future works in
Section V.

Figure 3: Overview of factor-aware rain generation. Please zoom in for noise
and rain details.

##  2 Related Work

###  2.1 Adversarial Noise Attacks

Extensive studies have shown that state-of-the-art DNNs are still vulnerable
to adversarial examples [3]. In their early study, [15] demonstrated that
intentionally designed perturbations added to images can mislead
classification results, and proposed to generate adversarial examples by using
a box-constrained L-BFGS method. To overcome the high complexity in [15], [16]
designed a fast gradient sign method (FGSM) to produce adversarial noises with
the sign of loss gradient in a one-step process. Since then, numerous
gradient-based adversarial noise attacks for image classification have been
proposed, including the attacks by developing the FGSM in multiple iterations
[17, 18, 19], integrating momentum iteratively into FGSM (MI-FGSM) [20],
employing diverse input method (DIM) in each iteration [21], optimizing
perturbations with a translation-invariant method (TIM) [22], and
incorporating variance tuning into iterative gradient [23].

Recently, adversarial examples are also applied to other computer vision
tasks. [24] proposed a dense adversary generation (DAG) algorithm to attack
object detection and semantic segmentation models, in which the loss of object
proposals with assigned false labels was minimized during gradient-descent
iterations. [25] suggested attacking the region proposal network (RPN) widely
adopted in object detectors, and designed a robust adversarial perturbation
(RAP) method. [26] generated adversarial examples by using the total loss of
both object detector and RPN. Differently, [27] proposed a unified and
efficient adversary (UEA) method for object detection based on a generative
adversarial network (GAN), and achieved high transferability and low
computation cost. Moreover, recent studies also explore physical adversarial
attacks against real-world object detectors [28, 29].

The current adversarial attacks are mainly performed under the disguise of
additive noises, which are artificial fabrications and may arouse suspicion.
We instead use common rainy weather as a natural camouflage for launching
adversarial attacks, from which we investigate the relation between attack
capability and rain factors.

###  2.2 Rain Rendering

Rain rendering refers to the generation or synthesis of photo-realistic rain
effects in computer graphics. Existing rain rendering methods can roughly fall
into two categories. The _first category_ simulates the texture patterns of
rain streaks, and post-processes the rain streaks to fit the scene [30, 31,
32]. Such methods are often simple and computationally efficient, but lack of
realistic rain appearance. The _second category_ leverages particle systems,
in which the rain rate, rain distribution, and rain velocity can be flexibly
controlled [11, 33, 12, 13, 14]. In particular, these methods consider the
physical and optical properties of rain, being able to generate visually
realistic rain effects, but at the expense of high computation costs.

Different from the above texture-based and physics-based rain rendering
methods, we generate the rain directly from random noises with the guidance of
DNNs, ensuring both high rain realism and low computation complexity. Our
noise-generated rain is also completely different from the synthesized rain
based on noises used in deraining tasks [34, 35, 9], which use a motion-blur
kernel to stretch the dotted noises to long rain streaks in one step [36]. In
contrast, we use an $N$-step translation transformation to transform the
noises into rain streaks inspired by the camera exposure process, enabling
flexible and controllable rain generation with learnable network parameters
and also effective adversarial attack capability.

###  2.3 Rain Removal

Single image rain removal is the opposite of rain rendering, and can also be
largely divided into two categories [6]. The _first category_ is model-based
methods, regarding the single image rain removal as an image decomposition or
signal separation problem between the rain layer and background layer. Typical
methods include dictionary-based sparse coding [37, 34], nonlocal mean
filtering [38], low rank representation model [39, 40], and patch-based
Gaussian mixture model [41].

The _second category_ is the recently fast-growing deep learning-based
methods, which automatically learn a non-linear mapping between rainy images
and clean images. Many DNN architectures have been proposed for rain removal.
Some representative methods include deep detail network for predicting rain
residues [35], multi-stream dense network for rain density estimation [9],
multi-task network for joint rain detection and removal [7], scale-free
network for two-stage deraining [42], progressive recurrent network for multi-
stage rain removal [43], and pixel-wise filter learning for efficient
deraining [44]. Furthermore, some recent attempts are also made by using GANs
[8, 45, 46] and semi-supervised learning [47, 48] for the deraining task.

The existing deraining models are trained on synthetic rainy images, which
cannot contain all kinds of rain cases. In the experimental section, we use
our adversarial rain to augment the rainy image datasets during the training
to improve the deraining performance.

##  3 Methodology

In this section, we first introduce the factor-aware rain generation, which
synthesizes the rainy images to be prepared for adversarial attack. Next, we
detail the implementation of adversarial rain attack on image classification
and object detection tasks. Finally, we describe the defensive deraining
strategy equipped with the adversarial rain augmentation.

###  3.1 Factor-aware Rain Generation

Natural rain is liquid water that falls visibly in small drops. Due to the
high speeds of drops and the exposure time limitation of a camera, the rain in
an image can usually be motion-blurred and displayed as longer streaks [10].
The latest rain rendering methods [14, 12] are mostly based on a ready-made
rain streak database [11] and/or a particle simulator, requiring strict
operating conditions. We instead propose a new mechanism, named factor-aware
rain generation, to generate photo-realistic rain streaks (see Fig. 3). We
directly leverage random noises as a starting point to generate rain streaks,
by taking the following consideration.

  * •

A raindrop typically has a diameter of 1-2 mm, and usually does not exceed 10
mm [10, 49]. The tiny raindrops in a far distance captured by the camera look
like image noises. Some deraining methods also treat the raindrops or rain
streaks in images as special high-frequency noises [40, 50]. These are
important physical rain properties and should be considered during method
design for better photo-realisticity.

  * •

Many synthesized rain databases commonly adopt random noises as the basis [35,
9], which can be an effective starting point.

  * •

Random noises can be easily modulated during gradient back-propagation, which
provides the chance and efficient way for the combination investigation of
worst-case rain effects to the decision of DNNs.

To generate realistic-looking rain from random noises, some properties of
noises, including noise distribution, noise strength and noise density, should
be carefully considered. The random noises are generated from a uniform
distribution ${\cal U}(a,b)$ and form a noise image with the same size as the
input clean image. Since the rain streak usually has larger brightness
intensities, the lower bound $a$ and upper bound $b$ of ${\cal U}(a,b)$ are
assigned large values (see experimental section). The noise image is then
uniformly sampled using a sampling rate $\varepsilon_{n}$ by setting the non-
selected elements to zeros, thus obtaining a sparse noise image. The sampling
rate $\varepsilon_{n}$ is the ratio of the desired number of noise elements to
the image size, and it determines the noise density of a noise image, and also
affects the rain rate in the final rainy image. The reason behind the uniform
distribution and uniform sampling is that the distribution of drops is uniform
over space and time [10].

Inspired by the formation of rain streaks in camera sensing process, we
simulate the motion-blur of the noise image as the rain effects. Since the
motion-blur is an integration of the positions of moving objects over the
period of exposure time, the motion-blur of the noise image can be simulated
by continuous translation along a specific direction.

Let $\mathbf{N}$ be a noise image, and $T_{\theta}$ the translation
transformation with translation parameters $\theta$. The transformed noise
image is denoted as ${\mathbf{N}}^{T_{\theta}}$. The $T_{\theta}$ can be
performed by using a spatial transformer network [51], and thus $\theta$
represents affine matrix parameters. To transform the noises into continuous
rain streaks, we further divide the translation transformation process into
$N$ parts. To this end, we apply an $N$-step translation transformation to
$\mathbf{N}$ using translation parameters $i\Delta\theta$, where
$\Delta\theta=\theta/N$ and $i\in\left[{0,N}\right]$. After the $N$-step
translation transformation, we obtain $N$ transformed noise images
${\mathbf{N}}^{{T_{i\Delta\theta}}}$. To smooth the rain streak formed by
summing up $N$ transformed noise images, we convolve the $N$ transformed noise
images with a kernel ${\mathbf{K}}_{p}$ with size
$(N\\!+\\!1)\\!\times\\!C\\!\times\\!1\\!\times\\!1$, where $C$ is the number
of color channels. Finally, the $p$-th rain streak ${\mathbf{R}}_{p}$ is
generated by

${{\mathbf{R}}_{p}}=g({{\mathbf{N}}_{p}},{T_{\theta}},{{\mathbf{K}}_{p}})=\sum\limits_{i,q\in[0,N]}{{\mathbf{N}}_{p}^{{T_{i\Delta\theta}}}{k_{pq}}}$
(1)

where ${\mathbf{N}}_{p}$ is the $p$-th noise element in $\mathbf{N}$,
${\mathbf{K}}_{p}$ is the kernel corresponding to ${\mathbf{N}}_{p}$, and
${k_{pq}}$ is the $q$-th weight of ${\mathbf{K}}_{p}$. All the
${\mathbf{R}}_{p}$ make up a rain layer $\mathbf{R}$.

Given a clean image $\mathbf{X}$ and a rain layer $\mathbf{R}$, the rainy
image $\mathbf{X}^{\prime}$ can be synthesized by various rain synthesis
models. Take the additive composite model [6] as an example, the rainy image
synthesis is formulated as

${\mathbf{X}^{\prime}}=h({\mathbf{X}},{\mathbf{R}})={\mathbf{X}}+{\mathbf{R}}$
(2)

where $h(\cdot,\cdot)$ represents a synthesis function.

Note that, in this initial attempt, we have not yet considered environmental
lighting for rain generation, and we use a simple scenario where the rain
streaks are thin enough to neglect the reflection and refraction of light.
More complicated and advanced can be integrated into our rain generation
process [10, 14] as future work.

The factor-aware rain generation method just provides initial rainy images,
which still lack of adversarial ability. In the following two subsections, we
use the DNNs deployed in image classification and object detection to further
guide the rain factors for producing adversarial rainy images.

Input: Clean image $\mathbf{X}$, classifier $f$ with loss function $L$, rain
factors $\mathbf{N}$, $\theta$ and ${\cal K}$

Output: Adversarial rainy image $\mathbf{X}^{\prime}$

1

2Initialize ${{\mathbf{N}}_{(1)}}\\!=\\!{\mathbf{N}}$,
${{\theta}_{(1)}}\\!=\\!{\theta}$, ${{\cal K}_{(1)}}\\!=\\!{\cal K}$,
${L_{max}}\\!=\\!0$;

3

4for  _$s=1,\cdots,S$_ do

5

6 Generate a rain layer ${\mathbf{R}}_{(s)}$ using ${\mathbf{N}}_{(s)}$,
${\theta}_{(s)}$ and ${\cal K}_{(s)}$ based on Eq. (1);

7

8 Synthesize a rainy image ${\mathbf{X}^{\prime}}_{(s)}$ using $\mathbf{X}$
and ${\mathbf{R}}_{(s)}$ based on Eq. (2);

9

10 Get the loss $L({\mathbf{X}^{\prime}}_{(s)},y)$ from
$f({\mathbf{X}^{\prime}}_{(s)})$, and calculate the gradient
${\nabla_{{\mathbf{X}^{\prime}}_{(s)}}}L({\mathbf{X}^{\prime}}_{(s)},y)$ of
$L({\mathbf{X}^{\prime}}_{(s)},y)$ w.r.t. ${\mathbf{X}^{\prime}}_{(s)}$;

11

12 Back-propagate the sign of gradient
${\nabla_{{\mathbf{X}^{\prime}}_{(s)}}}L({\mathbf{X}^{\prime}}_{(s)},y)$ to
the rain factors ${\mathbf{N}}_{(s)}$, ${\theta}_{(s)}$ and ${\cal K}_{(s)}$,
and update them as ${\mathbf{N}}_{(s+1)}$, ${\theta}_{(s+1)}$ and ${\cal
K}_{(s+1)}$ via a gradient ascent method;

13

14 if  _$L({\mathbf{X}^{\prime}}_{(s)},y) >{L_{max}}$_ then

15 ${L_{max}}=L({\mathbf{X}^{\prime}}_{(s)},y)$;

16 ${{\mathbf{X}^{\prime}}_{opt}}={{\mathbf{X}^{\prime}}_{(s)}}$;

17

18 end if

19

20 end for

21return ${{\mathbf{X}}^{\prime}}={{\mathbf{X}^{\prime}}_{opt}}$;

Algorithm 1 Adversarial Rain Attack on Image Classification

###  3.2 Adversarial Rain Attack on Classification

Given a classifier $f({\mathbf{X}}):{\mathbf{X}}\in{\cal X}\to y\in{\cal Y}$
that maps an input image $\mathbf{X}$ to a ground-truth class label $y$. The
adversarial attack aims to generate a constrained example
$\mathbf{X}^{\prime}$ that can mislead $f({\mathbf{X}^{\prime}})$ to output a
false label, usually by maximizing the loss $L({\mathbf{X}^{\prime}},y)$.

In conventional adversarial noise attacks, the adversarial example
$\mathbf{X}^{\prime}$ is typically generated by adding adversarial noises to
$\mathbf{X}$. For our adversarial rain attack, we generate
$\mathbf{X}^{\prime}$ by using the rain generation (Eq. (1)) and rain
synthesis model (Eq. (2)). To enable successful adversarial attacks of rainy
images for image classification, we need to optimize the rain factors under
the constraint of rain appearance. There are three types of rain factors
should be considered: noise $\mathbf{N}$, translation parameters $\theta$ and
kernels ${\cal
K}=\left\\{{{{\mathbf{K}}_{p}}|p\in\left[{1,{{\left\|{\mathbf{N}}\right\|}_{0}}}\right]}\right\\}$.
The $\mathbf{N}$ affects the rain intensity, $\theta$ controls the direction
of rain streaks, and $\cal K$ determines the brightness and smoothness of rain
streaks. The adversarial rainy images can be generated by solving the
following constrained optimization problem:

$\displaystyle\\!\mathop{\operatornamewithlimits{arg\,max}}\limits_{{\mathbf{N}},\,\theta,\,{\cal
K}}L(h({{\mathbf{X}}_{p}},\sum\limits_{i,q\in[0,N]}{{\mathbf{N}}_{p}^{{T_{i\Delta\theta}}}{k_{pq}}}),\;y)$
(3) $\displaystyle{\rm{s.t.}}\quad\forall
p,\;{\left\|{\mathbf{N}}\right\|_{0}}\leq{\varepsilon_{n}}M,\;\theta\leq{\varepsilon_{\theta}},\;{k_{pq}}\leq{\varepsilon_{k}}$

where $M$ is the number of the pixels in a clean image, $\varepsilon_{n}$ is
the sampling rate mentioned earlier, and $\varepsilon_{\theta}$ and
$\varepsilon_{k}$ denote the thresholds for $\theta$ and $k_{pq}$. For natural
rain, the rain streaks often exhibit certain ranges of intensity (relevant to
rain rate), directions (affected by wind) and brightness (depending on
environment lighting and camera settings). The $\varepsilon_{n}$,
$\varepsilon_{\theta}$ and $\varepsilon_{k}$ are used to curb the rain factors
to ensure realistic looking rain, and their values will be discussed in the
experimental section.

During the rain generation process, we can calculate the gradient of the loss
function w.r.t. all rain factors, and tune the rain factors along a gradient
ascent direction for adversarial attacking. Therefore, the adversarial rain
attack can be integrated into current gradient based adversarial noise attack
frameworks, such as FGSM [16], MI-FGSM [20], DIM [21] and TIM [22], _etc_.

Algorithm 1 presents the details of our adversarial rain attack for the image
classification task. In particular, each iteration generates a loss, a rain
layer and a rainy image. Eventually, only the rainy image with the largest
loss in $S$ iterations is selected as the final adversarial rainy image.

Input: Clean image $\mathbf{X}$, detoctor $f$ with loss function $L_{j}$
($j=1,\cdots,J$), rain factors $\mathbf{N}$, $\theta$ and ${\cal K}$

Output: Adversarial rainy image $\mathbf{X}^{\prime}$

1

2Initialize ${{\mathbf{N}}_{(1)}}\\!=\\!{\mathbf{N}}$,
${{\theta}_{(1)}}\\!=\\!{\theta}$, ${{\cal K}_{(1)}}\\!=\\!{\cal K}$,
${L_{sum}}\\!=\\!0$, ${L_{min}}\\!=\\!{+\infty}$;

3

4for  _$s=1,\cdots,S$_ do

5

6 Generate a rain layer ${\mathbf{R}}_{(s)}$ using ${\mathbf{N}}_{(s)}$,
${\theta}_{(s)}$ and ${\cal K}_{(s)}$ based on Eq. (1);

7

8 Synthesize a rainy image ${\mathbf{X}^{\prime}}_{(s)}$ using $\mathbf{X}$
and ${\mathbf{R}}_{(s)}$ based on Eq. (2);

9

10 for  _$j=1,\cdots,J$_ do

11

12 Get the loss $L_{j}({\mathbf{X}^{\prime}}_{(s)},y)$ from
$f({\mathbf{X}^{\prime}}_{(s)})$, and calculate the gradient
${\nabla_{{\mathbf{X}^{\prime}}_{(s)}}}L_{j}({\mathbf{X}^{\prime}}_{(s)},y)$
of $L_{j}({\mathbf{X}^{\prime}}_{(s)},y)$ w.r.t.
${\mathbf{X}^{\prime}}_{(s)}$;

13

14 Back-propagate the sign of gradient
${\nabla_{{\mathbf{X}^{\prime}}_{(s)}}}L_{j}({\mathbf{X}^{\prime}}_{(s)},y)$
to the rain factors ${\mathbf{N}}_{(s)}$, ${\theta}_{(s)}$ and ${\cal
K}_{(s)}$, and update them as ${\mathbf{N}}_{(s+1)}$, ${\theta}_{(s+1)}$ and
${\cal K}_{(s+1)}$ using a gradient descent method;

15

16 $L_{sum}=L_{sum}+L_{j}({\mathbf{X}^{\prime}}_{(s)},y)$

17 end for

18

19 if  _$L_{sum} <{L_{min}}$_ then

20 ${L_{min}}=L_{sum}$;

21 ${{\mathbf{X}^{\prime}}_{opt}}={{\mathbf{X}^{\prime}}_{(s)}}$;

22

23 end if

24

25 end for

26return ${{\mathbf{X}}^{\prime}}={{\mathbf{X}^{\prime}}_{opt}}$;

Algorithm 2 Adversarial Rain Attack on Object Detection

###  3.3 Adversarial Rain Attack on Object Detection

In this subsection, we further extend our adversarial rain attack to the
object detection task. Given an object detector
$f({\mathbf{X}}):{\mathbf{X}}\in{\cal X}\to t\in{\cal T}$ that maps an input
image $\mathbf{X}$ to a recognition target $t$. The $t$ can be of any
intermediate outputs or final outputs of a detector, not limited to the class
labels [24], such as final bounding-boxes [26], confidence scores and the
bounding-boxes of the object proposals in an region proposal network (RPN)
widely adopted in detectors [25]. All these types of targets can be utilized
to generate constrained adversarial examples $\mathbf{X}^{\prime}$ to fool
$f({\mathbf{X}^{\prime}})$. This process can be accomplished by minimizing the
loss $L({\mathbf{X}^{\prime}},{\cal T}^{\prime})$, where ${\cal T}^{\prime}$
is a set of one type of adversarial targets $t^{\prime}$.

Similar to the image classification task, the adversarial rainy images for
object detection can be generated by

$\displaystyle\\!\mathop{\operatornamewithlimits{arg\,min}}\limits_{{\mathbf{N}},\,\theta,\,{\cal
K}}\sum\limits_{j}{{L_{j}}(h({{\mathbf{X}}_{p}},\sum\limits_{i,q\in[0,N]}{{\mathbf{N}}_{p}^{{T_{i\Delta\theta}}}{k_{pq}}}),\;{{{\cal
T}_{j}}^{\prime}})}$ (4) $\displaystyle{\rm{s}}{\rm{.t}}{\rm{.}}\quad\forall
p,\;{\left\|{\mathbf{N}}\right\|_{0}}\leq{\varepsilon_{n}}M,\;\theta\leq{\varepsilon_{\theta}},\;{k_{pq}}\leq{\varepsilon_{k}}$

where $j$ is the index of adversarial target types (_e.g_., class labels,
bounding-boxes, or confidence scores). One or more types of adversarial target
set ${{\cal T}_{j}}^{\prime}$ can be considered for the total loss function,
in which $L_{j}$ denotes the loss function corresponding to ${{\cal
T}_{j}}^{\prime}$.

Similarly, we can tune the rain factors via a gradient descent algorithm, and
also apply the adversarial rain attack to current gradient based adversarial
noise attacks for object detection, such as DAG [24] and RAP [25], _etc_. The
adversarial rain attack on object detection is summarized in Algorithm 2, and
the attacking process is similar to that of image classification.

Input: Clean image $\mathbf{X}$, rainy image ${\mathbf{X}}^{\rm{r}}$

Output: Augmented rainy image ${\mathbf{X}}^{\rm{ar}}$

1

2Initialize an empty rain layer ${\mathbf{R}}$ with all zeros;

3Sample mixing weights
$\left({{w_{1}},{w_{2}},\cdots,{w_{k}}}\right)\sim{\rm{Dirichlet}}$;

4

5for  _$i=1,\cdots,k$_ do

6

7 Generate an Adversarial rain layer ${\mathbf{R}}_{i}$ with new noise
initialization;

8

9 ${\mathbf{R}}={\mathbf{R}}+{w_{i}}{\mathbf{R}}_{i}$;

10

11 end for

12

13Sample
${\hat{\mathbf{X}}}\sim\left\\{{\mathbf{X}},{\mathbf{X}}^{\rm{r}}\right\\}$;

14

15return ${\mathbf{X}}^{\rm{ar}}={\hat{\mathbf{X}}}+{\mathbf{R}}$;

Algorithm 3 Adversarial Rain Augmentation

###  3.4 Defensive Deraining

Rain causes severe visual degradation to captured images and thus poses a risk
for follow-up perception tasks, so rain removal is quite necessary for DNN
based perception systems in practical applications. However, existing
deraining models are always trained on synthesized rainy images in limited
situations, which cannot exhaustively represent all the possible rain patterns
(especially the hard rain examples for DNNs), thus limiting the deraining
performance.

In previous subsections, we have elaborated on the adversarial rain attack for
different vision tasks, and this motivates us to use it as a defense to
improve the performance of DNN perception. Our defensive deraining strategy
consists of two stages: we first leverage the adversarial rainy images to
augment the rain dataset to train more powerful deraining models, and then we
use the trained deraining model to remove the rain streaks in the rainy images
which are inputs for DNNs. The challenge of the defensive deraining strategy
is to augment the rain dataset with enough diverse rain situations, which
should encourage the deraining models to focus more on the hard-to-remove rain
streaks that are more important for perception recognition.

To address the challenge and to close the gap between synthesized rain and
real rain for training de-raining models, we propose an adversarial rain
augmentation method, which is inspired by AugMix [52]. Specifically, for each
rainy/clean image pair during the training, we generate $k$ different
adversarial rain layers for the clean image to diversify the rain cases. Then
we aggregate the adversarial rain layers with the weights sampled from a
Dirichlet distribution for a random blending. Finally we obtain the augmented
rainy image by adding the aggregated adversarial rain layer to the clean image
or rainy image. Different from AugMix [52] that mixes basic transformations to
improve model robustness, our adversarial rain augmentation uses mixed
adversarial rain layers, aiming to diversify the rain to match the real rain
situations. The detailed data augmentation process is described in Algorithm
3. Our augmentation method can be applied to existing rain datasets, and see
the experiments in Section 4.8.

Figure 4: The effect of ${\cal U}(a,b)$ on adversarial capability.

##  4 Experiments

###  4.1 Experimental Setup

####  4.1.1 Datasets

We perform a comprehensive evaluation of the adversarial rain attack on two
different tasks: image classification and object detection. For image
classification, we use NeurIPS’17 adversarial competition dataset DEV [53] for
experiments, which consists of 1,000 images, and is compatible with ImageNet.
For object detection, we perform experiments on MS COCO 2014 minival split
[54] and KITTI object benchmark [55], which contain 5,000 images and 7,480
images, respectively. We also use rain datasets Rain100H [7], Rain800 [8],
Rain1200 [9] and Rain1400 [35] for rain appearance comparison and deraining
experiments.

TABLE I: The effect of $\varepsilon_{n}$, $\varepsilon_{\theta}$, and
$\varepsilon_{k}$ on adversarial capability.

$\varepsilon_{n}$ | 0.001 | 0.0025 | 0.005 | 0.0075 | 0.01 | 0.02
---|---|---|---|---|---|---
Success rate | 81.5 | 90.3 | 98.6 | 99.4 | 99.8 | 99.9
$\varepsilon_{\theta}$ | 0.01 | 0.05 | 0.1 | 0.2 | 0.3 | 0.4
Success rate | 85.4 | 92.2 | 96.0 | 98.6 | 99.2 | 9.5
$\varepsilon_{k}$ | 0.05 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5
Success rate | 82.3 | 89.8 | 94.7 | 98.6 | 99.7 | 99.9

TABLE II: The success rates (in %) of normal rain and AdvRain under the same
rain magnitude on NeurIPS’17 DEV.

Rain Methods | Threat Models (Inc-v3)
---|---
Inc-v3 | Inc-v4 | IncRes-v2 | Xception
NormalRain1 | 39.3 | 38.5 | 36.4 | 38.8
AdvRain1 | 81.5 | 46.7 | 41.8 | 53.0
NormalRain2 | 45.5 | 43.2 | 41.6 | 44.0
AdvRain2 | 98.6 | 54.3 | 50.1 | 62.5

Figure 5: Visual comparison of our adversarial rainy images (a-c) and other
synthesized rainy images from Rain100H [7], Rain800 [8], Rain1200 [9],
Rain1400 [35] and physics-based rain rendering [14] (d-h).

####  4.1.2 Threat Models

To evaluate our adversarial rain attack on image classification, we exploit
four publicly-available pre-trained models, including Inception v3 (Inc-v3)
[4], Inception v4 (Inc-v4), Inception ResNet v2 (IncRes-v2) [56], and Xception
[57]. The threat model for object detection is Faster RCNN (FR) [5] with
different backbones, including VGG16 (v16) [58], MobileNet (mn) [59], ResNet50
(rn50), ResNet101 (rn101) and ResNet152 (rn152) [60].

####  4.1.3 Baselines

We compare the adversarial rain attack with six adversarial noise attacks on
image classification: FGSM [16], MI-FGSM [20], DIM [21] and TIM’s three
variants TI-FGSM, TI-MI-FGSM, and TI-DIM [22]. The comparative methods used
for object detection are RAP [25] and UEA [27].

####  4.1.4 Evaluation Metrics

We use attack success rate and mean average precision (mAP) as major metrics
to measure the adversarial capability on image classification and object
detection, respectively.

####  4.1.5 Implementation Details

To attack image classifiers and object detectors, we individually generate
adversarial rain examples by integrating with different existing adversarial
attacks (MI-FGSM [20], DIM [21] and TI-MI-FGSM [22] for classification, and
RAP [25] for detection). All the adversarial rain examples are obtained in
$S=20$ iterations, and the other hyper-parameters are consistent with the
default settings in [20, 21, 22, 25].

Figure 6: User study of visual realism for different rainy images.

###  4.2 Ablation Study

####  4.2.1 Rain Factors

We first conduct ablation experiments to study the impact of rain factors on
adversarial rain attack. When considering one rain factor, the other rain
factors are set as default values and remain unchanged. For simplicity, all
the ablation experiments are only performed on the image classification task.

As described in Section 3, the adversarial rain is generated from random
noises that follow a normal distribution ${\cal U}(a,b)$, and further
optimized under the constraints related to thresholds $\varepsilon_{n}$,
$\varepsilon_{\theta}$ and $\varepsilon_{k}$. We first show in Fig. 4 the
effects of different combinations of lower bound $a$ and upper bound $b$ in
${\cal U}(a,b)$ on the adversarial capability. We observe that the
combinations of $a$ and $b$ with larger values contribute to a higher attack
success rate due to larger noise strength. A larger distribution interval
increases noise diversity, and thus enriches the texture pattern of rain
appearance. Eventually, we choose ${\cal U}(0.7,1.0)$ as a suitable
compromise.

The $\varepsilon_{n}$ and $\varepsilon_{\theta}$ are scalar thresholds, and
the $\varepsilon_{k}$ is a vector containing thresholds for translation
parameters in affine transform. We report the effects of three types of
thresholds $\varepsilon_{n}$, $\varepsilon_{\theta}$ and $\varepsilon_{k}$ on
the adversarial capability in Table II. We can see that the adversarial rain
attack achieves higher attack success rates at larger $\varepsilon_{n}$,
$\varepsilon_{\theta}$ and $\varepsilon_{k}$, indicating that higher rain
intensity, longer rain streaks and brighter rain textures affect the DNNs more
severely. However, the attack success rate saturates at larger threshold
values, which also degrades the rain appearance. Therefore, we select
$\varepsilon_{n}=0.005$, $\varepsilon_{\theta}=0.2$ and $\varepsilon_{k}=0.3$
for following experiments.

####  4.2.2 Adversarial Rain v.s. Normal Rain

To validate whether or not the high attack performance of our adversarial rain
is gained by the large magnitude of the perturbation, we compare the
classification results of normal rain [11] and adversarial rain under the same
rain magnitude in Table II.

Specifically, we consider two rain magnitudes for the normal rain and
adversarial rain, and the magnitudes are decided by the area of rain streaks
in the binary rain maps. The AdvRain1 and AdvRain2 denote the adversarial rain
with noise density $\varepsilon_{n}=0.001$ and $\varepsilon_{n}=0.005$
respectively, and NormalRain1 and NormalRain2 have the same rain magnitudes
corresponding to AdvRain1 and AdvRain2. We can see from Table II that the
adversarial rain achieves consistently higher attack success rates than the
normal rain in all cases, indicating that our superior attack performance does
come from adversarially learned rain factors rather than large rain magnitude.

TABLE III: The success rates (in %) of various adversarial attack methods
against image classifiers on NeurIPS’17 DEV.

Attack Methods | Threat Models (Inc-v3)
---|---
Inc-v3 | Inc-v4 | IncRes-v2 | Xception
FGSM [16] | 79.6 | 35.9 | 30.6 | 42.1
MI-FGSM [20] | 97.8 | 47.1 | 46.4 | 47.7
AdvRain-MI-FGSM | 98.6 | 54.3 | 50.1 | 62.5
DIM [21] | 98.3 | 73.8 | 67.8 | 71.6
AdvRain-DIM | 98.5 | 76.4 | 69.5 | 74.2
TI-FGSM [22] | 75.4 | 37.3 | 32.1 | 38.6
TI-MI-FGSM [22] | 97.9 | 52.4 | 47.9 | 44.6
TI-DIM [22] | 98.5 | 75.2 | 69.2 | 61.3
AdvRain-TI-MI-FGSM | 98.8 | 79.8 | 74.3 | 66.2

TABLE IV: The success rates (in %) of various adversarial attack methods
against defense models on NeurIPS’17 DEV.

Attack Methods | Threat Models (Inc-v3)
---|---
Inc-v3ens3 | Inc-v3ens4 | IncRes-v2ens
FGSM [16] | 13.7 | 14.5 | 6.8
MI-FGSM [20] | 15.9 | 16.3 | 7.0
AdvRain-MI-FGSM | 65.7 | 68.2 | 59.2
DIM [21] | 17.9 | 21.8 | 9.7
AdvRain-DIM | 67.6 | 69.5 | 61.4
TI-FGSM [22] | 15.2 | 15.7 | 10.2
TI-MI-FGSM [22] | 22.8 | 24.6 | 14.8
TI-DIM [22] | 29.6 | 31.9 | 22.0
AdvRain-TI-MI-FGSM | 69.8 | 69.1 | 63.3

TABLE V: The mAP (in %) of various adversarial attack methods against object
detection on COCO.

Attack Methods | Threat Models (FR-v16)
---|---
FR-v16 | FR-mn | FR-rn50 | FR-rn101 | FR-rn152
Clean | 43.8 | 41.1 | 52.4 | 55.0 | 56.8
RAP [25] | 3.3 | 9.8 | 24.6 | 29.8 | 22.3
UEA [27] | 5.7 | 11.5 | 25.8 | 31.3 | 24.1
AdvRain | 2.2 | 8.3 | 21.5 | 27.5 | 19.7

TABLE VI: The mAP (in %) of various adversarial attack methods against object
detection on KITTI.

Attack Methods | Threat Models (FR-v16)
---|---
FR-v16 | FR-mn | FR-rn50 | FR-rn101 | FR-rn152
Clean | 74.6 | 72.3 | 75.8 | 78.1 | 78.4
RAP [25] | 8.7 | 17.5 | 20.4 | 23.1 | 23.9
UEA [27] | 10.3 | 21.6 | 23.2 | 25.0 | 26.3
AdvRain | 5.6 | 11.8 | 15.4 | 17.8 | 18.2

###  4.3 Evaluation of Rain Appearance

####  4.3.1 Visual Effect

We compare the visual effect of our adversarial rainy images with that of
other synthesized rainy images in Fig. 5. We observe that our adversarial
rainy images show more realistic visual effects than the synthesized rainy
images in databases Rain100H [7], Rain800 [8], Rain1200 [9] and Rain1400 [35],
which exhibit irregular rain directions or snow-like rain streaks. In
comparison with Fig. 5 (c) and Fig. 5 (h), the physics-based rain [14]
produces relatively more realistic appearances than ours. This is because the
physics-based rainy images are synthesized by warping the rain streaks in a
ready-made rain database [11] with a physical particle system [61], and also
use environmental lighting for rain rendering. While our rainy images are
generated directly from random noises, with which the adversarial ability
should also be taken into consideration. We believe the visual effect of our
adversarial rainy images can be further improved by employing lighting
information, which we leave as future work.

####  4.3.2 User study

We also conduct a user study to further evaluate the rain appearance following
[14]. Specifically, we randomly select 10 images from each rain database
(_i.e_., our adversarial rainy images and other five synthesized rain
databases), in collecting a total of 60 rainy images. Then, we invite 30
participants (12 undergraduate students, 18 graduate students; 13 female, 17
male; aged 20 to 34) to rate the visual effect of rainy images using a
5-points Likert scale. The results are summarized in Fig. 6. We can see that
our adversarial rainy images and physics-based rainy images obtain similar
high voting scores compared with other types of rainy images, confirming the
visual realism of our rain generation method.

###  4.4 Attacking Image Classification

We report the comparison results of different adversarial attacks for image
classification in Table IV. All adversarial attack methods are targeted at
Inc-v3 (white-box attack), and also transferred to Inc-v4, IncRes-v2 and
Xception (black-box attack). For the white-box attack, the AdvRain achieves
the highest success rate than all comparative methods. For the black-box
attack, our AdvRain also has better transferability than conventional
adversarial noise attacks.

We also compare our AdvRain with baselines on three defense models
(Inc-v3ens3, Inc-v3ens4 and IncRes-v2ens) based on adversarial training from
[62], and the results are listed in Table IV. We can see that our AdvRain
still clearly outperforms its competitors in attacking adversarially trained
models, which further demonstrates the advantages of our method.

###  4.5 Attacking Object Detection

Table VI and Table VI summarize the performance of various adversarial attack
methods against object detection on COCO and KITTI, respectively. For attack
methods, we use FR-v16 as a targeted model, and then perform the transfer
attack to FR-mn, FR-rn50, FR-rn101, and FR-rn152. For COCO and KITTI datasets,
the performance is measured by mAP with IoU 0.7 and with the PASCAL criteria.
As we can see in the tables, the AdvRain consistently outperforms the RAP and
UEA across all cases. Specifically, when attacking the white-box model FR-v16,
the mAP score of FR-v16 drops significantly from 74.6% to 5.6%.

Figure 7: Attack results in the real world. (a) Clean image. (b) AdvRain. (c)
Real AdvRain. Figure 8: Qualitative comparisons between RAP [25], UEA [27] and
our AdvRain. Please zoom in to see the details.

###  4.6 Real-world Adversarial Rain Attack

The adversarial rain attack produces adversarial examples by using the rain
factors guided by DNNs. The rain factors, including rain noises $\mathbf{N}$,
translation parameters $\theta$, and kernels ${\cal K}$, determine the
adversarial capability of a specific rainy image. Therefore, a natural
question arises, _i.e_., whether it is possible to use the obtained rain
factors to adjust the appearance and dynamics of real rain so that to enable
the rain to fool the DNNs. The resulting rain can be regarded as a type of
real-world adversarial rain attack, which is performed in the following steps.

  1. 1.

Fix the objects to be classified or detected by DNNs, and capture an image
with a camera.

  2. 2.

Generate the corresponding rainy image with our adversarial rain attack, and
also obtain the rain factor values.

  3. 3.

Leverage the rain factors to adjust the simulated rain to achieve adversarial
capability.

For this experiment, we perform a physical behavior study in the real-world
environment by our authors, using water-drops from cans to emulate the real
rain effects. In particular, we spray the water by a watering can so that the
water drops are freely falling from a higher place. The desired rain intensity
($\mathbf{N}$) can be achieved by adjusting the amount of water drops, and the
rain directions ($\theta$) are controlled by the wind speed and wind direction
of an electric fan. The kernels ${\cal K}$ cannot be obtained in the real
world, so we do not consider adjusting the rain textures.

We use the Faster-RCNN to detect the clean image, adversarial rainy image and
real-world adversarial rainy image, the results of which are illustrated in
Fig. 7. We observe that both two types of adversarial rainy images can fool
the object detector, demonstrating that the rain factors can be used as
guidance for launching real-world adversarial attacks. The appearances of two
adversarial rainy images are not very similar, but this problem can be
adjusted by using more sophisticated rain simulation equipment.

Yet, one might argue that the real natural rain cannot be controlled with
specific rain factors. Since the patterns and dynamics of natural rain are
varied and diverse, the rain effect on DNNs is unpredictable. Our experiment
is not to control the movement of natural rain, but yet to reveal the fact
that the natural rain which thwarts the DNNs can be simulated by using the
rain factors guided by the DNNs.

TABLE VII: Deraining performance in terms of PSNR and SSIM on Rain100H and
Rain1400.

Deraining Models | Rain100H | Rain1400
---|---|---
PSNR | SSIM | PSNR | SSIM
PreNet [43] | 29.44 | 0.8930 | 32.56 | 0.9452
PreNet-Flip-Rotation | 29.45 | 0.8931 | 32.58 | 0.9454
PreNet-AugMix | 30.10 | 0.8976 | 32.29 | 0.9468
PreNet-AdvRain | 30.65 | 0.9012 | 32.95 | 0.9480
EfDeRain [44] | 30.35 | 0.8971 | 32.06 | 0.9248
EfDeRain-Flip-Rotation | 30.37 | 0.8976 | 32.10 | 0.9255
EfDeRain-AugMix | 30.62 | 0.9032 | 32.49 | 0.9278
EfDeRain-AdvRain | 31.04 | 0.9085 | 32.72 | 0.9303

TABLE VIII: The top 1 accuracy rates (in %) of image classifiers on various
types of images (clean images, rainy images and derained images).

Deraining Models | Inc-v3 | Inc-v4 | IncRes-v2 | Xception
---|---|---|---|---
Clean images | 95.9 | 97.1 | 99.8 | 96.0
Rainy images | 54.5 | 56.8 | 58.4 | 56.0
PreNet [43] | 82.4 | 84.3 | 85.6 | 84.7
PreNet-AdvRain | 85.6 | 87.4 | 88.5 | 87.2
EfDeRain [44] | 82.9 | 84.5 | 85.8 | 85.2
EfDeRain-AdvRain | 85.8 | 88.0 | 88.7 | 87.6

Figure 9: Visual comparison of deraining models with and without adversarial
rain augmentation. The number below the figure indicates PSNR / SSIM values,
and the yellow arrow in the figure shows the main difference between de-rained
images.

###  4.7 Qualitative Comparisons of Adversarial Examples

We illustrate the qualitative comparisons between the adversarial examples of
RAP [25], UEA [27] and our AdvRain in Fig. 8. For simplicity, we only compare
the adversarial examples using FR-v16 (Faster RCNN with VGG16) [5] on COCO
dataset [54] in a white-box attack scenario.

For the RAP, the adversarial noises are minimal perturbations which are mostly
concentrated in the object areas to be detected. This type of small
adversarial noises is determined by the perception constraints in terms of
PSNR. However, the RAP still leaves perceptible artifacts appeared in the
smooth texture images (see the third and the fourth rows in Fig. 8 for RAP),
implying that the conventional adversarial noises are not applicable for all
kinds of images. For the UEA, the adversarial noises are irregular latticed
patterns, which are mainly located in object areas and also scattered
throughout the images. The noise pattern in UEA is easily noticeable, and this
is due to the generative mechanism of UEA which uses a pix2pix GAN [63] to
generate adversarial examples. Our adversarial rainy images are synthesized by
compositing learned rain layers and images instead of directly adding noises
to images, so they have high visual realism in spite of the visibility of
rain.

###  4.8 Evaluation of Defensive Deraining

In this subsection, we first evaluate the effectiveness of adversarial rain
augmentation on training deraining models. Besides, to explore how the
deraining models trained with our adversarial rain augmentation could improve
the performance of DNN perception, we evaluate the classification accuracy on
rainy images with and without deraining.

For the deraining task, we use PreNet [43] and EfDeRain [44] as baselines, and
report the deraining results on dataset Rain100H [7] and Rain1400 [35] in
Table VII. To verify the superiority of our adversarial rain augmentation , we
also use two conventional data augmentation methods for comparison, one is
AugMix [52] using multiple normal rainy layers, and the other one is flip and
rotation using a single normal rainy layer.

We can see from Table VII that for Flip&Rotation and AugMix, the improvement
of deraining performance is quite marginal by just adding more synthesized
rainy images into training. This is because the training data not only need
rainy images with more diverse rain situations, but also need to cover the
tough and formidable rainy cases which can more easily fool the DNNs. The
latter requirement can only be satisfied by our AdvRain. In contrast, the
deraining models trained with our adversarial rain (indicated by suffix
AdvRain) consistently achieves substantial PSNR and SSIM gains over the
corresponding baseline models, confirming that the adversarial rain does
simulate more rainy patterns and provide more hard rain examples, thus
improving the deraining performance. Furthermore, we also provide some visual
examples from dataset Rain100H [7] in Fig. 9, from which we can see that the
deraining models with adversarial rain augmentation produce more clear images
with more detail patterns.

Having demonstrated the advantage of our adversarial rain augmentation on
deraining, we further use the trained deraining models to improve the
performance of image classification by removing the rain streaks in images
before they are fed to classifiers. We use the images in DEV dataset as clean
images, and use the NormalRain2 in Section 4.2.2 as rainy images. We utilize
four image classifiers (Inc-v3, Inc-v4, IncRes-v2 and Xception) to evaluate
the effectiveness of deraining, and report the top 1 accuracy rates in Table
VIII, in which the results in the last four rows are obtained from the
derained images with different deraining models.

We observe from Table VIII that the image classifiers all achieve high
accuracy rates on clean images, but have low accuracy rates on rainy images.
After removing the rain streaks in the rainy images, the image classifiers
significantly increase the accuracy rates. Most notably, the derained images
using our augmented deraining model are more easily to be identified by image
classifiers, indicating that our defensive deraining with adversarial rain
augmentation is indeed beneficial to the DNN perception systems.

##  5 Conclusion

In this paper, we comprehensively study the potential risks of rain effect on
DNN perception. We propose a new type of adversarial rain attack, which
simulates natural rain situations with the guidance of DNNs and thus
synthesizes visually realistic rainy images to mislead image classification
and object detection. The adversarial rain attack reveals the essential and
inevitable threat factors of rain, which commonly exist in the real world. We
also study the elimination of of rain effect on DNNs, and propose a defensive
deraining strategy that leverages adversarial rain augmentation to enhance the
deraining models for downstream perception tasks. We conducted extensive
experiments to validate the effectiveness of our approaches on different
datasets and tasks. Our results show that the current state-of-the-art DNNs
can be vulnerable to the inevitable rain effects in the real-world, and the
proposed adversarial rain can also be used to augment the rainy image
databases for deraining tasks, all of which call the attention to take rain
effects into consideration for more advanced design of real-world DNN-based
perception systems.

In future work, we would like to further optimize our adversarial rain by
considering lighting conditions and fog-like rains, and also extend the
adversarial rain attack to a more broad range of applications, such as
semantic segmentation [64] and visual object tracking [65].

## References

  * [1] M. Zhang, Y. Zhang, L. Zhang, C. Liu, and S. Khurshid, “DeepRoad: Gan-based metamorphic testing and input validation framework for autonomous driving systems,” in _IEEE/ACM International Conference on Automated Software Engineering_ , 2018, pp. 132–142.
  * [2] C. H. Bahnsen and T. B. Moeslund, “Rain removal in traffic surveillance: Does it matter?” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 20, no. 8, pp. 2802–2819, 2018.
  * [3] X. Yuan, P. He, Q. Zhu, and X. Li, “Adversarial examples: Attacks and defenses for deep learning,” _IEEE transactions on neural networks and learning systems_ , vol. 30, no. 9, pp. 2805–2824, 2019.
  * [4] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2818–2826.
  * [5] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in _Advances in neural information processing systems_ , 2015, pp. 91–99.
  * [6] W. Yang, R. T. Tan, S. Wang, Y. Fang, and J. Liu, “Single image deraining: From model-based to data-driven and beyond,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [7] W. Yang, R. T. Tan, J. Feng, J. Liu, Z. Guo, and S. Yan, “Deep joint rain detection and removal from a single image,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1357–1366.
  * [8] H. Zhang, V. Sindagi, and V. M. Patel, “Image de-raining using a conditional generative adversarial network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2019.
  * [9] H. Zhang and V. M. Patel, “Density-aware single image de-raining using a multi-stream dense network,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 695–704.
  * [10] K. Garg and S. K. Nayar, “Vision and rain,” _International Journal of Computer Vision_ , vol. 75, no. 1, pp. 3–27, 2007.
  * [11] ——, “Photorealistic rendering of rain streaks,” _ACM Transactions on Graphics (TOG)_ , vol. 25, no. 3, pp. 996–1002, 2006.
  * [12] C. Creus and G. A. Patow, “R4: Realistic rain rendering in realtime,” _Computers & Graphics_, vol. 37, no. 1-2, pp. 33–40, 2013.
  * [13] Y. Weber, V. Jolivet, G. Gilet, and D. Ghazanfarpour, “A multiscale model for rain rendering in real-time,” _Computers & Graphics_, vol. 50, pp. 61–70, 2015.
  * [14] S. S. Halder, J.-F. Lalonde, and R. d. Charette, “Physics-based rendering for improving robustness to rain,” in _IEEE International Conference on Computer Vision_ , 2019, pp. 10 203–10 212.
  * [15] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _International Conference on Learning Representations_ , 2014, pp. 1–9.
  * [16] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _International Conference on Learning Representations_ , 2015, pp. 1–11.
  * [17] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” in _International Conference on Learning Representations_ , 2017, pp. 1–17.
  * [18] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE symposium on security and privacy (S &P)_, 2017, pp. 39–57.
  * [19] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” in _International Conference on Learning Representations_ , 2018.
  * [20] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, “Boosting adversarial attacks with momentum,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 9185–9193.
  * [21] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille, “Improving transferability of adversarial examples with input diversity,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2730–2739.
  * [22] Y. Dong, T. Pang, H. Su, and J. Zhu, “Evading defenses to transferable adversarial examples by translation-invariant attacks,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4312–4321.
  * [23] X. Wang and K. He, “Enhancing the transferability of adversarial attacks through variance tuning,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1924–1933.
  * [24] C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie, and A. Yuille, “Adversarial examples for semantic segmentation and object detection,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 1369–1378.
  * [25] Y. Li, D. Tian, X. Bian, and S. Lyu, “Robust adversarial perturbation on deep proposal-based models,” in _The British Machine Vision Conference_ , 2018\.
  * [26] Y. Wang, K. Wang, Z. Zhu, and F.-Y. Wang, “Adversarial attacks on Faster R-CNN object detector,” _Neurocomputing_ , vol. 382, pp. 87–95, 2020.
  * [27] X. Wei, S. Liang, N. Chen, and X. Cao, “Transferable adversarial attacks for image and video object detection,” in _The 28th International Joint Conference on Artificial Intelligence_ , 2019, pp. 954–960.
  * [28] S.-T. Chen, C. Cornelius, J. Martin, and D. H. P. Chau, “Shapeshifter: Robust physical adversarial attack on faster R-CNN object detector,” in _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_ , 2018, pp. 52–68.
  * [29] L. Huang, C. Gao, Y. Zhou, C. Xie, A. L. Yuille, C. Zou, and N. Liu, “Universal physical camouflage attacks on object detectors,” in _IEEE International Conference on Computer Vision_ , 2020, pp. 720–729.
  * [30] N. Wang and B. Wade, “Rendering falling rain and snow,” in _ACM SIGGRAPH 2004 Sketches_. ACM, 2004, p. 14.
  * [31] L. Wang, Z. Lin, T. Fang, X. Yang, X. Yu, and S. B. Kang, “Real-time rendering of realistic rain,” in _ACM SIGGRAPH 2006 Sketches_. ACM, 2006, p. 156.
  * [32] N. Tatarchuk, “Artist-directable real-time rain rendering in city environments,” in _ACM SIGGRAPH_. ACM, 2006, pp. 23–64.
  * [33] P. Rousseau, V. Jolivet, and D. Ghazanfarpour, “Realistic real-time rain rendering,” _Computers & Graphics_, vol. 30, no. 4, pp. 507–518, 2006\.
  * [34] Y. Luo, Y. Xu, and H. Ji, “Removing rain from a single image via discriminative sparse coding,” in _IEEE International Conference on Computer Vision_ , 2015, pp. 3397–3405.
  * [35] X. Fu, J. Huang, D. Zeng, Y. Huang, X. Ding, and J. Paisley, “Removing rain from single images via a deep detail network,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 3855–3863.
  * [36] (2021) Adding rain to a photo with photoshop. [Online]. Available: https://www.photoshopessentials.com/photo-effects/rain/
  * [37] L.-W. Kang, C.-W. Lin, and Y.-H. Fu, “Automatic single-image-based rain streaks removal via image decomposition,” _IEEE transactions on image processing_ , vol. 21, no. 4, pp. 1742–1755, 2011.
  * [38] J.-H. Kim, C. Lee, J.-Y. Sim, and C.-S. Kim, “Single-image deraining using an adaptive nonlocal means filter,” in _IEEE International Conference on Image Processing_ , 2013, pp. 914–917.
  * [39] Y.-L. Chen and C.-T. Hsu, “A generalized low-rank appearance model for spatio-temporally correlated rain streaks,” in _IEEE International Conference on Computer Vision_ , 2013, pp. 1968–1975.
  * [40] Y. Chang, L. Yan, and S. Zhong, “Transformed low-rank model for line pattern noise removal,” in _IEEE International Conference on Computer Vision_ , 2017, pp. 1726–1734.
  * [41] Y. Li, R. T. Tan, X. Guo, J. Lu, and M. S. Brown, “Rain streak removal using layer priors,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 2736–2744.
  * [42] W. Yang, J. Liu, S. Yang, and Z. Guo, “Scale-free single image deraining via visibility-enhanced recurrent wavelet learning,” _IEEE Transactions on Image Processing_ , vol. 28, no. 6, pp. 2948–2961, 2019.
  * [43] D. Ren, W. Zuo, Q. Hu, P. Zhu, and D. Meng, “Progressive image deraining networks: A better and simpler baseline,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3937–3946.
  * [44] Q. Guo, J. Sun, F. Juefei-Xu, L. Ma, X. Xie, W. Feng, and Y. Liu, “Efficientderain: Learning pixel-wise dilation filtering for high-efficiency single-image deraining,” in _The Thirty-Fifth AAAI Conference on Artificial Intelligence_ , 2021.
  * [45] R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 2482–2491.
  * [46] R. Li, L.-F. Cheong, and R. T. Tan, “Heavy rain image restoration: Integrating physics model and conditional adversarial learning,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1633–1642.
  * [47] W. Wei, D. Meng, Q. Zhao, Z. Xu, and Y. Wu, “Semi-supervised transfer learning for image rain removal,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3877–3886.
  * [48] R. Yasarla, V. A. Sindagi, and V. M. Patel, “Syn2real transfer learning for image deraining using gaussian processes,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2726–2736.
  * [49] J. H. Van Boxel, “Numerical model for the fall speed of rain drops in a rain fall simulator,” in _Workshop on wind and water erosion_ , 1997, pp. 77–85.
  * [50] D. Eigen, D. Krishnan, and R. Fergus, “Restoring an image taken through a window covered with dirt or rain,” in _IEEE International Conference on Computer Vision_ , 2013, pp. 633–640.
  * [51] M. Jaderberg, K. Simonyan, A. Zisserman _et al._ , “Spatial transformer networks,” in _Advances in neural information processing systems_ , 2015, pp. 2017–2025.
  * [52] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “AugMix: A simple data processing method to improve robustness and uncertainty,” in _ICLR_ , 2019.
  * [53] A. Kurakin, I. Goodfellow, S. Bengio, Y. Dong, F. Liao, M. Liang, T. Pang, J. Zhu, X. Hu, C. Xie _et al._ , “Adversarial attacks and defences competition,” _arXiv preprint arXiv:1804.00097_ , 2018.
  * [54] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _ECCV_ , 2014, pp. 740–755.
  * [55] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2012, pp. 3354–3361.
  * [56] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _The Thirty-First AAAI Conference on Artificial Intelligence_ , 2017, pp. 4278–4284.
  * [57] F. Chollet, “Xception: Deep learning with depthwise separable convolutions,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1251–1258.
  * [58] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _International Conference on Learning Representations_ , 2014.
  * [59] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [60] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778.
  * [61] R. De Charette, R. Tamburo, P. C. Barnum, A. Rowe, T. Kanade, and S. G. Narasimhan, “Fast reactive control for illumination through rain and snow,” in _IEEE International Conference on Computational Photography (ICCP)_ , 2012, pp. 1–10.
  * [62] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _ICLR_ , 2018.
  * [63] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” in _IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1125–1134.
  * [64] F. Lateef and Y. Ruichek, “Survey on semantic segmentation using deep learning techniques,” _Neurocomputing_ , vol. 338, pp. 321–348, 2019.
  * [65] M. Fiaz, A. Mahmood, S. Javed, and S. K. Jung, “Handcrafted and deep trackers: Recent visual object tracking approaches and trends,” _ACM Computing Surveys_ , vol. 52, no. 2, pp. 1–44, 2019.