# WaveCNet: Wavelet Integrated CNNs to Suppress Aliasing Effect for Noise-
Robust Image Classification

Qiufu Li, Linlin Shen*, Sheng Guo, Zhihui Lai The work is supported by the
Natural Science Foundation of China under grants no. 62006156, 91959108 and
U1713214, and the Science and Technology Project of Guangdong Province under
grant no. 2018A050501014. Corresponding author: Linlin Shen.Q. Li, L. Shen,
and Z. Lai are with the Computer Vision Institute, College of Computer Science
and Software Engineering, Shenzhen University, Shenzhen 518060, China,
Shenzhen Institute of Artificial Intelligence and Robotics for Society,
Shenzhen 518060, China, and Guangdong Key Laboratory of Intelligent
Information Processing, Shenzhen University, Shenzhen 518060, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>lai_zhi_hui@163.com). S. Guo is with
MyBank, Ant Group, Hangzhou 310012, China (e-mail: guosheng.guosheng@alibaba-
inc.com).

###### Abstract

Though widely used in image classification, convolutional neural networks
(CNNs) are prone to noise interruptions, i.e. the CNN output can be
drastically changed by small image noise. To improve the noise robustness, we
try to integrate CNNs with wavelet by replacing the common down-sampling (max-
pooling, strided-convolution, and average pooling) with discrete wavelet
transform (DWT). We firstly propose general DWT and inverse DWT (IDWT) layers
applicable to various orthogonal and biorthogonal discrete wavelets like Haar,
Daubechies, and Cohen, etc., and then design wavelet integrated CNNs
(WaveCNets) by integrating DWT into the commonly used CNNs (VGG, ResNets, and
DenseNet). During the down-sampling, WaveCNets apply DWT to decompose the
feature maps into the low-frequency and high-frequency components. Containing
the main information including the basic object structures, the low-frequency
component is transmitted into the following layers to generate robust high-
level features. The high-frequency components are dropped to remove most of
the data noises. The experimental results show that WaveCNets achieve higher
accuracy on ImageNet than various vanilla CNNs. We have also tested the
performance of WaveCNets on the noisy version of ImageNet, ImageNet-C and six
adversarial attacks, the results suggest that the proposed DWT/IDWT layers
could provide better noise-robustness and adversarial robustness. When
applying WaveCNets as backbones, the performance of object detectors (i.e.,
faster R-CNN and RetinaNet) on COCO detection dataset are consistently
improved. We believe that suppression of aliasing effect, i.e. separation of
low frequency and high frequency information, is the main advantages of our
approach. The code of our DWT/IDWT layer and different WaveCNets are available
at https://github.com/CVI-SZU/WaveCNet.

###### Index Terms:

CNN, down-sampling, aliasing effect, wavelet transform layers, noise-
robustness, basic object structure.

##  I Introduction

Small noise, including the common spatial noise [1] and the specially designed
adversarial noise [2, 3, 4, 5, 6], can drastically change the final
predication of well trained convolutional neuronal network (CNN) for image
classification. The recent studies [7, 8] show that the noise may be enlarged
as the image data flows through the deep networks. These phenomena illustrate
the weak noise-robustness of CNNs.

Figure 1: Comparison of max-pooling and wavelet transforms. Max-pooling is a
commonly used down-sampling operation in the deep networks, which could easily
breaks the basic object structures. Discrete Wavelet Transform (DWT)
decomposes an image X into its low-frequency component
$\textbf{\emph{X}}_{ll}$ and high-frequency components
$\textbf{\emph{X}}_{lh},\textbf{\emph{X}}_{hl},\textbf{\emph{X}}_{hh}$. While
$\textbf{\emph{X}}_{lh},\textbf{\emph{X}}_{hl},\textbf{\emph{X}}_{hh}$
represent image details including most of the noise, $\textbf{\emph{X}}_{ll}$
is a low resolution version of the image, where the basic object structures
are represented. In the figures, the window boundary in area A (AP) and the
poles in area B (BP) are broken by max-pooling, while the principal features
of these objects are kept in the DWT output (AW and BW).

The weak noise-robustness of CNNs is closely related to the down-sampling. The
commonly used down-sampling operations in deep networks, such as max-pooling,
average-pooling, and strided-convolution, usually ignore the classic sampling
theorem [9], which result in aliasing among the data components in different
frequency intervals. While the noise of data is mostly high-frequency
components, the low-frequency component contains the main information, such as
the basic object structures. Therefore, the aliasing introduces residual noise
in the down-sampled data and breaks the basic structures, which degrades the
accuracy and noise-robustness of CNNs. Fig. 1 presents a max-pooling example.

In signal processing, to avoid the aliasing, the data is usually decomposed
into different frequency intervals using time-frequency analysis tools, such
as wavelet [10, 11]. Discrete wavelet transform (DWT), consisting of filtering
and down-sampling, decomposes a 2D data into four low-frequency and high-
frequency components, as Fig. 1 shows. DWT separates the main information and
details of the data, while inverse DWT (IDWT), consisting of up-sampling and
filtering, reconstructs the original data using the DWT output.

In this paper, to suppress the aliasing effect in CNNs for noise-robust image
classification, we integrate the commonly used CNN architectures with discrete
wavelet transform. We firstly analyze the data forward and backward
propagation in the wavelet transforms (DWT and IDWT), and rewrite them as
general network layers in PyTorch [12]. Then, we design wavelet integrated
convolutional networks (WaveCNets). In WaveCNets, during the down-sampling,
the feature maps are decomposed by DWT into the low-frequency and high-
frequency components. While the low-frequency component is transmitted to the
following layers for robust high-level features, the high-frequency components
are dropped to resist the noise propagation. WaveCNets are evaluated on
ImageNet [13] and COCO [14] in terms of classification accuracy, noise-
robustness, adversarial robustness, and detection precision, when various
discrete wavelets and CNN architectures are used. In summary:

  1. 1.

We design general DWT/IDWT layer applicable to various discrete wavelets,
which could be used to design end-to-end wavelet integrated deep networks.

  2. 2.

We propose WaveCNets by transforming the feature maps using DWT during down-
sampling to suppress aliasing effect for noise-robust image classification.

  3. 3.

Our WaveCNets are evaluated by both classification and detection tasks. The
results on ImageNet classification show that our approach can achieve higher
accuracy, better noise-robustness, and increased adversarial robustness. The
detection results on COCO benchmark suggest that faster R-CNN [15] and
RetinaNet [16], using WaveCNets as the backbones, achieve better performance
on detecting small, medium, and large objects.

An earlier version of this paper [17] has been accepted by the 2020 IEEE
Conference on Computer Vision and Pattern Recognition (CVPR2020). We have
extensively extended the contents by presenting the introduction of wavelet
theory, exploring the details of 2D DWT/IDWT layers and giving the details of
backward propagation of both 1D and 2D DWT/IDWT layers. For experiments, we
tested the performance of using wavelet denoising as a preprocessing step and
the comparison shows that our DWT/IDWT layers achieves significantly better
noise-robustness. We have also tested the performance of our WaveCNets against
adversarial attacks. As expected, the results suggest that our DWT/IDWT layers
can reasonably improve the robustness of different CNNs against adversarial
samples. In addition, we also evaluate the performance of WaveCNets based
backbones for object detection task using COCO dataset. The results show that
wavelet could consistently improve the detection performances on small,
medium, and large objects.

##  II Related works

###  II-A Noise-robustness

The recent studies show that ImageNet-trained CNNs prefer to extract features
from object textures sensitive to noise [18, 8], and the noise could be
enlarged as the image data flows through layers in the CNNs [19, 7], leading
to the final wrong predictions. These works illustrate the weak noise-
robustness of modern CNNs. When the input image is corrupted by noise, the
output of CNN can be significantly changed, regardless of whether the noise is
easily perceived by human or not [1, 2, 3, 4, 5, 6]. The common spatial noise,
such as Gaussian noise, shot noise, and impulse noise, can noticeably degrades
the image quality, and decrease the classification accuracy of CNNs [1].
Adversarial noise, produced by specially designed algorithms [2, 3, 4, 5, 6],
can successfully attack the well trained CNNs, although the noise is not
perceived by human visual system. A benchmark evaluating CNN performance on
noisy images is proposed in [1]. Our WaveCNets will be evaluated using this
benchmark.

The conventional data augmentation by adding noise to the training images
could increase the CNN performance on noisy images [7]. Stylized ImageNet [8]
is proposed via stylizing ImageNet images with style transfer to train the
CNNs to extract more robust features from object structures. The augmentation
of training data could noticeably increase the training time or decrease the
accuracy on the normal clean images. The recent studies propose CNN
architectures integrating filtering block to denoise feature maps during
network inference. In [19], the authors design a high-level representation
guided denoiser to filter the contaminated images before inputting them into
CNN, which complicates the whole deep network architecture. A spatial
filtering block is presented in [7] to denoise the CNN feature maps and
suppress the noise effect on the CNN prediction. However, the filtering block
filter the feature maps using Gaussian filtering, mean filtering, or median
filtering, which do denoising in the whole frequency domain and easily break
basic object structure represented by the low-frequency component. Therefore,
this denoising block requires a residual structure for the CNN to converge.

###  II-B Down-sampling

The weak noise-robustness of CNNs is closely related to down-sampling in the
networks. The down-sampling operations, such as max-pooling, average-pooling,
and strided-convolution, are introduced into deep networks for local
connectivity and weight sharing. These operations could easily erase or dilute
image details [20, 21], and then break the basic object structures. While
mixed-pooling [20] and stochastic pooling [21] are proposed to address the
drawbacks, the max-pooling, average-pooling, and strided-convolution are still
the most widely applied down-sampling operations in CNNs [22, 23, 24, 25].

The above down-sampling operations usually ignore the Nyquist’s sampling
theorem [9, 26, 27], which result in aliasing among the data components, break
basic object structures and accumulate random noise during CNN inference. Fig.
1 presents a max-pooling example. Low-pass filtering is integrated with the
down-sampling in anti-aliased CNNs [27], to increase the shift-invariance of
CNNs. The author is surprised at the increased classification accuracy and
better noise-robustness. Our WaveCNets and anti-aliased CNNs are significantly
different in two aspects: (1) While anti-aliased CNNs are designed to increase
the CNN shift-invariance, the WaveCNets are conceived on the idea of
suppressing aliasing effect using wavelet transform to increase the noise-
robustness. (2) The low-pass filters used in anti-aliased CNNs are empirically
designed based on the row vectors of Pascal’s triangle, which is ad hoc and no
theoretical justifications are given. No up-sampling operation, i.e.,
reconstruction, of the low-pass filter is available. Therefore, the anti-
aliased U-Net [27], the encoder-decoder version of anti-aliased deep network,
has to apply the same filtering after normal up-sampling to achieve the anti-
aliasing effect. In comparison, our WaveCNets are justified by the well
defined wavelet theory [11, 10]. Both the usual down-sampling and up-sampling
operations can be replaced by DWT and IDWT [17, 28], respectively.

In deep networks for image-to-image translation tasks, the up-sampling
operations, such as max-unpooling in SegNet [29], deconvolution in U-Net [30],
and bilinear interpolation in DeepLab [31, 32] are widely applied to upgrade
the feature map resolution. However, these up-sampling operations can not
precisely recover the original data, due to the absence of the strict
mathematical terms. They do not perform well in the restoration of image
details, while the proposed DWT/IDWT layer could relieve this drawback [28].

###  II-C Wavelets in deep learning

Wavelet [11, 10] has wide applications in signal processing, pattern
recognition, etc., due to its superior performance in time-frequency analysis.
Discrete wavelet transform (DWT) decomposes a data into various components,
and separates the main information and details of the data. The original data
could be reconstructed by inverse DWT (IDWT) using the DWT output. In signal
processing, DWT is a useful tool for anti-aliasing, and we mainly explore in
this paper its application in suppressing the aliasing effect in CNNs for
noise-robust image classification.

When Mallat et al explore the optimal deep network from mathematical and
algorithmic perspective, they present ScatNet [33] by cascading wavelet
transform with average-pooling and nonlinear modulus operation. ScatNet
preserves the image detail information and extracts a translation invariant
feature robust to deformations. Compared with the CNNs of same period, ScatNet
achieves better performance on the texture discrimination and handwritten
digit recognition tasks. In [34], Wiatowski et al extend the idea of ScatNet
to semi-discrete frames and general nonlinear operations (ReLU, Sigmoid,
etc.). However, ScatNet is essentially a hand-designed feature extractor
without learnable parameters. Due to the strict mathematical terms, ScatNet
can not be easily transferred to image-to-image tasks, such as image
segmentation.

In the early studies of wavelet integrated neural networks, the researchers
implement wavelet transform using parameterized one-layer networks, and search
the optimal wavelet in parameter domain for function approximation [35],
signal representation [36]. The recent work [37] applies this method with
deeper network for image classification. However, this wavelet parameterized
deep network is difficult to train because of the significantly increased
computation [37].

In current deep learning, while wavelet transform is commonly applied as image
preprocessing or postprocessing [38, 39, 40, 41], it is also used as down-
sampling or up-sampling operations to design deep netwotks [42, 43, 44, 45].
Multi-level wavelet CNN (MWCNN) [42] is a wavelet integrated encoder-decoder
for image restoration, which implements wavelet package transform (WPT) by
processing the concatenation of various components of the input data in a
unified way. However, the details represented by the high-frequency components
may not be perceived by MWCNN, because the data amplitude of the high-
frequency components is much smaller than that of low-frequency one. In [44],
the authors apply dual-tree complex wavelet transform (DT-CWT) and design
convolutional-wavelet neural network (CWNN), to extract robust features from
SAR images. CWNN adopts DT-CWT to suppress the noise and keep the object
structures in the SAR images, which contains only two convolutional layers.
DT-CWT is redundant, and the average value of the two low-frequency components
is taken as the down-sampling output of CWNN. In [43], the authors propose
wavelet pooling layer using a two-level DWT and a one-level IDWT, while the
back-propagation is implemented using one-level DWT and a two-level IDWT,
which does not follow the mathematical principle of gradient. The author
design wavelet integrated networks and evaluate them on various datasets
(MNIST [46], CIFAR-10 [47], SHVN [48], and KDEF [49]). However, these wavelet
integrated networks consist of only four or five layers, and they are not
systematically studied on the standard large-scale image dataset ImageNet
[13]. The recent work also implement the application of wavelet transform in
deep network based image style transfer [45]. Due to the absence of general
wavelet transform layers, the above methods are only evaluated using only one
or two wavelets like Haar or dual-tree complex wavelet, which is not
extensively evaluated.

##  III Our method

Our method is trying to design the general discrete wavelet transform
(DWT/IDWT) layers and apply them to improve the performance of CNNs for image
classification. We firstly present the basic wavelet theory.

###  III-A Wavelet

TABLE I: Low-pass filters of the Daubechies wavelets. The high-pass filters of Daubechies wavelets could be deduced from the low-pass filters via Eq. (1). Daubechies wavelets are orthogonal. $p$ | $1$ | $2$ | $3$ | $4$ | $5$ | $6$
---|---|---|---|---|---|---
$l_{k}$ | $1$ | $1+\sqrt{3}$ | $~{}~{}~{}0.332670552950$ | $~{}~{}~{}0.230377813309$ | $~{}~{}~{}0.160102397974$ | $~{}~{}~{}0.111540743350$
$1$ | $3+\sqrt{3}$ | $~{}~{}~{}0.806891509311$ | $~{}~{}~{}0.714846570553$ | $~{}~{}~{}0.603829269797$ | $~{}~{}~{}0.494623890398$
| $3-\sqrt{3}$ | $~{}~{}~{}0.459877502118$ | $~{}~{}~{}0.630880767930$ | $~{}~{}~{}0.724308528438$ | $~{}~{}~{}0.751133908021$
| $1-\sqrt{3}$ | $-0.135011020010$ | $-0.027983769417$ | $~{}~{}~{}0.138428145901$ | $~{}~{}~{}0.315250351709$
|  | $-0.085441273882$ | $-0.187034811719$ | $-0.242294887066$ | $-0.226264693965$
|  | $~{}~{}~{}0.035226291886$ | $~{}~{}~{}0.030841381836$ | $-0.032244869585$ | $-0.129766867567$
|  |  | $~{}~{}~{}0.032883011667$ | $~{}~{}~{}0.077571493840$ | $~{}~{}~{}0.097501605587$
|  |  | $-0.010597401785$ | $-0.006241490213$ | $~{}~{}~{}0.027522865530$
|  |  |  | $-0.012580751999$ | $-0.031582039317$
|  |  |  | $~{}~{}~{}0.003335725285$ | $~{}~{}~{}0.000553842201$
|  |  |  |  | $~{}~{}~{}0.004777257511$
|  |  |  |  | $-0.001077301085$
factor | $1/\sqrt{2}$ | $1/(4\sqrt{2})$ | $1$ | $1$ | $1$ | $1$
TABLE II: Low-pass filters of the Cohen wavelets. The high-pass filters of Cohen wavelets could be deduced via Eqs. (2)-(3). The filters and dual filters of biorthogonal Cohen wavelets are applied to decompose and reconstruct image, respectively. $(p,\tilde{p})$ | $(2,2)$ | $(3,3)$ | $(4,4)$ | $(5,5)$
---|---|---|---|---
filter | l | $\tilde{\textbf{\emph{l}}}$ | l | $\tilde{\textbf{\emph{l}}}$ | l | $\tilde{\textbf{\emph{l}}}$ | l | $\tilde{\textbf{\emph{l}}}$
$l_{k}$ | $0$ | $0$ | $0$ | $~{}~{}~{}0.06629126$ | $0$ | $0$ | $~{}~{}~{}0.01345671$ | $0$
$0.35355339$ | $-0.17677670$ | $0$ | $-0.19887378$ | $-0.06453888$ | $~{}~{}~{}0.03782846$ | $-0.00269497$ | $0$
$0.70710678$ | $~{}~{}~{}0.35355339$ | $0.17677670$ | $-0.15467961$ | $-0.04068942$ | $-0.02384947$ | $-0.13670658$ | $~{}~{}~{}0.03968709$
$0.35355339$ | $~{}~{}~{}1.06066017$ | $0.53033009$ | $~{}~{}~{}0.99436891$ | $~{}~{}~{}0.41809227$ | $-0.11062440$ | $-0.09350470$ | $~{}~{}~{}0.00794811$
$0$ | $~{}~{}~{}0.35355339$ | $0.53033009$ | $~{}~{}~{}0.99436891$ | $~{}~{}~{}0.78848562$ | $~{}~{}~{}0.37740286$ | $~{}~{}~{}0.47680327$ | $-0.05446379$
$0$ | $-0.17677670$ | $0.17677670$ | $-0.15467961$ | $~{}~{}~{}0.41809227$ | $~{}~{}~{}0.85269868$ | $~{}~{}~{}0.89950611$ | $~{}~{}~{}0.34560528$
|  | $0$ | $-0.19887378$ | $-0.04068942$ | $~{}~{}~{}0.37740286$ | $~{}~{}~{}0.47680327$ | $~{}~{}~{}0.73666018$
|  | $0$ | $~{}~{}~{}0.06629126$ | $-0.06453888$ | $-0.11062440$ | $-0.09350470$ | $~{}~{}~{}0.34560528$
|  |  |  | $0$ | $-0.02384947$ | $-0.13670658$ | $-0.05446379$
|  |  |  | $0$ | $~{}~{}~{}0.03782846$ | $-0.00269497$ | $~{}~{}~{}0.00794811$
|  |  |  |  |  | $~{}~{}~{}0.01345671$ | $~{}~{}~{}0.03968709$
|  |  |  |  |  | $0$ | $0$

Wavelet [11, 10] is associated with scaling function $\phi(x)$ and wavelet
function $\psi(x)$, whose shifts and expansions compose stable basis for the
signal space $L^{2}(x)$. With the basis, a signal can be decomposed and
reconstructed. The scaling and wavelet functions of discrete wavelet are
closely related with low-pass filter
$\textbf{\emph{l}}=\\{l_{k}\\}_{k\in\mathbb{Z}}$ and high-pass filter
$\textbf{\emph{h}}=\\{h_{k}\\}_{k\in\mathbb{Z}}$, respectively. In practice,
these filters are applied for the data decomposition and reconstruction in DWT
and IDWT.

Orthogonal wavelets Daubechies wavelet is orthogonal, a set of orthogonal
basis for $L^{2}(x)$ could be derived from its scaling and wavelet functions.
Daubechies wavelet has an approximation order parameter $p$, and the length of
its filter is $2p$. Table I shows the low-pass filter
$\textbf{\emph{l}}=\\{l_{k}\\}$ of the wavelets with order $p,1\leq p\leq 6$,
while the high-pass filter $\textbf{\emph{h}}=\\{h_{k}\\}$ can be deduced from

$h_{k}=(-1)^{k}l_{\mathcal{N}-k},$ (1)

where $\mathcal{N}$ is an odd number. Daubechies(1) is Haar wavelet.

Biorthogonal wavelets Cohen wavelets are symmetric biorthogonal wavelets, and
each of them is associated with scaling function $\phi$, wavelet function
$\psi$, and their dual functions $\tilde{\phi},\tilde{\psi}$. Correspondingly,
it has four filters l, h, $\tilde{\textbf{\emph{l}}}$, and
$\tilde{\textbf{\emph{h}}}$. While a signal is decomposed using filters l and
h with DWT, it can be reconstructed using the dual filters
$\tilde{\textbf{\emph{l}}}$ and $\tilde{\textbf{\emph{h}}}$ with IDWT. Cohen
wavelet is with two order parameters $p$ and $\tilde{p}$. Table II shows the
low-pass filters with orders $2\leq p=\tilde{p}\leq 5$. Their high-pass
filters can be deduced from

$\displaystyle h_{k}=(-1)^{k}\tilde{l}_{\mathcal{N}-k},$ (2)
$\displaystyle\tilde{h}_{k}=(-1)^{k}l_{\mathcal{N}-k},$ (3)

where $\mathcal{N}$ is an odd number. Cohen$(1,1)$ is Haar wavelet.

Wavelet theory is valid for finite or infinite filters, but the infinite case
is rarely covered in practical interest. Besides the othogonal and
biorthogonal wavelets, various wavelets and beyond-wavelets, including multi-
wavelets [50], dual-tree complex wavelet [51], ridgelet [52], curvelet [53],
bandelet [54], and contourlet [55], etc., have been designed and studied.
Wavelets have been widely used in signal processing, numerical analysis,
pattern recognition, computer vision, quantum mechanics, etc.

###  III-B Wavelet transform layers

The key issues for the general wavelet transform (DWT/IDWT) layers are the
data forward and backward propagations. Although the following analysis is for
orthogonal wavelets and 1D/2D data, it can be generalized to other discrete
wavelets and 3D data with only slight changes.

####  III-B1 1D DWT/IDWT

DWT decomposes a given 1D data
$\textbf{\emph{x}}=\\{x_{j}\\}_{j\in\mathbb{Z}}$ into its low-frequency
component
$\textbf{\emph{x}}_{\textrm{low}}=\\{x_{k}^{(\textrm{low})}\\}_{k\in\mathbb{Z}}$
and high-frequency component
$\textbf{\emph{x}}_{\textrm{high}}=\\{x_{k}^{(\textrm{high})}\\}_{k\in\mathbb{Z}}$,
where

$\left\\{\begin{array}[]{l}x_{k}^{(\textrm{low})}=\sum_{j}l_{j-2k}x_{j},\\\
x_{k}^{(\textrm{high})}=\sum_{j}h_{j-2k}x_{j},\end{array}\right.$ (4)

and
$\textbf{\emph{l}}=\\{l_{k}\\}_{k\in\mathbb{Z}},\textbf{\emph{h}}=\\{h_{k}\\}_{k\in\mathbb{Z}}$
are the low-pass and high-pass filters of an orthogonal wavelet. According to
Eq. (4), DWT consists of filtering and down-sampling. IDWT reconstructs x
using $\textbf{\emph{x}}_{\textrm{low}},\textbf{\emph{x}}_{\textrm{high}}$,
where

$x_{j}=\sum_{k}\left(l_{j-2k}x_{k}^{(\textrm{low})}+h_{j-2k}x_{k}^{(\textrm{high})}\right).$
(5)

In expressions with matrices and vectors, Eq. (4) and Eq. (5) can be rewritten
as

$\displaystyle\textbf{\emph{x}}_{\textrm{low}}=\mathcal{L}\,\textbf{\emph{x}},$
(6)
$\displaystyle\textbf{\emph{x}}_{\textrm{high}}=\mathcal{H}\,\textbf{\emph{x}},$
(7)
$\displaystyle\textbf{\emph{x}}=\mathcal{L}^{T}\textbf{\emph{x}}_{\textrm{low}}+\mathcal{H}^{T}\textbf{\emph{x}}_{\textrm{high}},$
(8)

where

$\displaystyle\mathcal{L}$
$\displaystyle=\left(\begin{array}[]{ccccccc}\cdots&\cdots&\cdots&&&&\\\
\cdots&l_{-1}&l_{0}&l_{1}&\cdots&&\\\ &&\cdots&l_{-1}&l_{0}&l_{1}&\cdots\\\
&&&&&\cdots&\cdots\end{array}\right),$ (13) $\displaystyle\mathcal{H}$
$\displaystyle=\left(\begin{array}[]{ccccccc}\cdots&\cdots&\cdots&&&&\\\
\cdots&h_{-1}&h_{0}&h_{1}&\cdots&&\\\ &&\cdots&h_{-1}&h_{0}&h_{1}&\cdots\\\
&&&&&\cdots&\cdots\end{array}\right).$ (18)

Eqs. (6) – (8) present the forward propagations for 1D DWT and IDWT. The
backward propagation of 1D DWT is closely associated with the gradients
$\frac{\partial\textbf{\emph{x}}_{\textrm{low}}}{\partial\textbf{\emph{x}}}$
and
$\frac{\partial\textbf{\emph{x}}_{\textrm{high}}}{\partial\textbf{\emph{x}}}$,
which can be derived from Eqs. (6),

$\frac{\partial\textbf{\emph{x}}_{\textrm{low}}}{\partial\textbf{\emph{x}}}=\mathcal{L}^{T},~{}~{}~{}~{}\frac{\partial\textbf{\emph{x}}_{\textrm{high}}}{\partial\textbf{\emph{x}}}=\mathcal{H}^{T}.$
(19)

Similarly, for the back propagation of IDWT, the key issues are the gradients
$\frac{\partial\textbf{\emph{x}}}{\partial\textbf{\emph{x}}_{\textrm{low}}},\frac{\partial\textbf{\emph{x}}}{\partial\textbf{\emph{x}}_{\textrm{high}}}$,
which can be derived from Eq. (8),

$\frac{\partial\textbf{\emph{x}}}{\partial\textbf{\emph{x}}_{\textrm{low}}}=\mathcal{L},~{}~{}~{}~{}\frac{\partial\textbf{\emph{x}}}{\partial\textbf{\emph{x}}_{\textrm{high}}}=\mathcal{H}.$
(20)

####  III-B2 2D DWT/IDWT

Given a 2D data X, the DWT usually do 1D DWT on its every row and column,
i.e.,

$\displaystyle\textbf{\emph{X}}_{ll}$
$\displaystyle=\mathcal{L}\textbf{\emph{X}}\mathcal{L}^{T},$ (21)
$\displaystyle\textbf{\emph{X}}_{lh}$
$\displaystyle=\mathcal{H}\textbf{\emph{X}}\mathcal{L}^{T},$ (22)
$\displaystyle\textbf{\emph{X}}_{hl}$
$\displaystyle=\mathcal{L}\textbf{\emph{X}}\mathcal{H}^{T},$ (23)
$\displaystyle\textbf{\emph{X}}_{hh}$
$\displaystyle=\mathcal{H}\textbf{\emph{X}}\mathcal{H}^{T},$ (24)

and the corresponding 2D IDWT is implemented with

$\textbf{\emph{X}}=\mathcal{L}^{T}\textbf{\emph{X}}_{ll}\mathcal{L}+\mathcal{H}^{T}\textbf{\emph{X}}_{lh}\mathcal{L}+\mathcal{L}^{T}\textbf{\emph{X}}_{hl}\mathcal{H}+\mathcal{H}^{T}\textbf{\emph{X}}_{hh}\mathcal{H}.$
(25)

In the output of 2D DWT, $\textbf{\emph{X}}_{ll}$ is the low-frequency
component of input X, which represents the main information, including the
basic object structures;
$\textbf{\emph{X}}_{lh},\textbf{\emph{X}}_{hl},\textbf{\emph{X}}_{hh}$ are
three high-frequency components, which save the horizontal, vertical, and
diagonal details of X, respectively.

Suppose 2D DWT/IDWT are applied in deep network, then the backward propagation
of 2D DWT can be implemented by the gradients,
$\frac{\partial\textbf{\emph{X}}_{ll}}{\partial\textbf{\emph{X}}},\frac{\partial\textbf{\emph{X}}_{lh}}{\partial\textbf{\emph{X}}},\frac{\partial\textbf{\emph{X}}_{hl}}{\partial\textbf{\emph{X}}},\frac{\partial\textbf{\emph{X}}_{hh}}{\partial\textbf{\emph{X}}},$

$\displaystyle\frac{\partial\textbf{\emph{X}}_{ll}}{\partial\textbf{\emph{X}}}(G)$
$\displaystyle=\mathcal{L}^{T}G\,\mathcal{L},$ (26)
$\displaystyle\frac{\partial\textbf{\emph{X}}_{lh}}{\partial\textbf{\emph{X}}}(G)$
$\displaystyle=\mathcal{H}^{T}G\,\mathcal{L},$ (27)
$\displaystyle\frac{\partial\textbf{\emph{X}}_{hl}}{\partial\textbf{\emph{X}}}(G)$
$\displaystyle=\mathcal{L}^{T}G\,\mathcal{H},$ (28)
$\displaystyle\frac{\partial\textbf{\emph{X}}_{hh}}{\partial\textbf{\emph{X}}}(G)$
$\displaystyle=\mathcal{H}^{T}G\,\mathcal{H},$ (29)

where $G$ is the backward propagation output from the layer following the 2D
DWT.

Similarly, the backward propagation of 2D IDWT is implemented by gradients,
$\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{ll}},\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{lh}},\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{hl}},\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{hh}},$

$\displaystyle\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{ll}}(G)=\mathcal{L}\,G\,\mathcal{L}^{T},$
(30)
$\displaystyle\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{lh}}(G)=\mathcal{H}\,G\,\mathcal{L}^{T},$
(31)
$\displaystyle\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{hl}}(G)=\mathcal{L}\,G\,\mathcal{H}^{T},$
(32)
$\displaystyle\frac{\partial\textbf{\emph{X}}}{\partial\textbf{\emph{X}}_{hh}}(G)=\mathcal{H}\,G\,\mathcal{H}^{T},\vspace{-8pt}$
(33)

where $G$ is the backward propagation output from the layer following the 2D
IDWT.

The forward and backward propagations of 3D DWT and IDWT are slightly more
complicated, but similar to that of 1D/2D DWT and IDWT. In practice, we choose
the wavelets associated with finite filters, as that shown in Table I and
Table II. For finite data $\textbf{\emph{x}}\in\mathbb{R}^{N}$ and
$\textbf{\emph{X}}\in\mathbb{R}^{M\times N}$, the $\mathcal{L},\mathcal{H}$
are truncated to be the size of $\lfloor\frac{N}{2}\rfloor\times N$ or
$\lfloor\frac{M}{2}\rfloor\times M$.

We rewrite 1D/2D/3D DWT and IDWT as network layers in PyTorch [12], which are
applicable to various discrete orthogonal and biorthogonal wavelets, like
Haar, Daubechies, and Cohen. In the layers, we do DWT and IDWT channel by
channel for multi-channel data.

###  III-C WaveCNets

(a) The general denoising approach using wavelet.

(b) The simplest wavelet based “denoising” method, $\text{DWT}_{ll}$.

Figure 2: The general denoising approach based on wavelet transforms and the
one used in WaveCNet. While the general denoising approach keeps the data
size, the simplest “denoising” method, $\text{DWT}_{ll}$, halves the data
size.

The noise are mostly represented by the high-frequency components in a noisy
2D data X. Therefore, as Fig. 2(a) shows, the general wavelet based denoising
[56, 57] consists of three steps: (1) decompose the noisy data X using DWT
into low-frequency component $\textbf{\emph{X}}_{ll}$ and high-frequency
components
$\textbf{\emph{X}}_{lh},\textbf{\emph{X}}_{hl},\textbf{\emph{X}}_{hh}$, (2)
filter the high-frequency components, (3) reconstruct the data with the
processed components using IDWT. In this paper, we choose the simplest wavelet
based “denoising”, i.e., dropping the high-frequency components, as Fig. 2(b)
shows. $\text{DWT}_{ll}$ denotes the transform mapping the data to its low-
frequency component.

Based on the commonly used CNNs, including VGG16bn, ResNets, and DenseNet121,
we design WaveCNets (WVGG16bn, WResNets, WDenseNet121) by replacing their
down-sampling operation with $\text{DWT}_{ll}$. As Fig. 3 shows, WaveCNets
replace the max-pooling and average-pooling with $\text{DWT}_{ll}$, and
upgrade strided-convolution using convolution with stride of $1$ followed by
$\text{DWT}_{ll}$, i.e.,

$\displaystyle\text{MaxPool}_{s=2}\rightarrow$ $\displaystyle\
\text{DWT}_{ll},$ (34) $\displaystyle\text{Conv}_{s=2}\rightarrow$
$\displaystyle\ \text{DWT}_{ll}\circ\text{Conv}_{s=1},$ (35)
$\displaystyle\text{AvgPool}_{s=2}\rightarrow$ $\displaystyle\
\text{DWT}_{ll},$ (36)

where “$\text{MaxPool}_{s}$”, “$\text{Conv}_{s}$”, and “$\text{AvgPool}_{s}$”
denote the max-pooling, strided-convolution, and average-pooling with stride
$s$, respectively.

Figure 3: (a) Baseline, the commonly used down-sampling operations in deep
networks. (b) Wavelet integrated down-sampling in WaveCNets. While the
commonly used down-sampling operations may suffer from aliasing effects, our
wavelet integrated down-sampling do low-pass filter before down-sampling,
which helps suppressing the aliasing effects.

In the down-sampling of WaveCNets, while $\text{DWT}_{ll}$ halves the size of
the feature maps, it denoises them by removing their high-frequency
components. The output of $\text{DWT}_{ll}$, i.e., the low-frequency
component, saves the main information of the feature map to extract the
identifiable features. In other words, $\text{DWT}_{ll}$ could help WaveCNets
to suppress the aliasing effect, i.e., maintain the basic object structures in
the the CNN feature maps and resist the noise propagation. Therefore, it is
expected that wavelet will lead to higher accuracy and better noise-robustness
for CNN based image classification.

###  III-D Computation for DWT/IDWT

Compared with the original CNNs, WaveCNets do not employ new learnable
parameters, while the wavelet transform introduces additional computation.

We here analyze the multiply-add operations increased by wavelet transform in
WaveCNets. Given a 2D tensor X with size of $M\times N$ and $C$ channels, the
amount of multiply-adds used in 2D DWT is

$\displaystyle 4C\left(M^{2}N+\dfrac{MN^{2}}{2}-\dfrac{3MN}{4}\right),$ (37)

and the amount of multiply-adds used in 2D IDWT is

$\displaystyle 4C\left(MN^{2}+\dfrac{M^{2}N}{2}-\dfrac{3MN}{4}\right)+3,$ (38)

according to Eqs. (6) – (24).

Table III presents the ratios of wavelet related multiply-adds over the total
operations for WaveCNets, when the input size is $3\times 224\times 224$. We
only count the amount of multiply-adds in $\text{DWT}_{ll}$ for WaveCNets.

TABLE III: Multiply-add operation numbers in WaveCNets. DWTll are complicated
compared with the common down-sampling in the original CNNs, leading to more
multiply-add operations.

| WaveCNet
---|---
VGG | ResNet | DenseNet
16bn | 18 | 34 | 50 | 101 | 121
baseline* ($\times 10^{9}$) | $15.51$ | $1.82$ | $3.67$ | $4.12$ | $7.85$ | $2.88$
non-wavelet ($\times 10^{9}$) | $15.51$ | $1.82$ | $3.67$ | $4.12$ | $7.85$ | $2.88$
wavelet ($\times 10^{9}$) | $1.43$ | $0.22$ | $0.18$
1-7[2pt/2pt] ratio ($\%$) | $8.44$ | $10.85$ | $5.69$ | $5.11$ | $2.75$ | $5.82$

  * *

corresponding to the multiply-add operation numbers of original CNNs (i.e.,
VGG16bn, ResNets, DenseNet121), which are almost the same with the non-wavelet
operation numbers in the WaveCNets.

##  IV Experiments

TABLE IV: Top-1 accuracy of WaveCNets on ImageNet validation set. The
symmetric Haar and Cohen wavelets improve the accuracy of CNNs (VGG16bn,
ResNets, and DenseNet121), while the asymmetric Daubechies wavelets (“db4”,
“db5”, and “db6”) with high approximation orders may decrease the CNN
performance.

Wavelet | WVGG16bn | WResNet18 | WResNet34 | WResNet50 | WResNet101 | WDenseNet121
---|---|---|---|---|---|---
none (baseline)* | 73.37 | 69.76 | 73.30 | 76.15 | 77.37 | 74.65
Haar | 74.10 ($+$0.73) | 71.47 ($+$1.71) | 74.35 ($+$1.05) | 76.89 ($+$0.74) | 78.23 ($+$0.86) | 75.27 ($+$0.62)
Cohen | ch2.2 | 74.31 ($+$0.94) | 71.62 ($+$1.86) | 74.33 ($+$1.03) | 76.41 ($+$0.26) | 78.34 ($+$0.97) | 75.36 ($+$0.71)
ch3.3 | 74.40 ($+$1.03) | 71.55 ($+$1.79) | 74.51 ($+$1.21) | 76.71 ($+$0.56) | 78.51 ($+$1.14) | 75.44 ($+$0.79)
ch4.4 | 74.02 ($+$0.65) | 71.52 ($+$1.76) | 74.61 ($+$1.31) | 76.56 ($+$0.41) | 78.47 ($+$1.10) | 75.29 ($+$0.64)
ch5.5 | 73.67 ($+$0.30) | 71.26 ($+$1.50) | 74.34 ($+$1.04) | 76.51 ($+$0.36) | 78.39 ($+$1.02) | 75.01 ($+$0.36)
Daubechies | db2 | 74.08 ($+$0.71) | 71.48 ($+$1.72) | 74.30 ($+$1.00) | 76.27 ($+$0.12) | 78.29 ($+$0.92) | 75.08 ($+$0.43)
db3 | 73.64 ($+$0.27) | 71.08 ($+$1.32) | 74.11 ($+$0.81) | 76.38 ($+$0.23) | 77.80 ($+$0.43) | 74.66 ($+$0.01)
db4 | 72.71 ($-$0.66) | 70.35 ($+$0.59) | 73.53 ($+$0.23) | 75.65 ($-$0.50) | 76.85 ($-$0.52) | 73.68 ($-$0.97)
db5 | 71.15 ($-$2.22) | 69.54 ($-$0.22) | 73.41 ($+$0.11) | 74.90 ($-$1.25) | 76.19 ($-$1.18) | 72.35 ($-$2.30)
db6 | 69.06 ($-$4.31) | 68.74 ($-$1.02) | 72.68 ($-$0.62) | 73.95 ($-$2.20) | 75.36 ($-$2.01) | 70.99 ($-$3.66)

  * *

corresponding to the results of original CNNs, i.e., VGG16bn, ResNets,
DenseNet121.

In our experiments, we mainly evaluate classification accuracies and noise-
robustness of WaveCNets using ImageNet [13]. We also test the improvement of
WaveCNets on the classification of adversarial samples and detection of COCO.

###  IV-A ImageNet classification

ImageNet contains 1.2M training and 50K validation images from 1000
categories. WaveCNets are trained on the training set from scratch when
various wavelets are used. We train them 90 epochs using stochastic gradient
descent (SGD) with batch size of $256$. The initial learning rate is $0.05$
for WVGG16bn and $0.1$ for WResNets and WDenseNet121, which is decayed by
multiplying $0.1$ every $30$ epochs.

Table IV presents the top-1 accuracy of WaveCNets on ImageNet validation set,
where “haar”, “db$p$”, and “ch$p.\tilde{p}$” denote the Haar wavelet,
Daubechies wavelet with approximation order $p$, and Cohen wavelet with orders
$(p,\tilde{p})$. While Haar and Cohen wavelets are symmetric, Daubechies are
not.

In Table IV, the classification results of the original CNNs are taken as the
baseline, which are sourced from the official
PyTorch111https://pytorch.org/docs/stable/torchvision/models.html#classification.
The accuracy difference of WaveCNets compared with the baseline are
parenthesized in the table. The symmetric Haar and Cohen wavelets improve the
classification accuracy of all CNNs, while the best wavelet varies with CNN.
Take ResNet18 for example, the symmetric wavelets improve the accuracy by
about $1.50\%$, and the best performance ($71.62\%$) of WResNet18 is achieved
with wavelet “ch2.2”. However, as the order increases, the asymmetric
Daubechies wavelet decreases the classification accuracy of CNNs. Daubechies
wavelets with lower orders (“db2” and “db3”) could improve the CNN accuracy,
while that with higher orders (“db5” and “db6”) may reduce the accuracy. For
example, the top-1 accuracy of WResNet18 decreases from $71.48\%$ to
$68.74\%$, as the order increases from $2$ to $6$. In general, CNNs
transformed by the symmetric wavelets perform better than that by asymmetric
wavelets.

Figure 4: The loss of ResNet18 and WResNet18(Haar). In their first 10 epochs,
the loss of WResNet18(Haar) decreases faster than that of ResNet18, and the
tread keeps nearly the same in the following epochs, which suggests that
wavelet could accelerates the CNN training in the initial training stage.

(a) _suit_

(b) _espresso_

(c) _lycaenid_

(d) _palace_

(e) _jay_

(f) _minivan_

(g) _wall clock_

(h) _schooner_

(i) _stupa_

Figure 5: The feature maps of CNNs (top) and WaveCNets (bottom). In each
subfigure, the first row shows the input image and its two feature maps output
from the original CNNs, while the second row shows the related information
(the image, CNN, and WaveCNet names) and the feature maps output from the
WaveCNets. Compared with CNNs, the feature maps of WaveCNets are cleaner and
the object structures are more complete.

We retrain the original ResNet18 using the standard ImageNet classification
training repository in PyTorch. Fig. 4 compares the losses of ResNet18 and
WResNet18(Haar) during the training procedure, which adopts red dashed and
blue dashed lines to denote the train losses of ResNet18 and WResNet18(Haar),
respectively. In the initial training stage (the first 10 epochs), the loss of
WResNet18(Haar) decreases faster than that of ResNet18, and the trends become
similar in the following epochs, which suggests that wavelet accelerates the
ResNet18 training in the initial training stage. Finally, the training loss of
WResNet18(Haar) is about $0.08$ lower than that of ResNet18, although they
employ the same amount of learnable parameters. On the validation set,
WResNet18 loss (blue solid line) is also always lower than ResNet18 loss (red
solid line), which leads to the increase of classification accuracy by
$1.71\%$.

Fig. 5 presents nine example feature maps produced by well trained CNNs and
WaveCNets for the ImageNet validation images. In each subfigure, the first row
shows the input image with size of $224\times 224$ and its two feature maps
produced by original CNN, while the second row shows the related information
(image, CNN, and WaveCNet names) and the feature maps produced by the
WaveCNet. The two feature maps are captured from the 16th output channel of
the final layer in the network blocks with tensor size of $56\times 56$
(middle) and $28\times 28$ (right), respectively. We enlarge and color the
feature maps for better illustration.

The examples in Fig. 5 cover the various deep network architectures (VGG16bn,
ResNet18, ResNet34, ResNet50, and ResNet101) and various objects (_suit_ ,
_espresso_ , _lycaenid_ , _palace_ , _jay_ , _minican_ , _wall clock_ ,
_stupa_ , etc.). While the original CNNs adopt various down-sampling
operations, including max-pooling, average-pooling, and strided-convolution,
WaveCNets replace them using DWTll. From Fig. 5, one can find that the
backgrounds of the feature maps produced by WaveCNets are cleaner than that
produced by CNNs, and the object structures in the former are more complete
than that in the latter. For example, in the first row of Fig. 5(g), the _wall
clock_ boundary in the ResNet50 feature map with size of $56\times 56$ are
fuzzy, and the basic structures of _wall clocks_ have been totally hidden by
strong noise in the feature map with size of $28\times 28$. In the second row,
the backgrounds of feature maps produced by WResNet50(ch3.3) are very clean,
and it is easy to figure out the _wall clock_ structures in the feature map
with size of $56\times 56$ and the _wall clock_ areas in the feature map with
size of $28\times 28$. The above observations illustrate that the common down-
sampling operations could result in aliasing effect, i.e., accumulating noise
and breaking the basic object structures, while DWT in WaveCNets relieves the
drawbacks. We believe that this is the reason why WaveCNets achieve increased
accuracy.

In [27], the author is surprised at the increased classification accuracy of
CNNs after low-pass filtering is integrated into the down-sampling. In [8],
the authors show that “ImageNet-trained CNNs are strongly biased towards
recognising textures rather than shapes”. Our experimental results suggest
that this may be sourced from the commonly used down-sampling operations,
which tend to break the object structures and accumulate noise in the feature
maps.

###  IV-B Noise-robustness

In [1], the authors corrupt the ImageNet validation set using 15 visual
corruptions with five severity levels, to create ImageNet-C and test the
robustness of ImageNet-trained classifiers to the input corruptions. The 15
corruptions are sourced from four categories, i.e., noise (Gaussian noise,
shot noise, impulse noise), blur (defocus blur, frosted glass blur, motion
blur, zoom blur), weather (snow, frost, fog, brightness), and digital
(contrast, elastic, pixelate, JPEG-compression). $E_{s,c}^{f}$ denotes the
top-1 error of a trained classifier $f$ on corruption type $c$ at severity
level $s$. The authors present the Corruption Error $\text{CE}_{c}^{f}$,
computed with

$\displaystyle\text{CE}_{c}^{f}$
$\displaystyle=\sum_{s=1}^{5}E_{s,c}^{f}\left/\sum_{s=1}^{5}E_{s,c}^{\text{AlexNet}}\right.,$
(39)

to evaluate the performance of a trained classifier $f$. In Eq. (39), the
authors normalize the error using the top-1 error of AlexNet [58] to adjust
the difference of various corruptions.

In this section, we use the noise part (750K images, 50K $\times$ 3 $\times$
5) of ImageNet-C and

$\displaystyle\text{mCE}_{\text{noise}}^{f}$
$\displaystyle=\dfrac{1}{3}\left(\text{CE}_{\text{Gaussian}}^{f}+\text{CE}_{\text{shot}}^{f}+\text{CE}_{\text{impulse}}^{f}\right)$
(40)

to evaluate the noise-robustness of WaveCNet $f$.

Figure 6: The noise mCE of WaveCNets. In the figure, the “baseline”
corresponds to the noise mCEs of original CNNs, while “db$p$”,
“ch$p.\tilde{p}$” and “haar” correspond to the mCEs of WaveCNets with
different wavelets. Except VGG16bn, our method obviously increase the noise-
robustness of the CNNs for image classification.

(a) _vase_

(b) _moped_

Figure 7: The feature maps sourced from clean and noisy images. In each
subfigure, the first column shows the clean example image from ImageNet
validation set and its three noisy versions corrupted by Gaussian noise, shot
noise, and impulse noise, respectively. The second column shows their feature
maps generated by ResNet18, while the third column shows the feature maps
generated by WResNet18 using wavelet “ch2.2”. It is difficult for the original
CNN to suppress noise, while WaveCNet could suppress the noise and maintain
the object structure during its inference.

We test the top-1 errors of WaveCNets and AlexNet on each noise corruption
type $c$ at each level of severity $s$, when WaveCNets and AlexNet are trained
on the clean ImageNet training set. Then, we compute
$\text{mCE}_{\text{noise}}^{\text{WaveCNet}}$ according to Eqs. (39) and (40).
The Table V shows the detailed results (the “noisy” columns). In Fig. 6, we
show the noise mCEs of WaveCNets for different network architectures and
various wavelets. The “baseline” corresponds to the noise mCEs of original CNN
architectures, while “db$p$”, “ch$p.\tilde{p}$” and “haar” correspond to the
mCEs of WaveCNets with different wavelets. Except VGG16bn, our method
obviously increase the noise-robustness of the CNN architectures for image
classification. For example, the noise mCE of ResNet18 (with navy blue color
and down triangle marker in Fig. 6) decreases from $88.97$ (“baseline”) to
$80.38$ (“ch2.2”). One can find that all wavelets including “db5” and “db6”
improve the noise-robustness of ResNet18, ResNet34, and ResNet50, although the
classification accuracy of the WResNets with “db5” and “db6” for the clean
images may be lower than that of the original ResNets. It means that our
methods indeed increase the noise-robustness of these network architectures.

Fig. 7 shows two example feature maps for well trained ResNet18 and WResNet18
with clean and noisy images as input. In each subfigure, the first column
shows the clean example image with size of $224\times 224$ from ImageNet
validation set and its three noisy versions corrupted by Gaussian noise, shot
noise, and impulse noise. The second column shows their feature maps generated
by ResNet18, while the third column shows the feature maps generated by
WResNet18 integrated with wavelet “ch2.2”. These feature maps are captured
from the 16th output channel of the last layer in the network blocks with
tensor size of $56\times 56$. From these examples, one can find that it is
difficult for the original CNN to suppress noise, while WaveCNet could
suppress the noise and maintain the object structure during its inference. For
example, in Fig. 7(a), the _vase_ structures in the two feature maps generated
by both ResNet18 and WResNet18(ch2.2) are complete, when the clean _vase_
image is fed into the networks. However, after the image is corrupted by
noise, the ResNet18 feature maps contain very strong noise and the _vase_
structures vanish, while the basic structures could still be observed from the
WResNet18 feature maps. For the _moped_ in Fig. 7(b), the network without
wavelet, ResNet18, tends to focus its response on the two indigo areas of the
_moped_ in the feature maps, and the noise totally breaks the _moped_
structures in the feature maps for the three noisy images. In contrary,
wavelet suppresses the noise and the basic _moped_ structures are still
complete in the feature maps generated by WResNet18, whether the input image
is corrupted by noise or not. This advantage improves the robustness of
WaveCNets against the noise.

The noise-robustness of VGG16bn is inferior to that of ResNet34, although they
achieve similar accuracy ($73.37\%$ and $73.30\%$). Our method does not
significantly improve the noise-robustness of VGG16bn, although it increases
the accuracy by $1.03\%$. It means that the VGG16bn may not be a proper
architecture in terms of noise-robustness.

###  IV-C Wavelet denoising block

TABLE V: Noise mCE of WaveCNets with different versions of input data (lower
is better). Wavelets could improve the noise-robustness of vanilla CNNs. The
wavelet denoising block consistently increase the noise-robustness of the all
CNNs, no matter they are integrated with DWT layer, or not.

Wavelet | WVGG16bn | WResNet18 | WResNet34 | WResNet50 | WResNet101 | WDenseNet121
---|---|---|---|---|---|---
noisyb | denoisedc | noisy | denoised | noisy | denoised | noisy | denoised | noisy | denoised | noisy | denoised
none (baseline)a | 87.62 | 86.76 ($-$0.86) | 88.97 | 87.77 ($-$1.20) | 83.10 | 81.52 ($-$1.58) | 81.31 | 79.45 ($-$1.86) | 75.28 | 73.72 ($-$1.56) | 82.29 | 80.92 ($-$1.37)
Haar | 87.43 | 85.93 ($-$1.50) | 80.91 | 79.68 ($-$1.23) | 76.63 | 75.62 ($-$1.01) | 78.48 | 77.18 ($-$1.30) | 70.46 | 68.50 ($-$1.96) | 78.33 | 77.02 ($-$1.31)
Cohen | ch2.2 | 87.79 | 86.18 ($-$1.61) | 80.38 |  78.93 ($-$1.45) | 77.61 | 75.98 ($-$1.63) | 76.87 | 75.84 ($-$1.03) | 65.06 |  64.01 ($-$1.05) | 76.97 | 76.19 ($-$0.78)
ch3.3 | 88.28 | 87.58 ($-$0.70) | 81.02 | 79.80 ($-$1.22) | 74.30 | 73.26 ($-$1.04) | 75.04 | 73.42 ($-$1.62) | 69.82 | 68.46 ($-$1.36) | 78.31 | 76.88 ($-$1.43)
ch4.4 | 85.63 |  84.67 ($-$0.96) | 82.19 | 80.66 ($-$1.53) | 76.19 | 74.70 ($-$1.49) | 76.73 | 75.05 ($-$1.68) | 68.91 | 67.62 ($-$1.29) | 76.01 |  74.84 ($-$1.17)
ch5.5 | 89.84 | 88.70 ($-$1.14) | 83.77 | 81.97 ($-$1.80) | 76.00 | 74.09 ($-$1.91) | 72.08 |  70.99 ($-$1.09) | 70.30 | 68.90 ($-$1.40) | 76.27 | 75.18 ($-$1.09)
Daubechies | db2 | 87.44 | 85.97 ($-$1.47) | 82.54 | 81.23 ($-$1.31) | 72.73 |  71.45 ($-$1.28) | 78.80 | 77.21 ($-$1.59) | 70.76 | 68.82 ($-$1.94) | 79.82 | 78.69 ($-$1.13)
db3 | 88.53 | 86.92 ($-$1.61) | 83.92 | 82.02 ($-$1.90) | 76.43 | 74.83 ($-$1.60) | 79.34 | 77.31 ($-$2.03) | 72.70 | 70.97 ($-$1.73) | 79.03 | 78.02 ($-$1.01)
db4 | 90.02 | 88.53 ($-$1.49) | 86.23 | 84.32 ($-$1.91) | 78.96 | 77.39 ($-$1.57) | 77.99 | 76.85 ($-$1.14) | 73.50 | 71.89 ($-$1.61) | 80.89 | 80.12 ($-$0.77)
db5 | 91.26 | 89.88 ($-$1.38) | 85.47 | 83.92 ($-$1.55) | 73.15 | 72.03 ($-$1.12) | 78.33 | 77.16 ($-$1.17) | 75.29 | 73.73 ($-$1.56) | 82.35 | 81.63 ($-$0.72)
db6 | 93.73 | 92.12 ($-$1.61) | 86.57 | 84.76 ($-$1.81) | 82.02 | 80.06 ($-$1.96) | 79.78 | 78.61 ($-$1.17) | 78.43 | 76.64 ($-$1.79) | 84.44 | 83.87 ($-$0.57)

  * a

corresponding to the results of original CNNs, i.e., VGG16bn, ResNets,
DenseNet121.

  * b

taking the original noisy images in ImageNet-C as input.

  * c

taking the denoised version of noisy images in ImageNet-C as input.

While wavelet integrated CNNs like WVGG16bn, WResNets and WDenseNet121 have
been show to achieve better classification accuracy on ImageNet and noise-
robustness on ImageNet-C, the wavelet denoising block shown in 2(a) can
actually be applied to preprocess the noisy images and further improve the
performance of various CNNs.

For the noisy images in ImageNet-C, We test a so called soft threshold
filtering operation to filter the noises in high-frequency components
$\textbf{\emph{X}}_{lh},\textbf{\emph{X}}_{hl},\textbf{\emph{X}}_{hh}$, and
test the performance of different CNNs when such a denoising is applied, or
not. The commonly used soft threshold filter operation is defined as below:

$\displaystyle\text{SoftShrinkage}(x)=\begin{cases}x-\lambda,&\text{ if
}x>\lambda,\\\ x+\lambda,&\text{ if }x<-\lambda,\\\
0,&\text{otherwise}.\end{cases}$ (41)

For multiple levels of DWT and IDWT, various thresholds $\lambda$, or
functions $\lambda(x)$, could be selected. In our work, we perform one level
DWT and IDWT and the value of $\lambda$ is set as 0.1.

Table V shows the performance of different CNNs when the wavelet based
denoising block is applied to denoise the input images, or not. The numbers in
brackets denote the differences of mCE when the denoising is applied, or not.
One can observe from the table that the denoising module consistently decrease
the mCE for all of the networks like VGG16bn, ResNets and DenseNet121, no
matter the CNNs are integrated with our DWT layer, or not. For vanilla CNNs,
the denoising operation reduces the mCE of ResNet50 from 81.31 to 79.45. For
wavelet integrated CNNs, such operation reduces the mCE of WResNet101 from
70.46 to 68.50, when Haar wavelet is concerned. The lowest mCE (64.01) is
achieved by WResNet101 integrated with wavelet Cohen(2,2). The results also
suggest that our proposed DWT integration in CNNs produces a much better
improvement of the noise-robustness than the wavelet based denoising block,
e.g., the mCE (80.38) of WResNet18(ch2.2) using the original noisy input is
significantly lower than that of ResNet18 (87.77) input with the denoised
data.

###  IV-D Comparison with other wavelet based down-sampling

(a) Average mode [44].

(b) Concatenation mode [42].

Figure 8: Wavelet integrated down-sampling in various modes. In these two modes, the high-frequency components are added into or concatenated with the low-frequency component, which would damage the useful information because of the high-frequency noise. TABLE VI: Comparison with other wavelet based down-sampling. The networks using wavelet based down-sampling generally achieve better accuracy and noise mCE. Comprehensively, WResNet18 using DWTll performs the best among them.  Network | Top-1 Accuracy (higher is better) | Params.
---|---|---
haar | ch2.2 | ch3.3 | ch4.4 | ch5.5 | db2
WResNet18 | 71.47 | 71.62 | 71.55 | 71.52 | 71.26 | 71.48 |  11.69M
WResNet18_A [44] | 70.06 | 69.24 | 69.91 | 69.98 | 70.31 | 70.52 | 11.69M
WResNet18_C [42] | 71.94 | 71.75 | 71.66 | 71.99 | 72.03 | 71.88 | 21.62M
1-8[2pt/2pt] WResNet34 | 74.35 | 74.33 | 74.51 | 74.61 | 74.34 | 74.30 | 21.80M
| Noise mCE (lower is better) |
WResNet18 | 80.91 | 80.38 | 81.02 | 82.19 | 83.77 | 82.54 |
WResNet18_A [44] | 83.17 | 86.02 | 86.07 | 85.22 | 82.96 | 84.01 |
WResNet18_C [42] | 81.79 | 83.67 | 83.51 | 82.13 | 82.60 | 80.11 |
1-8[2pt/2pt] WResNet34 | 76.64 | 77.61 | 74.30 | 76.19 | 76.00 | 72.73 |

Different with our wavelet based down-sampling (Fig. 2(b)), there are other
wavelet integrated down-sampling modes in literatures. In [44], the authors
adopt as down-sampling output the average value of the multiple components of
wavelet transform, as Fig. 8(a) shows. In [42], the authors concatenate all
the components output from DWT, and process them in a unified way, as Fig.
8(b) shows.

Taking ResNet18 as backbone, we compare our wavelet based down-sampling with
the previous approaches, in terms of classification accuracy and noise-
robustness. We rebuild ResNet18 using the three down-sampling modes shown in
Fig. 2(b) and Fig. 8, and denote them as WResNet18, WResNet18_A, and
WResNet18_C, respectively. We train them on ImageNet when various wavelets are
used. Table VI shows the accuracy on ImageNet and the noise mCEs on the
ImageNet-C. Generally, the networks using wavelet based down-sampling achieve
better accuracy and noise mCE than that of original ResNet18 ($69.76\%$
accuracy and $88.97$ mCE).

Similar to WResNet18, the number of parameters of WResNet18_A is the same with
that of original ResNet18. However, the added high-frequency components in the
feature maps damage the information contained in the low-frequency component,
because of the high-frequency noise. WResNet18_A performs the worst among the
networks using wavelet based down-sampling.

TABLE VII: Accuracy of WaveCNets on adversarial samples. The adversarial
noises significantly decrease the classification accuracy of well-trained
CNNs, while the wavelets could increase the performance of CNNs on the various
adversarial samples.

Network | Attacker
---|---
FGSM | IFGSM | iterLL | RFGSM | CW | PGD
WResNet101 | none* | 69.43 | 66.53 | 72.25 | 50.52 | 68.76 | 53.94
haar | 70.91 ($+$1.47) | 68.33 ($+$1.80) | 73.32 ($+$1.08) | 53.22 ($+$2.70) | 70.03 ($+$1.27) | 57.30 ($+$3.36)
2-8[2pt/2pt] | ch2.2 | 71.81 ($+$2.38) | 69.22 ($+$2.69) | 74.33 ($+$2.09) | 54.08 ($+$3.56) | 70.82 ($+$2.06) | 58.28 ($+$4.35)
| ch3.3 | 71.55 ($+$2.12) | 68.83 ($+$2.29) | 74.22 ($+$1.97) | 52.86 ($+$2.34) | 70.86 ($+$2.10) | 56.99 ($+$3.06)
| ch4.4 | 71.32 ($+$1.89) | 68.58 ($+$2.05) | 74.02 ($+$1.77) | 53.15 ($+$2.63) | 70.45 ($+$1.69) | 56.84 ($+$2.90)
| ch5.5 | 71.31 ($+$1.88) | 68.61 ($+$2.07) | 74.03 ($+$1.78) | 52.62 ($+$2.10) | 70.87 ($+$2.11) | 56.93 ($+$3.00)
2-8[2pt/2pt] | db2 | 71.66 ($+$2.23) | 68.81 ($+$2.28) | 74.25 ($+$2.00) | 52.56 ($+$2.04) | 70.75 ($+$1.99) | 56.80 ($+$2.86)
| db3 | 70.47 ($+$1.04) | 67.71 ($+$1.18) | 73.06 ($+$0.81) | 51.67 ($+$1.15) | 69.22 ($+$0.46) | 55.69 ($+$1.75)
| db4 | 69.56 ($+$0.13) | 66.89 ($+$0.36) | 72.12 ($-$0.13) | 51.10 ($+$0.58) | 68.11 ($-$0.65) | 55.17 ($+$1.23)
| db5 | 68.88 ($-$0.55) | 66.21 ($-$0.32) | 71.08 ($-$1.17) | 50.16 ($-$0.36) | 67.16 ($-$1.60) | 54.41 ($+$0.47)
| db6 | 67.85 ($-$1.58) | 65.12 ($-$1.41) | 69.98 ($-$2.27) | 49.45 ($-$1.07) | 65.74 ($-$3.02) | 53.74 ($-$0.20)
WDenseNet121 | none | 65.92 | 62.46 | 68.38 | 46.11 | 64.66 | 48.74
haar | 67.27 ($+$1.35) | 63.92 ($+$1.46) | 69.79 ($+$1.41) | 47.92 ($+$1.81) | 66.35 ($+$1.69) | 51.58 ($+$2.84)
2-8[2pt/2pt] | ch2.2 | 67.81 ($+$1.89) | 64.59 ($+$2.13) | 70.15 ($+$1.77) | 48.91 ($+$2.80) | 66.13 ($+$1.47) | 52.43 ($+$3.69)
| ch3.3 | 67.62 ($+$1.70) | 64.45 ($+$1.99) | 70.13 ($+$1.75) | 48.35 ($+$2.34) | 66.31 ($+$1.65) | 51.83 ($+$3.09)
| ch4.4 | 67.75 ($+$1.84) | 64.79 ($+$2.33) | 70.18 ($+$1.80) | 48.83 ($+$2.72) | 66.47 ($+$1.81) | 52.53 ($+$3.78)
| ch5.5 | 67.71 ($+$1.79) | 64.60 ($+$2.14) | 70.11 ($+$1.73) | 48.43 ($+$2.31) | 66.72 ($+$2.06) | 51.96 ($+$3.22)
2-8[2pt/2pt] | db2 | 67.39 ($+$1.47) | 64.17 ($+$1.71) | 69.98 ($+$1.60) | 47.64 ($+$1.53) | 66.73 ($+$2.07) | 51.11 ($+$2.37)
| db3 | 67.18 ($+$1.26) | 63.81 ($+$1.35) | 69.65 ($+$1.27) | 47.54 ($+$1.43) | 65.83 ($+$1.17) | 51.02 ($+$2.28)
| db4 | 66.02 ($+$0.10) | 62.80 ($+$0.34) | 68.27 ($-$0.11) | 46.60 ($+$0.49) | 64.49 ($-$0.17) | 50.19 ($+$1.45)
| db5 | 64.85 ($-$1.07) | 61.63 ($-$0.83) | 66.81 ($-$1.57) | 46.06 ($-$0.05) | 62.95 ($-$1.71) | 49.42 ($+$0.68)
| db6 | 63.47 ($-$2.45) | 60.30 ($-$2.16) | 65.16 ($-$3.22) | 44.75 ($-$1.36) | 61.18 ($-$3.48) | 48.56 ($-$0.18)

  * *

corresponding to the accuracy of original CNNs on the adversarial samples,
i.e., ResNet101, DenseNet121.

Due to the tensor concatenation, WResNet18_C employs much more parameters
($21.62\times 10^{6}$) than WResNet18 and WResNet18_A ($11.69\times 10^{6}$).
WResNet18_C thus increase the accuracy of WResNet18 by $0.11\%$ to $0.77\%$,
when various wavelets are used. However, due to the included noise, the
concatenation does not evidently improve the noise-robustness. In addition,
the amount of parameters for WResNet18_C is almost the same with that for
WResNet34 ($21.80\times 10^{6}$), while the accuracy and noise mCE of
WResNet34 are obviously superior to that of WResNet18_C.

###  IV-E Adversarial robustness

A lot of researches [2, 3, 4, 5, 6, 7] have shown that CNNs can be fooled by
adversarial samples, which are generated by adding specially designed noises
to the normal samples. While such noises are usually high-frequency components
[59] and not noticeable to human eyes, we believe that our WaveCNets can
filter such noises and increase the robustness of the CNNs against such
adversarial attacks.

We test the adversarial robustness of WaveCNets using six attack algorithms,
i.e., fast gradient sign method (FGSM) [2], iterative FGSM (IFGSM) [3],
iterative least-likerly class attack (iterLL) [3], random FGSM (RFGSM) [4],
Carlini and Wagner’s $L_{2}$ attacker (CW) [5], and projected gradient descent
(PGD) [6]. Using a public code of the above
attackers222/https://github.com/Harry24k/adversarial-attacks-pytorch, we
generate adversarial samples using the ImageNet validation set with the
scenario of black-box attacks, and the attack model is “InceptionV3” trained
on ImageNet.

Table VII shows the accuracy of WaveCNets (WResNet101 and WDenseNet121) on the
six adversarial sample sets; the classifiers are only trained on the clean
ImageNet training set. In Table VII, “none” corresponds to the accuracy of
original CNNs (ResNet101 and DenseNet121) on ImageNet validation set attracted
by the six attack algorithms. The parenthesized numbers are accuracy
differences of WaveCNets compared with the CNN results. From Table VII, we
find that the wavelets could increase the accuracy of CNNs on the adversarial
samples. Take PGD attack for example, our WResNet101 using different wavelets
generally achieve 3% higher accuracy than the original ResNet101 (53.94%). The
best performance is achieved by “ch2.2”, which is abut 4.35% higher than that
of ResNet101.

While our WaveCNets achieve consistent resistance to various attackers, their
performances are inferior to that of the specially trained defense methods
[7]. With adversarial training, the feature denoising method proposed in [7]
achieves the state-of-the-art result in defending adversarial attackers.
However, as the authors point out in their paper, the custom-designed
denoising block requires residual connection for stable training of their deep
networks. It seems that the spatial filtering used in their method, such as
mean filtering and median filtering, tend to perform denoising in the whole
frequency domain, which might easily break basic object structures in the
feature maps. In contrast, the general DWT module in WaveCNets denoises the
feature maps in the high-frequency interval, which can keep the basic object
structure and thus lead to increased adversarial-robustness without
requirement of adversarial training.

###  IV-F Object detection

To further illustrate the efficiency of wavelets in deep learning, we conduct
detection experiments on COCO [14] benchmark, by comparing the performance of
faster R-CNN [15], RetinaNet [16], and their wavelet integrated versions. We
apply the original off-the-shelf detectors in mmdetection [60], and build
their wavelet integrated versions by replacing original backbones (ResNet50
and ResNet101) with WaveCNets (WResNet50 and WResNet101). In the wavelet
integrated detectors, the DWT layer is applied to down-sample the feature maps
and suppress the aliasing effects. Wavelet “haar” and “ch3.3” are applied into
these two backbones, since they show the best performance on ImageNet
classification task.

TABLE VIII:  Results of wavelet integrated detectors on COCO. The detection
performance of original detectors are taken as baseline. For all detectors,
our method could improve their detection performance. Wavelet consistently
increase the detectors’ performance on all three categories of objects, i.e.,
small, medium, and large objects.

| Backbone | initial lr | BSa | Epoch | FPS | minival | test-dev
---|---|---|---|---|---|---|---
| AP | AP50 | AP75 | APS | APM | APL | AP | AP50 | AP75 | APS | APM | APL
faster R-CNN | ResNet50 | 0.02 | 16 | 12 | 18.1 | $37.3$ | $58.2$ | $40.4$ | $21.4$ | $41.0$ | $48.5$ | $37.7$ | $58.7$ | $41.0$ | $21.8$ | $40.6$ | $46.8$
WResNet50(haar) | 0.02 | 16 | 12 | 16.3 | 38.5 | 59.8 | 41.7 | 22.6 | 42.4 | 49.2 | 38.7 | 60.2 | 42.1 | 22.7 | 41.7 | 47.8
|  |  |  | -1.8 | +1.2 | +1.6 | +1.3 | +1.2 | +1.4 | +0.7 | +1.0 | +1.5 | +1.1 | +0.9 | +1.1 | +1.0
2-18[2pt/2pt] | ResNet50 | 0.02 | 16 | 24b | 18.1 | $38.4$ | $59.0$ | $42.0$ | $21.5$ | $42.1$ | $50.3$ | $38.7$ | $59.6$ | $42.1$ | $22.1$ | $41.4$ | $48.6$
|  |  |  |  | ${}^{~{}~{}~{}0}$ | +1.1 | +0.8 | +1.6 | +0.1 | +1.1 | +1.8 | +1.0 | +0.9 | +1.1 | +0.3 | +0.8 | +1.8
| WResNet50(haar) | 0.01 | 8 | 24 | 16.3 | 39.4 | 60.7 | 42.6 | 22.1 | 43.3 | 50.3 | 39.5 | 60.6 | 43.0 | 22.8 | 43.2 | 49.4
|  |  |  |  | -1.8 | +2.1 | +2.5 | +2.2 | +0.7 | +2.3 | +1.8 | +1.8 | +1.9 | +2.0 | +1.0 | +2.6 | +2.6
| ResNet101 | 0.01 | 8 | 12 | 14.8 | $39.5$ | $60.1$ | $43.3$ | $22.3$ | $43.5$ | $51.5$ | $39.8$ | $60.9$ | $43.4$ | $23.0$ | $43.0$ | $50.0$
| WResNet101(ch3.3) | 0.01 | 8 | 12 | 12.5 | 40.8 | 61.8 | 44.6 | 23.5 | 44.9 | 53.0 | 41.2 | 62.6 | 45.1 | 24.1 | 44.7 | 51.2
|  |  |  |  | -2.3 | +1.3 | +1.7 | +1.3 | +1.2 | +1.4 | +1.5 | +1.4 | +1.7 | +1.7 | +1.1 | +1.7 | +1.2
2-18[2pt/2pt] | ResNet101 | 0.02 | 16 | 24 | 14.8 | $39.8$ | $60.1$ | $43.3$ | $22.5$ | $43.6$ | $52.8$ | $40.5$ | $61.2$ | $44.1$ | $22.3$ | $43.7$ | $51.6$
|  |  |  |  | ${}^{~{}~{}~{}0}$ | +0.3 | +0.0 | +0.0 | +0.2 | +0.1 | +1.3 | +0.7 | +0.3 | +0.7 | -0.7 | +0.7 | +1.6
| WResNet101(ch3.3) | 0.01 | 8 | 24 | 12.5 | 40.9 | 61.5 | 44.6 | 23.1 | 44.9 | 54.1 | 41.5 | 62.4 | 45.1 | 23.5 | 44.6 | 52.5
|  |  |  |  | -2.3 | +1.4 | +1.4 | +1.3 | +0.8 | +1.4 | +2.6 | +1.7 | +1.5 | +1.7 | +0.5 | +1.6 | +2.5
RetinaNet | ResNet50 | 0.005 | 8 | 12 | 17.2 | $36.5$ | $55.9$ | $39.0$ | $21.0$ | $40.2$ | $46.4$ | $36.9$ | $56.3$ | $39.4$ | $20.8$ | $39.9$ | $45.3$
WResNet50(haar) | 0.005 | 8 | 12 | 15.2 | 37.4 | 57.0 | 40.2 | 21.9 | 41.4 | 48.0 | 37.7 | 57.6 | 40.4 | 22.0 | 40.6 | 45.9
|  |  |  | -2.0 | +0.9 | +1.1 | +1.2 | +0.9 | +1.2 | +1.6 | +0.8 | +1.3 | +1.0 | +1.2 | +0.7 | +0.6
2-18[2pt/2pt] | ResNet50 | 0.01 | 16 | 24 | 17.2 | $37.4$ | $56.7$ | $39.6$ | $20.0$ | $40.7$ | 49.7 | $37.7$ | $57.2$ | $40.2$ | $20.4$ | $40.2$ | 48.1
|  |  |  |  | ${}^{~{}~{}~{}0}$ | +0.9 | +0.8 | +0.6 | -1.0 | +0.5 | +3.3 | +0.8 | +0.9 | +0.8 | -0.4 | +0.3 | +2.8
| WResNet50(haar) | 0.005 | 8 | 24 | 15.2 | 38.0 | 57.5 | 40.4 | 21.5 | 41.3 | $49.4$ | 38.2 | 58.1 | 40.8 | 21.4 | 40.6 | $47.6$
|  |  |  |  | -2.0 | +1.5 | +1.6 | +1.4 | +0.5 | +1.1 | +3.0 | +1.3 | +1.8 | +1.4 | +0.6 | +0.7 | +2.3
| ResNet101 | 0.005 | 8 | 12 | 13.6 | $38.5$ | $57.9$ | $41.4$ | $22.1$ | $43.2$ | $49.7$ | $39.1$ | $58.7$ | $41.9$ | $22.4$ | $42.3$ | $48.7$
| WResNet101(ch3.3) | 0.005 | 8 | 12 | 11.5 | 40.1 | 60.1 | 43.1 | 23.7 | 44.2 | 51.9 | 40.2 | 60.6 | 43.1 | 23.4 | 43.7 | 49.7
|  |  |  |  | -2.1 | +1.6 | +2.2 | +1.7 | +1.6 | +1.0 | +2.2 | +1.1 | +1.9 | +1.2 | +1.0 | +1.4 | +1.0
2-18[2pt/2pt] | ResNet101 | 0.01 | 16 | 24 | 13.6 | $38.9$ | $58.0$ | $41.5$ | $21.0$ | $42.8$ | $52.4$ | $39.6$ | $59.1$ | $42.4$ | $21.3$ | $42.6$ | 51.1
|  |  |  |  | ${}^{~{}~{}~{}0}$ | +0.3 | +0.1 | +0.1 | -1.1 | -0.4 | +2.7 | +0.5 | +0.4 | +0.5 | -1.1 | +0.3 | +2.4
| WResNet101(ch3.3) | 0.005 | 8 | 24 | 11.5 | 39.9 | 59.6 | 42.8 | 22.4 | 43.7 | 52.7 | 40.1 | 60.1 | 43.1 | 22.4 | 43.1 | $50.6$
|  |  |  |  | -2.1 | +1.4 | +1.7 | +1.4 | +0.3 | +0.5 | +3.0 | +1.0 | +1.4 | +1.2 | +0.0 | +0.8 | +1.9

  * a

in mmdetection, the default ratio of initial learning rate to batch size is
$\frac{1}{800}$ for faster R-CNN, and $\frac{1}{1600}$ for RetinaNet.

  * b

the original detectors trained for 24 epochs are downloaded from mmdetection.
In their training, the learning rates are decayed at epoch 16 and 22.

(a) Two example images detected by RetinaNet.

(b) Two example images detected by Faster R-CNN.

Figure 9:  Four example images detected by RetinaNet and Faster R-CNN,
integrated with or without wavelet. The manual detections are taken as ground
truths. While the baseline detectors (trained for 24 epochs) falsely detect or
miss the small target objects, the detectors integrated with wavelet (trained
for only 12 epochs) correctly detect the target objects.

With the default hyper-parameters, the detectors are trained on trainval135k
(115K images) of COCO. Table VIII shows the hyper-parameter setting. We train
each detector for 12 epochs from scratch, and drop the learning rate by a
factor of 0.1 at epoch 8 and 11. The trained detectors are tested on the
minival (5K images) and test-dev (20K images) sets; Table VIII shows the
detection results. In the experiments, AP, AP50, AP75, APS, APM, and APL are
used to evaluate the detection performance, where

  * •

AP: mean value of 10 average precisions (AP) for intersection over union (IoU)
larger than 0.50, 0.55, 0.60, $\cdots$, and 0.95;

  * •

AP50: AP at IoU $\geq 0.50$;

  * •

AP75: AP at IoU $\geq 0.75$;

  * •

APS: AP for small objects (area $\leq 32^{2}$);

  * •

APM: AP for medium objects ($32^{2}<$ area $\leq 96^{2}$);

  * •

APL: AP for large objects ($96^{2}<$ area).

The performances of original detectors trained by 12 epochs are taken as
baseline, and the numbers with smaller size in Table VIII indicate the
performance difference with the baseline. For comparison, we also list the
detection performance of original detectors trained by 24 epochs.

As shown in Table VIII, with 12-epoch training, the detection performance of
wavelet integrated detectors are superior to that of the original detectors,
and comparable to or even better than that of original detectors trained by 24
epochs. Taking faster R-CNN with ResNet101 for example, on test-dev set, the
AP of the 12-epoch trained detector is $39.8\%$, which is increased by $1.4\%$
to $41.2\%$, after integrating with wavelet “ch3.3”. The increased AP is even
higher than that of original detector ($40.5\%$) trained by 24 epochs.
Comparing their detection performance on the objects with different sizes, one
can find that the wavelet could consistently improve the detector performance
on the all three categories of objects. For faster R-CNN with ResNet101 on
test-dev set, the wavelet integrated detector increases the performance on the
small, medium, and large objects by $1.1\%,1.7\%$, and $1.2\%$, respectively.
While extension of training time to 24 epochs could further increase the APM
and APL of the baseline detector by $0.7\%$ and $1.6\%$, such extension
decreases APS by $0.7\%$. Since aliasing effects are available in original
detectors, extending training time seems to enlarge such effects. As the basic
structures of small objects are more easily damaged by the aliasing effects,
the detection performance on small objects decreases with the increase of
training epochs. In contrast, wavelet could suppress the aliasing effects in
the deep networks, and consistently increase the performance of detectors for
objects with different sizes.

Fig. 9 visually shows four example images detected by various detectors, i.e.,
Faster R-CNNs and RetinaNets with different backbones (ResNet101 and
WResNet101). In Fig. 9, the first column shows the manual detections, which
are taken as ground truths. While the second and third columns show the images
detected by detectors trained for 12 epochs, integrated with or without using
wavelet, the fourth column shows the images detected by the one trained for 24
epochs without using wavelet. As Fig. 9 shows, the detectors without using
wavelet usually falsely detect or miss the small target objects in the images,
while the detectors integrated with wavelet correctly detect all target
objects. Taking the “bench” image (shown in the first row of Fig. 9) for
example, the confidence of detected bench increases from $0.810$ to $0.861$
when wavelet “ch3.3” is integrated to the backbone of ResNet101. Though
increase of training epoch (24) could also improve such confidence of baseline
RetinaNet with backbone of ResNet101, it falsely detects some poles as birds.

Compared with the commonly used down-sampling, DWT needs more multiply-add
operations, which would decreases the detection speed. As shown in the 6th
column of Table VIII, the detection speed of faster R-CNN slightly decreases
from 18.1 FPS (frames per second) to 16.3 FPS, when wavelet “haar” is
integrated to the backbone of ResNet50 and Nvidia Tesla V100 GPU is used as
the computing platform.

###  IV-G 3D object classification

(a) 3D point cloud data.

(b) 3D tensor.

Figure 10:  An example of 3D point cloud in ModelNet40. The 3D point cloud data is transformed to 3D tensor with size of $32\times 32\times 32$, as the input of deep networks.  TABLE IX: Classification results on ModelNet40 of 3D WaveCNets. The symmetric Haar and Cohen wavelets can consistently improve the performance of 3D CNNs, while the improvements of asymmetric Daubechies wavelet are not guaranteed. architecture | 3D CNN | 3D WaveCNet
---|---|---
(baseline) | haar | ch2.2 | ch3.3 | ch4.4 | ch5.5 | db2 | db3 | db4 | db5 | db6
3D VGG9 | $87.08$ | $88.01_{+0.93}$ | $87.73_{+0.65}$ | $87.85_{+0.77}$ | $88.09_{+1.01}$ | 88.25+1.17 | $87.40_{+0.32}$ | $87.36_{+0.28}$ | $86.06_{-1.02}$ | $84.97_{-2.11}$ | $81.43_{-5.65}$
3D ResNet16 | $84.32$ | $85.41_{+1.09}$ | 85.90+1.58 | $85.37_{+1.05}$ | $85.62_{+1.30}$ | $85.86_{+1.54}$ | $85.62_{+1.30}$ | $85.70_{+1.38}$ | $85.43_{+1.11}$ | $85.33_{+1.01}$ | $85.13_{+0.81}$

We conduct 3D object classification experiments on 3D point cloud dataset
ModelNet40 [61], using 3D VGG and ResNet. ModelNet40 consists of 12311 (9843
for training and 2468 for test) 3D point cloud data captured from 40
categories of objects. As shown in Fig. 10(a), the point cloud data
representing a 3D object contains a series of 3D coordinates. As shown in Fig.
10(b), we transform the 3D point cloud data to 3D tensor with size of
$32\times 32\times 32$, as the input of deep networks. We perform the
classification experiments using 3D VGG9, 3D ResNet16, and their wavelet
integrated versions (3D WVGG9 and WResNet16). As 3D versions of VGG and
ResNet, 3D VGG9 and ResNet16 are designed by removing the last two
convolutional layers of 3D VGG11 and ResNet18. The two 3D WaveCNets, 3D WVGG9
and WResNet16, apply 3D DWT to down-sample 3D feature maps.

Using the ModelNet40 training set, we train 3D WVGG9 and WResNet16 from
scratch for 80 epochs, with batch size of 256 and initial learning rate of
$0.1$. The learning rate is dropped by a factor of 0.1 every 25 epochs. For
comparison, we also train the 3D VGG9 and 3D ResNet16, using the same hyper-
parameter setting. We evaluate their performance on the test set of
ModelNet40, and list the results in Table IX. In Table IX, “3D CNN (baseline)”
corresponds to the results of original 3D VGG9 and ResNet16, and “3D WaveCNet”
represents that of 3D WVGG9 and WResNet16, with different wavelets like Haar
(“haar”), Cohen (“ch$p.\tilde{p}$”), and Daubechies (“db$p$”). On the test
set, the accuracy of 3D VGG9 and ResNet16 are 87.08% and 84.32%, respectively.
The integration of Cohen wavelet “ch5.5” and “ch2.2” improve the accuracy to
88.25% and 85.90%, respectively. Similarly, Haar and Cohen wavelets also
improve the performance of both 3D VGG9 and 3D ResNet16. While all of
different Daubechies wavelets could improve the accuracy of 3D ResNet16,
“db4”, “db5”, and “db6” decrease the performance of 3D VGG9.

In summary, similar to the results of 2D image classification, the symmetric
Haar and Cohen wavelets can consistently improve the performance of 3D CNNs,
while the improvements of asymmetric Daubechies wavelet are not guaranteed.

##  V Conclusions

In deep networks, the commonly used down-sampling operations result in the
aliasing effects on the feature maps, accumulating noise and breaking the
object structures, which leads to the weak noise-robustness of CNNs for image
classification. To suppress the aliasing effects, we design the general
discrete wavelet transform (DWT) and inverse DWT (IDWT) layers, and propose
wavelet integrated convolutional networks (WaveCNets) by replacing the down-
sampling operations in the common CNNs with DWT. During the inference of
WaveCNets, wavelet helps the networks to keep the basic object structures and
resist the noise propagation. WaveCNets achieve higher image classification
accuracy, better noise-robustness, and increased adversarial robustness with
various commonly used CNN architectures on ImageNet. Our method can also
improve the detection performance of faster R-CNN and RetinaNet on COCO
benchmark.

In future, we will exhaustively study the application of DWT/IDWT layers in
the image-to-image tasks. We will also explore the applications of 3D DWT/IDWT
integrated deep networks for 3D medical image processing.

## References

  * [1] D. Hendrycks and T. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” _arXiv preprint arXiv:1903.12261_ , 2019.
  * [2] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [3] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” _arXiv preprint arXiv:1607.02533_ , 2016.
  * [4] F. Tramer, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: attacks and defences,” _arXiv preprint arXiv:1705.07204_ , 2017.
  * [5] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” _arXiv preprint arXiv:1608.04644_ , 2016.
  * [6] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [7] C. Xie, Y. Wu, L. v. d. Maaten, A. L. Yuille, and K. He, “Feature denoising for improving adversarial robustness,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 501–509.
  * [8] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, “Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness,” _arXiv preprint arXiv:1811.12231_ , 2018.
  * [9] H. Nyquist, “Certain topics in telegraph transmission theory,” _Transactions of the American Institute of Electrical Engineers_ , vol. 47, no. 2, pp. 617–644, 1928.
  * [10] S. G. Mallat, “A theory for multiresolution signal decomposition: the wavelet representation,” _IEEE Transactions on Pattern Analysis & Machine Intelligence_, no. 7, pp. 674–693, 1989.
  * [11] I. Daubechies, _Ten lectures on wavelets_. Siam, 1992, vol. 61.
  * [12] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [13] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [14] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _European Conference on Computer Vision (ECCV)_. Springer, 2014, pp. 740–755.
  * [15] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 6, pp. 1137–1149, 2017\.
  * [16] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_. IEEE, Oct 2017, pp. 2980–2988.
  * [17] Q. Li, L. Shen, S. Guo, and Z. Lai, “Wavelet integrated cnns for noise-robust image classification,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [18] W. Brendel and M. Bethge, “Approximating cnns with bag-of-local-features models works surprisingly well on imagenet,” _arXiv preprint arXiv:1904.00760_ , 2019.
  * [19] F. Liao, M. Liang, Y. Dong, T. Pang, X. Hu, and J. Zhu, “Defense against adversarial attacks using high-level representation guided denoiser,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 1778–1787.
  * [20] D. Yu, H. Wang, P. Chen, and Z. Wei, “Mixed pooling for convolutional neural networks,” in _International Conference on Rough Sets and Knowledge Technology_. Springer, 2014, pp. 364–375.
  * [21] M. D. Zeiler and R. Fergus, “Stochastic pooling for regularization of deep convolutional neural networks,” _arXiv preprint arXiv:1301.3557_ , 2013.
  * [22] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [23] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [24] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 4510–4520.
  * [25] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [26] A. Azulay and Y. Weiss, “Why do deep convolutional networks generalize so poorly to small image transformations?” _arXiv preprint arXiv:1805.12177_ , 2018.
  * [27] R. Zhang, “Making convolutional networks shift-invariant again,” _arXiv preprint arXiv:1904.11486_ , 2019.
  * [28] Q. Li and L. Shen, “Wavesnet: Wavelet integrated deep networks for image segmentation,” _arXiv preprint arXiv:2005.14461_ , 2020.
  * [29] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE transactions on PAMI_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [30] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on MICCAI_. Springer, 2015, pp. 234–241.
  * [31] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” _arXiv preprint arXiv:1412.7062_ , 2014.
  * [32] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the ECCV_ , 2018, pp. 801–818.
  * [33] J. Bruna and S. Mallat, “Invariant scattering convolution networks,” _IEEE transactions on PAMI_ , vol. 35, no. 8, pp. 1872–1886, 2013.
  * [34] W. Thomas and H. B$\ddot{o}$lcskei, “A mathematical theory of deep convolutional neural networks for feature extraction,” _IEEE Transactions on Information Theory_ , vol. 64, no. 3, pp. 1845–1866, 2018.
  * [35] Q. Zhang and A. Benveniste, “Wavelet networks,” _IEEE transactions on Neural Networks_ , vol. 3, no. 6, pp. 889–898, 1992.
  * [36] H. H. Szu, B. A. Telfer, and S. L. Kadambe, “Neural network adaptive wavelets for signal representation and classification,” _Optical Engineering_ , vol. 31, no. 9, pp. 1907–1917, 1992.
  * [37] D. De Silva, H. Vithanage, K. Fernando, and I. Piyatilake, “Multi-path learnable wavelet neural network for image classification,” _arXiv preprint arXiv:1908.09775_ , 2019.
  * [38] H. Huang, R. He, Z. Sun, and T. Tan, “Wavelet-srnet: A wavelet-based cnn for multi-scale face super resolution,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2017, pp. 1689–1697.
  * [39] Y. Liu, Q. Li, and Z. Sun, “Attribute enhanced face aging with wavelet-based generative adversarial networks,” _arXiv preprint arXiv:1809.06647_ , 2018\.
  * [40] B. A. Savareh, H. Emami, M. Hajiabadi, S. M. Azimi, and M. Ghafoori, “Wavelet-enhanced convolutional neural network: a new idea in a deep learning paradigm,” _Biomedical Engineering/Biomedizinische Technik_ , vol. 64, no. 2, pp. 195–205, 2019.
  * [41] B. Yuan, C. Wang, F. Jiang, M. Long, P. S. Yu, and Y. Liu, “Waveletfcnn: A deep time series classification model for wind turbine blade icing detection,” _arXiv preprint arXiv:1902.05625_ , 2019.
  * [42] P. Liu, H. Zhang, K. Zhang, L. Lin, and W. Zuo, “Multi-level wavelet-cnn for image restoration,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 773–782.
  * [43] T. Williams and R. Li, “Wavelet pooling for convolutional neural networks,” in _International Conference on Learning Representations_ , 2018.
  * [44] Y. Duan, F. Liu, L. Jiao, P. Zhao, and L. Zhang, “Sar image segmentation based on convolutional-wavelet neural network and markov random field,” _Pattern Recognition_ , vol. 64, pp. 255–267, 2017.
  * [45] J. Yoo, Y. Uh, S. Chun, B. Kang, and J.-W. Ha, “Photorealistic style transfer via wavelet transforms,” _arXiv preprint arXiv:1903.09760_ , 2019.
  * [46] Y. LeCun, L. Bottou, Y. Bengio, P. Haffner _et al._ , “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [47] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” Citeseer, Tech. Rep., 2009.
  * [48] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” 2011.
  * [49] D. Lundqvist, A. Flykt, and A. Öhman, “The karolinska directed emotional faces (kdef),” _CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet_ , vol. 91, p. 630, 1998.
  * [50] F. Keinert, _Wavelets and multiwavelets_. Chapman & Hall / CRC Press, 2004.
  * [51] I. W. Selesnick, R. G. Baraniuk, and N. G. Kingsbury, “The dual-tree complex wavelet transform,” _IEEE Signal Processing Magazine_ , vol. 22, no. 6, pp. 123–151, 2005.
  * [52] M. Jianwei and F. Markus, “Combined complex ridgelet shrinkage and total variation minimization,” _SIAM Journal on Scientific Computing_ , vol. 28, no. 3, pp. 984–1000, 2006.
  * [53] E. Candès, L. Demanet, D. Donoho, and L. Ying, “Fast discrete curvelet transforms,” _Multiscale Modeling & Simulation_, vol. 5, no. 3, 2006.
  * [54] E. L. Pennec and S. Mallat, “Image compression with geometrical wavelets,” in _Proceedings of the IEEE International Conference on Image Processing_ , 2000, pp. 661–664.
  * [55] M. N. Do and M. Vetterli, “The contourlet transform: an efficient directional multiresolution image representation,” _IEEE Transactions on Image Processing_ , vol. 14, no. 12, pp. 2091–2106, 2005.
  * [56] D. L. Donoho, “De-noising by soft-thresholding,” _IEEE transactions on information theory_ , vol. 41, no. 3, pp. 613–627, 1995.
  * [57] D. L. Donoho and J. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” _biometrika_ , vol. 81, no. 3, pp. 425–455, 1994.
  * [58] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [59] H. Wang, X. Wu, Z. Huang, and E. P. Xing, “High-frequency component helps explain the generalization of convolutional neural networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8684–8694.
  * [60] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng, C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang, J. Shi, W. Ouyang, C. C. Loy, and D. Lin, “MMDetection: Open mmlab detection toolbox and benchmark,” _arXiv preprint arXiv:1906.07155_ , 2019.
  * [61] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1912–1920.

|  Qiufu Li received the B.S. degree in mathematics and information science
from North Minzu University, Yinchuan, China, M.S. degree in wavelet analysis
and signal processing from North Minzu University, and Ph.D degree from
Beijing Insitute of Technology, Beijing China, in 2010, 2013, and 2017,
respectively. He has been a postdoctoral fellow at School of Computer Science
and Software Engineering, Shenzhen University, Shenzhen, China. He is
currently research associate fellow at the same school. His research interests
are in the areas of wavelet analysis, digital image processing, and deep
learning.
---|---
|  Linlin Shen is currently the “Pengcheng Scholar” Distinguished Professor at
School of Computer Science and Software Engineering, Shenzhen University,
Shenzhen, China. He is also an Honorary professor at School of Computer
Science, University of Nottingham, UK. He serves as the director of Computer
Vision Institute, AI Research Center for Medical Image Analysis & Diagnosis
and China-UK joint research lab for visual information processing. He also
serves as the Co-Editor-in-Chief of the IET journal of Cognitive Computation
and Systems. He received the BSc and MEng degrees from Shanghai JiaoTong
University, Shanghai, China, and the Ph.D. degree from the University of
Nottingham, Nottingham, U.K. He was a Research Fellow with the University of
Nottingham, working on MRI brain image processing. His research interests
include deep learning, facial analysis and medical image processing. Prof.
Shen is listed as the Most Cited Chinese Researcher by Elsevier. He received
the Most Cited Paper Award from the journal of Image and Vision Computing. His
cell classification algorithms were the winners of the International Contest
on Pattern Recognition Techniques for Indirect Immunofluorescence Images held
by ICIP 2013 and ICPR 2016.
---|---
|  Sheng Guo received the Ph.D degree from Shenzhen Institutes of Advanced
Technology, Chinese Academy of Sciences, Shenzhen, China, in 2017. He is
currently a senior algorithm expert with Ant Financial. His current research
interests are object classification, scene/object detection, video
classification. He was the first runner-up at the ImageNet Large Scale Visual
Recognition Challenge 2015 in scene recognition and winner at WebVision
Challenge 2017.
---|---
|  Zhihui Lai received the B.S. degree in mathematics from South China Normal
University, M.S. degree from Jinan University, and the Ph.D degree in pattern
recognition and intelligence system from Nanjing University of Science and
Technology (NUST), China, in 2002, 2007 and 2011, respectively. He has been a
research associate, postdoctoral fellow and research fellow since 2010 at The
Hong Kong Polytechnic University. He also has been a Postdoctoral Fellow at
Bio-Computing Research Center, Shenzhen Graduate School, Harbin Institute of
Technology (HIT) for 2011-2013. He has published over 100 scientific articles,
including more than 30 IEEE transactions. He won the best paper award in 2008
Chinese Conference on Pattern Recognition. He was recognized as Overseas High-
Caliber Personnel of Shenzhen (Peacock Plan) in 2015. Now, he is a professor
of the College of Computer Science and Software Engineering, Shenzhen
University. Dr. Lai is a reviewer of more than 20 international journals,
including the IEEE TPAMI, TNNLS, TIP, TCSVT, T-Cybernetics, Pattern
Recognition, Information Sciences and so on. He is currently an Associate
Editor of the International Journal of Machine Learning and Cybernetics.
---|---