# Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and
Activation Function

Md Tahmid Hossain, Shyh Wei Teng, Ferdous Sohel, Guojun Lu Md Tahmid Hossain,
Shyh Wei Teng, and Guojun Lu are with the School of Engineering, Information
Technology and Physical Sciences, Federation University Australia, Gippsland
Campus Churchill, VIC 3842, Australia. E-mail:
<EMAIL_ADDRESS>Sohel is with
the Discipline of Information Technology , Murdoch University, Perth, WA 6150,
Australia. E-mail<EMAIL_ADDRESS>
###### Abstract

Deep convolutional networks are vulnerable to image translation or shift,
partly due to common down-sampling layers, e.g., max-pooling and strided
convolution. These operations violate the Nyquist sampling rate and cause
aliasing. The textbook solution is low-pass filtering (blurring) before down-
sampling, which can benefit deep networks as well. Even so, non-linearity
units, such as ReLU, often re-introduce the problem, suggesting that blurring
alone may not suffice. In this work, first, we analyse deep features with
Fourier transform and show that Depth Adaptive Blurring is more effective, as
opposed to monotonic blurring. To this end, we outline how this can replace
existing down-sampling methods. Second, we introduce a novel activation
function – with a built-in low pass filter, to keep the problem from
reappearing. From experiments, we observe generalisation on other forms of
transformations and corruptions as well, e.g., rotation, scale, and noise. We
evaluate our method under three challenging settings: (1) a variety of image
translations; (2) adversarial attacks – both $\ell_{p}$ bounded and unbounded;
and (3) data corruptions and perturbations. In each setting, our method
achieves state-of-the-art results and improves clean accuracy on various
benchmark datasets.

###### Index Terms:

Convolutional Neural Network (CNN), robust CNN, anti-aliasing CNN, translation
invariant CNN, image noise, corruption, perturbation.

##  I Introduction

Convolutional Neural Networks (CNNs) have achieved state-of-the-art results in
a wide range of vision tasks, including large-scale image classification [1].
To understand the extent of CNN’s robustness, researchers have started looking
beyond performance on i.i.d. (independent and identically distributed)
datasets only. Recent studies reveal that CNNs are vulnerable to subtle
changes in input, e.g., simple one pixel shift [2, 3, 4], quasi-imperceptible
noise [5, 6, 7, 8, 9], blur [10, 11], and adversarial attacks [12, 13, 14].
This is concerning for real-world multi-media applications, where the i.i.d.
assumption does not always hold. For instance, an autonomous car should not
flip predictions for the same object between consecutive video frames due to
marginal spatial shift or image noise [15, 16]. Recently, aliasing has been
identified as one of the main reasons behind CNN’s lack of robustness –
especially against small image transformations, such as shift [2, 3].

Figure 1: Features from an original image (top), and its shifted variant
(bottom). The baseline (ResNet-101) – without any low-pass filtering, and
MaxBlurPool (MBP) [3] – with monotonic blurring, both experience signal
degeneration due to aliasing. Our method retains bulk of the expected signal.
Here, $28\times 28$ features are taken from the res3 block. Figure 2: (a)
Fourier energy at different network depths (lower frequencies are in the
centre). Heat maps are generated using feature maps, at different network
depths, belonging to the ImageNet validation set. For all three networks,
deeper feature maps have greater high frequency energy, suggesting the
importance of depth adaptive blurring. (b) Our proposed method ensures better
anti-aliasing by learning stronger blur filters for deeper layers, in this
instance, using a ResNet-101 backbone.

In CNNs, aliasing occurs when down-sampling (DS) operations do not satisfy the
Nyquist sampling rate [17, 2]. The theorem states that if the DS factor is not
at least double of the maximum signal frequency, the output will be corrupted.
In the context of CNNs, the problem aggravates with increasing network depth,
leading to loss of the feature structure, and in effect, loss of shift-
invariance. Blurring before sub-sampling – a well-known signal processing fix,
could be effective here [3], but naively using the same blur kernel across the
network, as used in MaxBlurPool (MBP [3]), or using separate ones for each
spatial location [18] does not yield satisfactory performance. In Figure 1, we
compare feature maps from three different networks, and find that the baseline
(ResNet-101), as well as MBP suffer from signal degeneration.

In this work, we propose a Depth Adaptive Blur-pool (DAB-pool) module and
recommend replacing common DS methods with it (details in Section III-A). DAB
is inspired by the Fourier analysis, where we find that the feature maps –
belonging to deeper CNN layers become increasingly higher frequency in nature,
as Figure 2(a) shows. This heuristic motivates us to let the network learn
progressively stronger blur kernels, and Figure 2(b) shows such a set of
learnt kernels.

Recently, Azulay et al. [2] have pointed out that aliasing persists even after
blurring, as Activation Functions (AFs) often allow high frequency noise to
alias back into the signal, and yet, they remain largely unaddressed. As a
solution, in this work, we propose a novel AF that not only serves as an
effective non-linearity unit, but also acts as a secondary low-pass filter.
Similar to DAB-pool, this can also be readily used in existing CNNs for
further performance gain.

To assess shift-invariance, we follow benchmark evaluation protocols in [2,
19, 3]. We also test our defence against recent (translation-based)
adversarial attacks [13], covering both black-box and white-box settings.
Although primarily pursuing shift-invariance, we assess performance
generalisation on a variety of corruptions and spatial perturbations on
ImageNet-C and ImageNet-P respectively [7]. In summary, we make the following
contributions:

  * •

We investigate ways of instilling anti-aliasing properties in CNNs. Since CNNs
host a variety of layers, simply inserting a low-pass filter does not suffice.
To this end, we propose to redefine the conventional DS and AFs as potential
anti-aliasing units in the form of DAB-pool and AA-ReLU respectively.

  * •

We analyse the Fourier or spectral properties of deep features and propose to
use DAB-pool for DS as the primary anti-aliasing unit.

  * •

We propose a novel AF with a built-in low-pass filter as a secondary anti-
aliasing unit.

  * •

We use the proposed DS method and AF in VGG16, ResNet-101, and DenseNet-121 by
replacing ReLU to demonstrate their effectiveness across architectures.

  * •

In addition to shift and shift-based perturbations, we also evaluate on
adversarial attacks, data corruptions, and image transformations. Our method
consistently outperforms contemporary methods across datasets and challenges.

Figure 3: Our proposed AF (AA-ReLU) gates between ReLU and C-ReLU by
exploiting the roll-off phase in $\sin x$.

##  II Related Work

Robust Learning.  Recently, deep CNN’s robustness is put to test in various
ways, ranging from adversarial attacks [20, 21, 22, 23, 24], to expose CNNs to
common corruptions [5, 7, 25, 8, 26, 27, 28, 29]. Latest works [2, 4, 30, 31]
have shown that even simple image transformations, e.g., shift, can cause
misclassification in CNNs. Although data augmentation is effective against
such vulnerabilities [11, 32, 33, 34, 35, 36, 37], we delve deeper into the
functional blocks of CNN. To be specific, we focus on replacing commonly used
DS methods and AFs, largely responsible for shift variance in CNNs [2].

Down-sampling. Spatial dimension reduction has been an integral part of CNNs
as it reduces the computational overhead and provides local translation
invariance [38]. Of late, max-pooling and strided convolution (S-conv) have
been predominantly used, owing to their superior task performance [39].
Nevertheless, these operations do not use a blur-prefix, and in effect,
violates the sampling theorem [2, 3]. Interestingly, average pooling – a well
known down-sampling method [40], is effectively a moving blur-filter, and
resists aliasing, but cannot match max-pool’s superiority in vision tasks
[39]. Upon realising the potential, Zhang et al. [3] extended the notion of
average pooling, and encouraged the use of stronger filters, e.g., a bilinear
or Laplacian [41]. Here, authors also provided ways to improve the task
performance, and yet, used a single filter (MBP) for all layers, which is sub-
optimal. In SABP [18], authors stressed the importance of an adaptive blur
scheme but did not explore the relation with network depth; instead, they used
a separate kernel for each local neighbourhood. While this method does
alleviate performance, using too many filters per feature map is
computationally exhaustive, and more importantly, leads to overfitting.
Hypothetically, using a dataset without any location bias could avoid such
overfitting; however, in practice, a bias-less dataset hardly exists, e.g.,
$90\%$ of dog images in ImageNet have the main subject in the centre [2], due
to a well-known phenomenon – photographer’s bias [2]. Such bias is prevalent
in almost all large-scale datasets, as such, learning separate and spatially
local filters do not generalise well (details in Section IV).

Kayhan et al. [19] argued that CNN filters exploit absolute spatial location
due to image boundary effect – a form of convolution irregularity at image
borders, and leads to shift variance. Here, an extension of the standard
padding scheme is proposed (termed Full-Convolution or F-Conv) to lessen the
boundary effects. Nevertheless, performance gain remains marginal.

As an alternative to blurring in the spatial domain, WaveCNet [42] does so in
the frequency domain via Discrete Wavelet Transform (DWT). Here, authors
replaced common DS layers with the Low-Low (LL) DWT output. Although effective
to some extent, a single DWT low-pass filter is used throughout the network,
limiting the overall improvement. Moreover, the back-and-forth wavelet
transforms add significant computational overhead. Similar to WaveCNet, Ryu et
al. [43] also operated in the Fourier domain but only replaced the last
average pooling layer with cropped Fourier coefficients (in ResNet). We argue
that shift invariance is progressively lost, and only fixing the deepest layer
is insufficient. To address this, in an upgraded variant ($DFT^{+}$) [43],
Fourier features from shallower layers are extracted in a separate sub-network
and fused with the backbone features. Later, SVM is used for final prediction.
Even so, $DFT^{+}$ marginally improves performance while incurring a lot of
additional parameters. In [44], authors resort to rectifying Batch
Normalisation Statistics (BNS) to improve robustness.

Activation Functions.  AFs inject non-linearity in otherwise linear CNNs, and
enable a deep classifier to draw highly non-linear decision boundaries, in
high dimensional input space. Rectified Linear Unit (ReLU) is the de-facto AF
in modern CNNs [45, 46, 47], and although studied for years [48, 49, 50], its
true significance was realized in the last decade, when Krizhevsky et al. [51,
52] successfully replaced Sigmoidal functions with it. Unlike its
predecessors, ReLU offers early sparsity – a much-desired property that
enables training very large and deep networks. Such sparsity gives a network
enough ‘room’ to draw accurate decision boundaries among distinct classes. On
the flip side though, it lets high frequency noise slip through – leaving the
network vulnerable to aliasing, even with blur routines in place [2]. Clipped-
ReLU (C-ReLU [51, 53]) – a bounded variant of ReLU, could be used instead, as
it caps the output to a constant threshold $\alpha$. However, C-ReLU leaves a
sudden and large saturated region beyond $\alpha$, where a derivative is
absent. Furthermore, finding an optimal $\alpha$ is difficult as the buffer
between the signal and noise is unknown. This highlights the importance of a
‘soft’ capping procedure – much like how signal rolls-off in a physical low-
pass filter [54]. Different to rectified units, Sigmoidal functions, e.g.,
hyperbolic tangent ($tanh$ [55]), squashes the input and keeps the output
within a finite range. In tanh, the roll-off effect can be regulated as well,
but ensuring early feature sparsity remains a problem [51]. To strike the
right balance between early sparsity and noise filtering, in this work, we
softly gate between ReLU and C-ReLU using a learnable $\alpha$, as Figure 3
shows.

##  III Proposed Method

In this section, we first define the problem set-up. Then we describe DAB-pool
and how to integrate it into modern CNNs, followed by the details of our anti-
aliasing AF.

Problem Formulation. Let a CNN classifier be $\mathcal{G}$ – a function that
takes $X$ as the input and maps it to an output class $\mathcal{O}_{c}$.
Considering $X$ has a spatial resolution of $H\times W\times C$, $\mathcal{G}$
can be expressed as $\mathcal{G}(X^{H\times W\times
C})\rightarrow\mathcal{O}_{c}$. For translation invariance to hold,
$\mathcal{G}$ should predict the same class label for $X$, and its translated
variant as shown in Equation 1.

$\displaystyle\mathcal{G}(X^{H\times W\times C})$
$\displaystyle=\mathcal{G}(X^{H\times W\times C}+\Delta{T})=\mathcal{O}_{c}$
(1)

where, $\Delta{T}$ represents a marginal input translation.

###  III-A Depth Adaptive Blur-pool (DAB-pool)

For effective anti-aliasing in CNNs, sufficient high frequencies should be
removed before sub-sampling. A signal, containing mostly high frequencies,
needs stronger blurring, as opposed to a low frequency signal. This is because
a weaker than needed filter might leave frequencies beyond the limit imposed
by the Nyquist theorem. On the other hand, using over-aggressive blurring
might remove part of the signal itself.

Our spectral analysis, according to Figure 2(a), reveals that the feature maps
in deeper layers have greater energy quotient for high frequencies. A similar
trend is observed in all three networks, which reinforces the consistency of
this observation. It also suggests that a ‘one size fits all’ approach, i.e.,
using the same blur kernel for all layers would be sub-optimal, and hence we
propose DAB-pool.

To put these in formal notation, let the feature map for an input $X$ at layer
$L$ be $x^{L}$, a Gaussian blur kernel at depth level $D$111Depth level
denotes the number of DS layers, e.g., a network with four DS layers has four
depth levels, i.e., $D\in\\{1,2,3,4\\}$. be $G^{\sigma}_{D}$, and the output
of the convolutional DAB be $Y_{DAB}$. Here, $\sigma$ is the standard
deviation of the kernel, and a higher $\sigma$ denotes stronger blurring. Also
let the $(L+1)^{th}$ layer be a DS layer and hence we aim to blur $x^{L}$. Now
we can define our convolutional DAB operation as Equation 2 as follows:

$Y_{DAB}(i,j)=\sum_{p=1}^{m-1}\sum_{q=1}^{n-1}x^{L}_{i+p,j+q}\cdot(G^{\sigma}_{D})_{p,q}$
(2)

where $Y_{DAB}(i,j)$ is the output at location $(i,j)$ and the Gaussian kernel
resolution is $m\times n$.

Rather than using pre-fixed filters [3], we let the network learn them by
casting $\sigma$ as a learnable parameter. To ensure progressively stronger
blurring with increasing depth $D$, we enforce the following learning
constraints:
$G^{\sigma}_{1}<G^{\sigma}_{2}<G^{\sigma}_{3}<G^{\sigma}_{4}\ldots<G^{\sigma}_{M}$,
where $M=max(D)$.

Replacing Down-sampling Layers. We replace max-pool, S-conv, and avg-pool as
follows:

$\displaystyle\text{Max-pool}_{\textit{k,s}}$
$\displaystyle\rightarrow\text{Subsample\textsubscript{{k,s}} $\circ$
DAB\textsubscript{{k,s}} $\circ$ DenseMax\textsubscript{{k,s}=1}}$ (3)
$\displaystyle=\text{ DAB-pool\textsubscript{{k,s}} $\circ$
DenseMax\textsubscript{{k,s}=1}}$

$\displaystyle\text{Conv}_{\textit{k,s}>1}$
$\displaystyle\rightarrow\text{Subsample\textsubscript{{k,s}}$\circ$
DAB\textsubscript{{k,s}} $\circ$ Conv\textsubscript{{k,s}=1}}$ (4)
$\displaystyle=\text{DAB-pool\textsubscript{{k,s}} $\circ$
Conv\textsubscript{{k,s}=1}}$

$\displaystyle\text{AvgPool}_{\textit{k,s}}$
$\displaystyle\rightarrow\text{Subsample\textsubscript{{k,s}} $\circ$
DAB\textsubscript{{k,s}}}$ (5) $\displaystyle=\text{DAB-
pool\textsubscript{{k,s}}}$

Here, $k$ and $s$ refer to kernel dimensions and stride respectively. DenseMax
is simply stride one max-pooling, found effective in improving task
performance [3]. Each sub-sampling operation, in Equations 3, 4, and 5, can be
easily combined with DAB as a single operation for efficiency, and hence,
jointly represented as DAB-pool in the equations.

Gradient Backpropagation.  In a CNN, partial derivatives of the final cost
function are propagated backwards and the learnable parameters are updated
accordingly. Here, we demonstrate backpropagation through our proposed DAB
layer.

As shown in Equation 2, $x^{L}$ is the input, $G^{\sigma}_{D}$ is the Gaussian
blur kernel, and $Y_{DAB}$ is the output. Assume $dy$ is the error that has
already backpropagated up to $Y_{DAB}$. From Equation 2, we can further derive
the following as Equation 6.

$\displaystyle\text{ }Y_{DAB}(i,j)=$
$\displaystyle\sum_{p=1}^{m-1}\sum_{q=1}^{n-1}x^{L}_{i+p,j+q}\cdot(G^{\sigma}_{D})_{p,q}$
(6) $\displaystyle\text{Where, }(G^{\sigma}_{D})_{p,q}=$ $\displaystyle\text{
}\frac{H_{g}\left(p,q\right)}{\sum_{p}\sum_{q}H_{g}}$ $\displaystyle\text{and
}H_{g}\left(p,q\right)=$ $\displaystyle\text{
}e^{\frac{-\left(p^{2}+q^{2}\right)}{2\sigma^{2}}}$

As shown later in Section IV, we use a fixed $3\times 3$ blur kernel for all
DAB layers and hence, $p$ and $q$ always have the same size, i.e., $m=3$ and
$n=3$. Standard deviation $\sigma$ is the only learnable parameter here that
gets updated during training.

During backpropagation, partial derivatives can be calculated by applying the
chain rule [56]. However, replicating the chain rule via convolution is a more
efficient way [57]. Partial derivatives of the weight matrix $G^{\sigma}_{D}$
is calculated by convolving the input $x^{L}$ with the error matrix $dy$ [57]
and represented as $dG^{\sigma}_{D}$. Equation 7 shows the calculation of
$dG^{\sigma}_{D}$. Similarly, error $dy$ is dispersed backwards by convolving
it with $G^{\sigma}_{D}$ [57] as shown in Equation 8.

$\displaystyle dG^{\sigma}_{D}(p,q)=$
$\displaystyle\sum_{p=1}^{m-1}\sum_{q=1}^{n-1}x^{L}_{i+p,j+q}\cdot(dy)_{i,j}$
(7)

$\displaystyle dx^{L}(i+p,j+q)=$
$\displaystyle\sum_{p=1}^{m-1}\sum_{q=1}^{n-1}(dy)_{i,j}\cdot(G^{\sigma}_{D})_{p,q}$
(8)

###  III-B Anti-Aliasing ReLU (AA-ReLU)

In common signal processing tasks, low pass filtering is theoretically
sufficient to avoid aliasing. For example, an analogue to digital sound
converter inside a voice recorder can guarantee alias-free output, as long as
an optimal low-pass filter is used. In contrast, blurring before sub-sampling
cannot provide such guarantee in CNNs due to non-linearities in the form of
AFs [2]. We argue that an AF can largely address this by acting as a secondary
AA unit that complements DAB-Pool by resisting noise. Such an AF should: allow
the signal to go through, have a low-pass filtering or function roll-off
mechanism, and function continuity described as below:

  1. I.

Early feature sparsity [51]: Small positive features are mostly true signal
[58] and hence should be allowed to propagate forward – similar to what C-ReLU
offers prior to the clipping point.

  2. II.

Smooth roll-off: This is required to complement DAB-pool by high frequency
noise suppression. Contrary to a hard-clipping used in C-ReLU, a smooth roll-
off is a better choice as the buffer between the signal and noise is unknown.
A hard early clipping might affect the original signal, as well as deny early
feature sparsity.

  3. III.

Function continuity: For any AF to be effective, it has to be continuous so
that there is no point in the function without a derivative.

Considering these, we propose an AF in Equation 9 and subsequently describe
how it satisfies all three properties.

$F(x)=\left\\{\begin{array}[]{ll}0,&x\in(-\infty,0]\\\ x,&x\in(0,\alpha)\\\
f_{1}^{x}=\alpha\sin{(\ln\frac{x}{\alpha})}+\alpha,&x\in\left[\alpha,\alpha\exp\left(\frac{\pi}{2}\right)\right]\\\
f_{2}^{x}=max(f_{1}^{x})=2\alpha,&x\in\left(\alpha\exp\left(\frac{\pi}{2}\right),+\infty\right)\end{array}\right.$
(9)

where $\alpha$ is a learnable parameter representing the roll-off start point,
as well as the amplitude, frequency, and phase shift parameter of our
sinusoidal roll-off function $f_{1}^{x}$.

To ensure early feature sparsity (I), the proposed AA-ReLU, as in Equation 9,
is identical to ReLU for $x\in[-\infty,\alpha)$, however, for $x\geq\alpha$,
the signal starts to smoothly roll-off (II). $f_{1}^{x}$ dictates the roll-off
severity and span, whereas $f_{2}^{x}=2\alpha$ denotes the capping value as
shown in Equation 9. For further analysis, let $f_{1}^{x}$ in Equation 9 be
represented as Equation 10:

$\displaystyle f_{1}^{x}$
$\displaystyle=\alpha\sin\left(\ln\frac{x}{\alpha}\right)+\alpha$ (10)
$\displaystyle=\alpha\sin(g\left(x\right))+\alpha,$ $\displaystyle\text{here,
}g(x)=\ln\frac{x}{\alpha}$

Unlike the linear growth in ReLU, we aim to roll-off after a certain $\alpha$.
To ensure that happens, $f_{1}^{x}$ starts by marginally suppressing the
output for inputs slightly greater than $\alpha$, i.e., $x>\alpha$. Harder
suppression takes effect for sufficiently large $x$, i.e., $x\gg\alpha$.
Contrary to the oscillating property of a typical $\sin$ function, $F(x)$, for
$x\gg\alpha$, should plateau, i.e., it should reach and stay at the function
maximum such that $F(x)=max(F(x))\\{\forall{x}|x\gg\alpha\\}$. A $\sin{x}$
function in the interval of $(0,\frac{\pi}{2}]$ has these desirable roll-off
properties and our proposed AF in Equation 9 exploits it.

It is worth noting that the input to $f_{1}^{x}$ can be large real numbers,
i.e., $x\in\mathbb{R}_{>0}$. Therefore, plugging $x$ directly in a $\sin{x}$
function would be impractical, as we do not want our sinusoidal output to
oscillate periodically. Rather, we want a substantially lower $\sin$ frequency
$f$, so that we have access to a wider and smoother roll-off region. This is
why we use a natural logarithmic function $g(x)$ in Equation 10, and in
effect, inside our proposed AF in Equation 9.

In addition to denoting the roll-off start point, $\alpha$ also increases the
amplitude of $f_{1}^{x}$ (since $\alpha\in\mathbb{Z}_{>0}$), which allows
early sparsity in $F(x)$, as Figure 3 shows. $\alpha$ also vertically shifts
$f_{1}^{x}$, and further modulates the frequency (by dividing $x$ inside the
$\log$ function) such that $F(x)$ remains continuous at each point.

To show AA-ReLU is continuous (III), we prove that $F(x)$ in Equation 9 is
continuous:

  * •

$F(x)=0$ and $F(x)=x$ are continuous since constant and identity functions are
continuous everywhere [59]. At the joining point, i.e., for $x=0$, both the
functions produce $0$ and hence they are jointly continuous as well.

  * •

$f_{1}^{x}$ is continuous in the given domain as it is a function of $\sin$
[59]. At the joining point with $F(x)=x$, i.e., for $x=\alpha$, both functions
produce the same output since
$f_{1}^{x}=\alpha\sin{(\ln\frac{\alpha}{\alpha})}+\alpha=\alpha$ and $F(x)=x$
is an identity function.

  * •

$f_{2}^{x}$ is another continuous function and at the joining point with
$f_{1}^{x}$, i.e., for $x=\alpha\exp\left(\frac{\pi}{2}\right)$,
$f_{1}^{x}=\alpha\sin{(\ln\frac{\alpha\exp\left(\frac{\pi}{2}\right)}{\alpha})}+\alpha=2\alpha=f_{2}^{x}$,
and hence, they are also jointly continuous.

A case study is presented in the following section for further understanding.

Case Study.  Let a 1D sample signal be $x=$ [-9, 3, 8.9, 9, 10, 43.3, 81].
Assuming $x$ as the input to Equation 9, and considering $\alpha=9$, $F(x)$ is
evaluated as follows:

  * •

$F(-9)$, $F(3)$, and $F(8.9)$ output $0,3,$ and $8.9$ respectively. By
definition, these outputs are identical to ReLU as $x<\alpha$.

  * •

$F(9)=f_{1}^{9}=9\cdot\sin{(\ln\frac{9}{9})}+9=9\cdot 0+9=9$ (note
$x==\alpha$). From here on, the roll of function smoothly takes over from
ReLU. Notice that using the same $\alpha$ as the amplitude, frequency, and
phase shift parameter ensures function continuity.

  * •

$F(10)=f_{1}^{10}=9.95$. $x$ has been slightly suppressed since $x>\alpha$.

  * •

$F(43.3)=f_{1}^{43.3}=9\cdot\sin{(\ln\frac{43.30}{9})}+9=9\cdot\sin{(\ln
e^{1.57})}+9=9\cdot\sin{\frac{\pi}{2}}+9=18$. $x$ has been heavily suppressed
since $x>>\alpha$.

By calculating the second order derivative of $f_{1}^{x}$, we can calculate
the function maximum. For this example, solving Equation 11 yields
$max(F(x))=18$.

$\displaystyle f_{1}^{x^{\prime\prime}}$
$\displaystyle=-\frac{\alpha\left(\sin\left(\ln\left(\frac{x}{\alpha}\right)\right)+\cos\left(\ln\left(\frac{x}{\alpha}\right)\right)\right)}{x^{2}}$
(11)

  * •

$F(81)=f_{2}^{81}=max(f_{1}^{x})=max(F(x))=18$. As shown in Equation 11,
$x=43.3$ induces F(x) to reach its maximum value, and therefore, according to
Equation 9, $F(81)=F(x>43.3)=18$.

Gradient.  Equation 12 shows the derivatives of $F(x)$.

$\frac{d}{dx}F(x)=\left\\{\begin{array}[]{ll}0,&x\in(-\infty,0],\\\
1,&x\in(0,\alpha),\\\
\frac{\alpha}{x}\cos{\left(\ln\frac{x}{\alpha}\right)},&x\in\left[\alpha,\alpha\exp\left(\frac{\pi}{2}\right)\right]\\\
0,&x\in\left(\alpha\exp\left(\frac{\pi}{2}\right),+\infty\right)\end{array}\right.$
(12)

Here, beyond $\alpha$, saturated points exist, but an optimally learned
$\alpha$ accounts for this issue.

|  |  |  | Shift Consistency (higher is better)
---|---|---|---|---
Backbone | Methods | Clean Accuracy | Diagonal Shift | Rescale Shift | Double Rescaling
|  | Abs | $\Delta$ | Abs | $\Delta$ | Abs | $\Delta$ | Abs | $\Delta$
| Baseline (VGG16) | 71.66 | - | 88.52 | - | 83.89 | - | 85.04 | -
| MBP(Bin-5) [3] (ICML ’19) | 72.29 | +0.63 | 90.11 | +1.59 | 83.99 | +0.10 | 86.98 | +1.94
| SABP [18] (BMVC ’20) | 72.95 | +1.29 | 90.36 | +1.84 | 84.20 | +0.31 | 87.10 | +2.06
VGG16 | BNS [44] (WACV ’21) | 71.75 | +0.09 | 90.36 | +1.84 | 84.06 | +0.17 | 85.24 | +0.20
| WaveCNet(ch-3,3) [42] (CVPR ’20) | 71.96 | +0.30 | 90.42 | +1.90 | 85.35 | +1.46 | 86.05 | +1.01
| F-Conv [19] (CVPR ’20) | 70.65 | -1.01 | 90.01 | +1.49 | 81.15 | -2.74 | 84.78 | +0.74
| DABP + AA-ReLU (ours) | 73.91 | +2.25 | 91.85 | +3.33 | 87.47 | +3.58 | 90.25 | +5.21
| Baseline (ResNet-101) | 77.37 | - | 89.95 | - | 80.20 | - | 83.54 | -
| MBP(Bin-5) ([3]) | 77.40 | +0.03 | 91.31 | +1.36 | 86.10 | +5.90 | 83.97 | +0.43
| SABP ([18]) | 79.32 | +1.95 | 92.19 | +2.24 | 85.94 | +5.74 | 83.81 | +0.27
ResNet-101 | BNS [44] | 77.25 | -0.12 | 90.05 | +0.10 | 84.11 | +3.91 | 82.89 | -0.65
| WaveCNet(ch-3,3) [42] | 78.21 | +0.84 | 90.89 | +0.94 | 85.95 | +5.75 | 83.15 | -0.39
| F-Conv ([19]) | 78.31 | +0.94 | 90.05 | +0.10 | 84.66 | +4.46 | 82.23 | -1.31
| DABP + AA-ReLU (ours) | 81.45 | +4.08 | 94.11 | +4.16 | 91.36 | +11.16 | 86.45 | +2.91
| Baseline (DenseNet-121) | 74.44 | - | 88.81 | - | 82.35 | - | 84.25 | -
| MBP(Bin-5) ([3]) | 75.03 | +0.59 | 90.39 | +1.58 | 84.21 | +1.86 | 85.74 | +1.49
| SABP ([18]) | 75.36 | +0.92 | 88.85 | +0.04 | 83.55 | +1.20 | 85.77 | +1.52
DenseNet-121 | BNS [44] | 74.65 | +0.21 | 88.96 | +0.15 | 82.46 | +0.11 | 85.09 | +0.16
| WaveCNet(ch-3,3) [42] | 74.75 | +0.31 | 89.02 | +0.21 | 83.50 | +1.15 | 85.16 | +0.91
| F-Conv ([19]) | 74.09 | -0.35 | 88.84 | +0.03 | 80.88 | -1.47 | 84.55 | +0.03
| DABP + AA-ReLU (ours) | 77.44 | +3.00 | 92.85 | +4.04 | 88.25 | +5.90 | 88.08 | +3.83

TABLE I: Top-1 clean accuracy ($\%$) and shift consistency ($\%$) on ImageNet.
Our method outperforms others under different shifts. Here, $\Delta$
represents the improvements over baseline and the best result is highlighted
in bold.

##  IV Experiments and Results

We first outline the training details and hyper-parameter settings. Later, for
evaluation, we test using: (1) three shift-based perturbations, (2) three
recent translation-based adversarial attacks [13, 14], and (3) a range of
corruptions and perturbations. In each set-ups, we refer to the following
works for comparison: MaxBlurPool (MBP) [3], Spatially Adaptive Blur Pooling
(SABP) [18], Wavelet Integrated CNN (WaveCNet) [42], Batch Normalisation
Statistics (BNS) [44], and Full-Convolution (F-Conv) [19].

###  IV-A Implementation and Training Details

We train and test with three baseline networks – VGG16 [45], ResNet-101 [46],
and DenseNet-121 [47], following the original network architectures presented
in respective papers. We use Stochastic Gradient Descent ($SGD$) with
$Momentum=0.9$, and train for 100 epochs, while using an initial learning rate
$L_{r}=0.1$. $L_{r}$ is dropped by a factor of 0.2 after 30, 60, and 90 epochs
and a batch-size of 128 is maintained. Zero-centre input normalisation is used
for all experiments, with an L2 Regularization factor of 0.0005.

As for our learnable parameter $\sigma$, i.e., the standard deviation of
Gaussian blur filters, we initialise based on their depth and set
$\sigma_{D}=\frac{D}{2}$. Take for example ResNet-101: it has five down-
sampling layers, and hence, has five depth levels, yielding the following
initialisations: $\sigma_{1}=0.5$, $\sigma_{2}=1$, $\sigma_{3}=1.5$,
$\sigma_{4}=2$, and $\sigma_{5}=2.5$. For all the depth levels, we use
$3\times 3$ blur kernels (see Ablation study in Section V).

To initialise the other learnable parameter $\alpha$, placed inside AA-ReLU,
we start off with 6 based on heuristic [51].

###  IV-B Shift Invariance on ImageNet

ImageNet has 1,000 classes with 1.2M training and 50k test images. We follow
three shift evaluation protocols from [2, 3]. A network $\mathcal{G}$’s
consistency (evaluation metric) is the measure of stable Top-1 predictions for
an input $X$, and a translated $X_{\Delta T}$, formally put in Equation 13.

$\textit{consistency}=\mathbb{E}_{X}\llbracket\\{\mathcal{G}\left(X\right)==\mathcal{G}\left(X_{\Delta
T}\right)\\}\rrbracket$ (13)

where $\llbracket\cdot\rrbracket$ denotes Iverson Bracket that outputs 1 if
the proposition inside is true and 0 otherwise. A summary of all three shift
perturbation protocols is provided below:
Diagonal Shift.  here, each $X$ from the validation set is diagonally shifted
by $N$ pixels, where $N\in[1,64]$, and $N$ is random.
Rescale Shift.  Here, each $X$ is first down-scaled to a $100\times 100$
image, and later, embedded in a $224\times 224$ canvas, filling rest of the
blank space with black pixels. Finally, the resultant image is diagonally
shifted by a single pixel to obtain the perturbed variant, i.e., $X_{\Delta
T}$.
Double Rescaling.  Here, $X$ is first down-scaled and embedded, similar to
Rescale Shift. However, rather than shifting the rescaled embedding itself, it
is spatially altered by $\pm 1$ pixel to obtain $X_{\Delta T}$ (e.g., from an
embedding of $100\times 100$ to $101\times 101$).

Top-1 Accuracy (higher is better)
---
|  | Adversarial Attack
Methods | Clean | First Order | Grid Search | Worst-of-10
Baseline (ResNet-101) | 77.37 | 74.78 | 61.50 | 73.39
MBP(Bin-5)([3]) | 77.40 | 74.90 | 63.29 | 73.85
SABP([18]) | 79.32 | 74.96 | 63.55 | 73.44
BNS([44]) | 77.89 | 72.70 | 61.65 | 73.45
WaveCNet(ch-3,3)([42]) | 78.05 | 75.05 | 62.30 | 73.66
F-Conv([19]) | 78.31 | 73.85 | 62.60 | 72.88
DABP+AA-ReLU | 81.45 | 76.28 | 65.05 | 74.70

TABLE II: Top-1 classification accuracy ($\%$) against different adversarial
attacks.

| ImageNet-C (Corruption Error CE, lower is better)
---|---
Methods | Noise | Blur | Weather | Digital |
| Gauss | Shot | Impulse | Speckle | Gauss | Defocus | Glass | Motion | Zoom | Snow | Frost | Fog | Bright | Contr | Elastic | Pixel | Jpeg | mCE
Baseline | 68.91 | 72.12 | 74.66 | 73.55 | 64.96 | 61.45 | 74.65 | 60.20 | 61.75 | 66.45 | 60.55 | 54.10 | 30.87 | 60.87 | 54.28 | 54.34 | 44.61 | 61.08
MBP(Bin-5) ([3]) | 64.45 | 66.05 | 68.98 | 71.56 | 62.34 | 61.06 | 70.30 | 61.25 | 60.35 | 66.01 | 58.21 | 50.24 | 29.91 | 59.87 | 55.36 | 48.75 | 39.54 | 58.48
SABP ([18]) | 65.32 | 65.87 | 69.50 | 71.10 | 63.80 | 61.11 | 68.32 | 60.05 | 60.14 | 64.98 | 58.33 | 49.87 | 29.87 | 59.70 | 53.88 | 46.68 | 38.22 | 58.04
BNS ([44]) | 67.11 | 69.89 | 70.98 | 74.52 | 67.23 | 64.90 | 73.85 | 67.21 | 62.92 | 71.66 | 63.24 | 54.87 | 33.87 | 60.45 | 57.32 | 52.82 | 43.75 | 62.15
WaveCNet(ch-3,3) ([42]) | 66.32 | 69.15 | 70.08 | 74.14 | 66.14 | 63.36 | 71.98 | 65.05 | 62.44 | 69.70 | 61.34 | 54.01 | 31.22 | 60.14 | 56.99 | 52.32 | 41.66 | 60.94
F-Conv ([19]) | 66.98 | 65.21 | 69.37 | 72.88 | 62.77 | 60.81 | 72.66 | 60.32 | 61.45 | 66.02 | 60.24 | 52.36 | 29.56 | 59.86 | 54.44 | 46.87 | 39.35 | 58.89
DABP+AA-ReLU | 61.45 | 62.19 | 65.32 | 68.95 | 59.87 | 60.77 | 66.25 | 60.01 | 59.98 | 64.21 | 58.12 | 47.87 | 29.41 | 59.35 | 53.21 | 44.24 | 36.97 | 56.36

TABLE III: Corruption Error rate ($\%$) on ImageNet-C [7] (Baseline:
ResNet-101). Results in each category represent the average across all five
severity levels in the dataset. The rightmost column (mCE) denotes the mean
Corruption Error.

Results.  Compared to the baseline (VGG16, ResNet-101 and DenseNet-121),
DABP+AA-ReLU improves shift-invariance across all three deep networks in Table
I despite substantial architectural difference. For instance, VGG16 has four
intermediate max-pool layers as opposed to three average pooling layers in
DenseNet-121. On the other hand, ResNet-101 and DenseNet-121 use residual
block and depth concatenation, respectively, whereas VGG16’s architecture is
much simpler with sequential input and output layers. Interestingly, our
method improves the clean Top-1 accuracy as well exceeding the baseline by
$+2.25\%$ in VGG16, by $+4.08\%$ in ResNet-101, and by $+3\%$ in DenseNet-121.
Compared to a fixed blur kernel in MBP [3] and a spatially adaptive blur
kernel in SABP [18], DABP+AA-ReLU shows better consistency in each of the
evaluation protocols in Table I. Unlike other methods, it addresses the role
of AFs as well, which complements DAB-pool in achieving strong shift-
invariance.

###  IV-C Robustness Against Adversarial Attacks

By definition, an image $X^{\prime}$ is an adversarial variant of some valid
image $X$, if both appear visually similar to a human, and yet, a network
misclassifies $X^{\prime}$. Although visual perception is subjective and
similarity between images is hard to quantify, the relevant literature
presumes that $X^{\prime}$ is adversarial if and only if
$\left\|X-X^{\prime}\right\|_{p}\leq\varepsilon$, where $p\in[1,\infty)$ and
$\varepsilon$ is small. Traditional first-order attacks, e.g., Fast Gradient
Sign Method (FGSM [20]), operate in pixel space and generate adversaries from
a bounded $\ell_{p}$ ball. As recently pointed out in [13], spatial
adversarial vulnerability, e.g., due to translation, do not obey the bounded
protocol. This is because a translated image $X_{\Delta T}$ can still appear
visually similar, and yet, have a substantially large norm
$\left\|X-X^{\prime}\right\|_{p}$, compared to the norms allowed in
$\ell_{p}$. Hence, finding adversaries within a confined $\ell_{p}$ space is
difficult with first-order methods. In this work, we adhere to the spatial
attacks outlined in [13] and summarised below:

First Order Method (FO).  FO is a white-box attack, meaning it requires access
to the target network architecture, and in effect, to its gradients. In
$\ell_{p}$ based adversaries, each input pixel is cast as a learnable
parameter in a bid to maximise loss for a particular class (to inflict
misclassification). However, here, FO has to perturb the spatial tuning of an
image $X$ to generate a translated adversary $X^{\prime}$. This is why input
translation is parameterised in FO, and the loss for the correct class is
maximised by iteratively updating these parameters. Following [13], the
maximum perturbation, i.e., translation, allowed in this attack is 25 pixels,
and in terms of implementation, it is 200 steps of projected gradient descent
with step size 0.24.

Grid Search Attack (GSA).  This is a black-box attack and requires only a few
queries to the target model. GSA attacks the network by exhaustively
translating $X$ in each spatial direction until $X$ becomes an adversary
$X^{\prime}$ (if such an $X^{\prime}$ exists in the search grid). Here,
maximum translation allowed in each direction is 5 pixels.

Worst-of-$k$.  Here, to attack a target, $X$ is translated randomly $k$ times
by keeping the translation parameters within the allowed limit, i.e., 25
pixels. An $X^{\prime}$, within this limit, inducing the worst classification
performance is the one chosen as the adversary. For example, in worst-of-10
attack, it performs 10 random translations per $X$ and feeds the corresponding
$X^{\prime}$s to find the most effective adversary. It is a black-box attack
as well.

Results.  As Table II shows, our method exhibits better resistance against all
three attacks in terms of Top-1 accuracy. Interestingly, DABP+AA-ReLU performs
best against FO – the most effective attack in pixel-level $\ell_{p}$ bounded
space, performing $1.5\%$ over baseline (from here on, we refer to ResNet-101
as the baseline unless mentioned otherwise). This can be attributed to
different loss landscapes in natural image transformations versus typical
$\ell_{p}$ settings [13]. In $\ell_{p}$ loss landscape, the maxima are
consistent and concentrate well in a locality, making it easier to find
adversaries by taking small iterative steps towards it [21, 12]. However, the
loss landscape of translation is non-convex and tend to have multiple
scattered local maxima [13], meaning there is no guarantee that taking small
steps would lead to translated adversaries, and hence, it is sub-optimal. GSA,
on the other hand, turns out to be more challenging, and the baseline accuracy
drops from $77.4\%$ to $61.5\%$. DABP+AA-ReLU performs better than MBP by
$1.76\%$, SABP by $1.50\%$, BNS by $3.40\%$, WaveCNet by $2.75\%$, and F-Conv
by $2.45\%$ under GSA. A similar improvement trend is observed against Worst-
of-10 attack as well.

Figure 4: Mean Corruption Error (%) on increasing corruption severity levels
in ImageNet-C (lower is better).

###  IV-D Corruption and Perturbation Robustness

In this section, we test our corruption and perturbation stability on
ImageNet-C and ImageNet-P respectively.
ImageNet-C.  ImageNet-C [7] contains images from the ImageNet validation set
with 17 different corruptions, broadly falling into the following four
categories: noise, blur, weather, and digital. Each of the 17 corruptions has
five severity levels and mean Corruption Error (mCE) is used as the evaluation
metric.

Results.  Table III shows that DABP+AA-ReLU achieves state-of-the-art
robustness against a variety of corruptions in ImageNet-C with an mCE $4.71\%$
lower than the Baseline, $2.12\%$ lower than MBP [3] with Binomial-5 filter,
$1.68\%$ than SABP [18], $5.79\%$ than BNS [44], $4.58\%$ than WaveCNet with
cohen-3,3 filter, and $2.53\%$ than F-Conv. Our method exhibits marked
robustness in the noise category consisting of Gaussian, shot, impulse, and
speckle noise, achieving an overall $7.83\%$ lower mCE than the baseline.
Compared to competing methods, our overall noise mCE is $3.28\%$ lower than
MBP [3] and $3.47\%$ lower than SABP [18]. This is an expected byproduct of
blurred DS as most of the high frequency noise gets neutralised in this step.
However, signal noise – much like aliasing, can reappear in the non-linear
AFs, which we believe is the case in the baseline and in [3, 18]. Our method
is better suited to counter this challenge as AA-ReLU itself acts as a
secondary low-pass filter. The baseline often performs well under level-1
corruption severity, but struggles beyond level 2 and 3 (see Figure 4). Our
method shows better consistency, even with increasing corruptions.

ImageNet-P.  ImageNet-P [7] introduces a range of perturbations in the
validation set to test performance stability. In this work, we evaluate on
four affine transformation-based perturbations involving translation,
rotation, scaling, and tilt. In ImageNet-P, each test instance is a short
video of about 30 frames with increasing perturbation severity. Flip
Probability FP is used as the stability evaluation metric. A flip event occurs
when two consecutive frames’ predictions mismatch. To put it formally, let us
denote $k$ perturbation sequences (each with $v$ number of frames) with
$S=\left\\{\left(x_{1}^{(i)},x_{2}^{(i)},\ldots,x_{v}^{(i)}\right)\right\\}_{i=1}^{k}$.
For a fixed $i$, i.e., perturbation type $k$, $x_{1}^{(i)}$ denotes the clean
image (no perturbation) and $x_{v}^{(i)}$ denotes a frame with maximum ($k$
type) perturbation. The Flip Probability (FP) of a deep classifier
$\mathcal{G}$ on perturbation sequence $S$ is:

$FP_{p}^{\mathcal{G}}=\frac{1}{k(l-1)}\sum_{i=1}^{k}\sum_{j=2}^{v}1\left(\mathcal{G}\left(x_{j}^{(i)}\right)\neq\mathcal{G}\left(x_{j-1}^{(i)}\right)\right)$
(14)

Results.  As Table IV shows, our method shows better stability ($0.87\%$ lower
mFP than the next best SABP), and interestingly, it generalises to
perturbations other than translation as well, e.g., in rotation perturbation,
our FP is $2.27\%$ lower than the baseline, $1.34\%$ than MBP, and $1.17\%$
than SABP; gain is observed for tilt and scale as well.

ImageNet-P Flip Probability FP (lower is better)
---
| Affine Transform Perturbation
Methods | Translate | Rotate | Tilt | Scale | mFP
Baseline | 4.15 | 6.39 | 3.96 | 10.85 | 6.34
MBP(Bin-5) ([3]) | 3.70 | 5.46 | 3.37 | 8.41 | 5.24
SABP ([18]) | 3.42 | 5.29 | 3.32 | 8.06 | 5.02
BNS ([44]) | 4.46 | 5.98 | 4.20 | 9.35 | 6.00
WaveCNet ([42]) | 3.58 | 5.87 | 3.68 | 8.89 | 5.51
F-Conv ([19]) | 4.15 | 6.74 | 4.25 | 9.01 | 6.04
DABP+AA-ReLU | 2.26 | 4.12 | 3.22 | 6.98 | 4.15

TABLE IV: Flip Probability rate ($\%$) on transformation-based perturbations
in ImageNet-P [7] (Baseline: ResNet-101). mFP denotes mean Flip Probability.

Network | Time/epoch (mins) | No. of epochs
---|---|---
ReLU+(S-conv, Max-pool) (baseline) | 16.3 | $[105,115]$
C-ReLU+(S-conv, Max-pool) | 18.2 | $[85,95]$
ReLU+SABP [18] | 24.9 | $[100,110]$
AA-ReLU+DAB-pool (Ours) | 22.6 | $[90,100]$

TABLE V: Comparison of training time (on ImageNet). Average time required to
complete an epoch is presented in minutes and the corresponding number of
epochs for convergence is shown within a range. ResNet-101 is used as the
baseline in all the networks.

###  IV-E Training Time

By definition, AA-ReLU executes more conditional statements than ReLU during
training. Similarly, our proposed DAB-pool layer has an additional blurring
operation compared to Strided Convolution (S-conv) or Max-pool with a single
operational stage. Therefore, our CNN requires more training time compared to
the baseline, i.e., vanilla ResNet-101 with ReLU + (S-conv and Max-pool). As
shown in Table V, this baseline is the fastest with 16.3 mins/epoch.
ResNet-101 with C-ReLU is slightly slower with 18.2 mins/epoch. Our network
takes 22.6 mins/epoch which is 2.3 minutes lesser than the SABP network. It
should be noted that our method requires $[90,100]$ training epochs to
converge, whereas SABP and the baseline require more epochs to converge. We
argue that the unbounded nature of ReLU results in additional training
iterations for convergence. In contrast, AA-ReLU has a bounded output and
hence, the search space for the global minima is much smaller. This means our
network requires fewer iterations of weight updating to reach this minima
resulting in a reduced convergence time. Similar to AA-ReLU, C-ReLU also has
faster convergence due to its bounded nature.

All the training hyper-parameters are same as discussed earlier in Section
IV-A.

##  V Ablation Study

To justify our design choices, we conduct an ablation study on ImageNet using
ResNet-101 backbone. All the other settings remain identical to that of
Section IV.

As Table VI shows, a $3\times 3$ Gaussian blur filter performs better than
$5\times 5$ and $7\times 7$ filters. We hypothesise that too big of a blur
kernel spreads the features across an unnecessarily large region. This is
particularly concerning for deeper layers where filters have large receptive
fields, e.g., in ResNet-101, a $224\times 224$ input reduces down to $14\times
14$ feature maps in the last DS layer. Using a $7\times 7$ filter which is
half as big as the feature map dimension turns out to be sub-optimal.

We also test the compatibility of different AFs with DAB-pool and max-pool.
Although C-ReLU performs better than ReLU under all three shifts, it lags
behind in clean accuracy. We argue that because of C-ReLU’s bounded nature, it
has some anti-aliasing property, but a large saturated region abstains it from
achieving higher clean accuracy. Overall, DABP3*3+AA-ReLU is the best
configuration achieving $2.41\%$ and $2.42\%$ higher average accuracy than
DABP3*3+C-ReLU and DAB3*3+ReLU respectively.

Ablation Study (ImageNet)
---
|  | Shift Consistency (higher is better)
| Clean | Diagonal | Rescale | Double
Methods | Accuracy | shift | shift | Rescaling
DABP3*3+ReLU | 79.95 | 91.19 | 88.41 | 84.15
DABP5*5+ReLU | 78.85 | 88.54 | 87.37 | 83.15
DABP7*7+ReLU | 77.03 | 85.21 | 83.87 | 81.29
DABP3*3+C-ReLU | 79.03 | 91.58 | 88.80 | 84.33
DABP5*5+C-ReLU | 78.25 | 88.75 | 87.54 | 83.36
DABP7*7+C-ReLU | 74.87 | 87.25 | 86.66 | 81.89
DABP3*3+AA-ReLU | 81.45 | 94.11 | 91.36 | 86.45
DABP5*5+AA-ReLU | 78.94 | 90.66 | 88.96 | 84.90
DABP7*7+AA-ReLU | 76.15 | 88.96 | 87.32 | 83.08
Max-pool+ReLU | 76.37 | 88.95 | 79.20 | 82.54
Max-pool+C-ReLU | 75.01 | 88.98 | 79.89 | 83.04
Max-pool+AA-ReLU | 76.90 | 89.06 | 89.23 | 83.85

TABLE VI: Ablation study under varying architectural settings.

##  VI Conclusion

To avoid aliasing and make CNNs invariant to small input changes, we have
shown that benefits can be harnessed if depth is considered for blurred down-
sampling. To this end, we have introduced DAB-pool, and outlined ways to
replace common down-sampling operations with it. To ensure the problem does
not reappear, we have proposed a new activation function AA-ReLU, which also
serves as a low-pass filter. We have evaluated against a broad array of shift,
adversarial attack, corruption, and perturbation, and demonstrated the
efficacy of our method under challenging conditions. In doing so, we have also
integrated our modules in three different networks and shown that the benefits
generalise well despite architectural differences. Since classification
network backbones are an integral part of many more vision tasks such as
object detection and semantic segmentation, we believe our proposed methods
are easily transferable to these tasks as well.

## References

  * [1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2009, pp. 248–255.
  * [2] A. Azulay and Y. Weiss, “Why do deep convolutional networks generalize so poorly to small image transformations?” _The Journal of Machine Learning Research (JMLR)_ , 2018.
  * [3] R. Zhang, “Making convolutional networks shift-invariant again,” _ICML_ , 2019\.
  * [4] A. Bietti and J. Mairal, “Group invariance, stability to deformations, and complexity of deep convolutional representations,” _The Journal of Machine Learning Research (JMLR)_ , vol. 20, no. 1, pp. 876–924, 2019.
  * [5] S. F. Dodge and L. J. Karam, “Quality robust mixtures of deep neural networks,” _IEEE Transactions on Image Processing_ , vol. 27, no. 11, pp. 5553–5562, 2018.
  * [6] S. Dodge and L. Karam, “A study and comparison of human and deep learning recognition performance under visual distortions,” in _26th International Conference on Computer Communication and Networks (ICCCN)_. IEEE, 2017, Conference Proceedings, pp. 1–7.
  * [7] D. Hendrycks and T. G. Dietterich, “Benchmarking neural network robustness to common corruptions and perturbations,” _ICLR_ , 2019.
  * [8] M. T. Hossain, S. W. Teng, D. Zhang, S. Lim, and G. Lu, “Distortion robust image classification using deep convolutional neural network with discrete cosine transform,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 659–663.
  * [9] Y. Zhou, S. Song, and N.-M. Cheung, “On classification of distorted images with deep convolutional neural networks,” in _Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on_. IEEE, 2017, pp. 1213–1217.
  * [10] S. Dodge and L. Karam, “Understanding how image quality affects deep neural networks,” in _Eighth International Conference on Quality of Multimedia Experience (QoMEX)_. IEEE, 2016, Conference Proceedings, pp. 1–6.
  * [11] D. Hendrycks, N. Mu, E. D. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, “Augmix: A simple data processing method to improve robustness and uncertainty,” _International Conference on Learning Representations (ICLR)_ , 2020.
  * [12] J. Mohapatra, T.-W. Weng, P.-Y. Chen, S. Liu, and L. Daniel, “Towards verifying robustness of neural networks against a family of semantic perturbations,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, Conference Proceedings, pp. 244–252.
  * [13] L. Engstrom, B. Tran, D. Tsipras, L. Schmidt, and A. Madry, “Exploring the landscape of spatial robustness,” in _International Conference on Machine Learning (ICML)_ , 2019, pp. 1802–1811.
  * [14] L. Engstrom, “A rotation and a translation suffice: Fooling cnns with simple transformations,” _arXiv preprint arXiv:1712.02779_ , 2017.
  * [15] G. Sundaramoorthi and T. E. Wang, “Translation insensitive cnns,” _arXiv preprint arXiv:1911.11238_ , 2019.
  * [16] M. Manfredi and Y. Wang, “Shift equivariance in object detection,” _arXiv preprint arXiv:2008.05787_ , 2020.
  * [17] H. Nyquist, “Certain topics in telegraph transmission theory,” _Transactions of the American Institute of Electrical Engineers_ , vol. 47, no. 2, pp. 617–644, 1928.
  * [18] X. Zou, F. Xiao, Z. Yu, and Y. J. Lee, “Delving deeper into anti-aliasing in convnets,” in _BMVC_ , 2020.
  * [19] O. S. Kayhan and J. C. v. Gemert, “On translation invariance in cnns: Convolutional layers can exploit absolute spatial location,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, Conference Proceedings, pp. 14 274–14 285.
  * [20] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _ICLR_ , 2015.
  * [21] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _ICLR_ , 2018.
  * [22] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, Conference Proceedings, pp. 427–436.
  * [23] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _ACM on Asia Conference on Computer and Communications Security (ACCS)_. ACM, 2017, Conference Proceedings, pp. 506–519.
  * [24] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” _arXiv preprint_ , 2017.
  * [25] D. Hendrycks, M. Mazeika, S. Kadavath, and D. Song, “Using self-supervised learning can improve model robustness and uncertainty,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2019, Conference Proceedings, pp. 15 637–15 648.
  * [26] D. Yin, R. G. Lopes, J. Shlens, E. D. Cubuk, and J. Gilmer, “A fourier perspective on model robustness in computer vision,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2019, Conference Proceedings.
  * [27] R. Geirhos, C. R. Temme, J. Rauber, H. H. Schütt, M. Bethge, and F. A. Wichmann, “Generalisation in humans and deep neural networks,” in _Advances in Neural Information Processing Systems (NIPS)_ , 2018, pp. 7538–7550.
  * [28] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt, “Measuring robustness to natural distribution shifts in image classification,” _arXiv preprint arXiv:2007.00644_ , 2020.
  * [29] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge, “Improving robustness against common corruptions by covariate shift adaptation,” _arXiv preprint arXiv:2006.16971_ , 2020.
  * [30] A. Fawzi and P. Frossard, “Manitest: Are classifiers really invariant?” _BMVC_ , 2015.
  * [31] C. Kanbak, S.-M. Moosavi-Dezfooli, and P. Frossard, “Geometric robustness of deep networks: analysis and improvement,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 4441–4449.
  * [32] E. Kauderer-Abrams, “Quantifying translation-invariance in convolutional neural networks,” _arXiv preprint arXiv:1801.01450_ , 2017.
  * [33] E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, “Autoaugment: Learning augmentation strategies from data,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 113–123.
  * [34] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, Journal Article.
  * [35] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint:1708.04552_ , 2017.
  * [36] R. G. Lopes, D. Yin, B. Poole, J. Gilmer, and E. D. Cubuk, “Improving robustness without sacrificing accuracy with patch gaussian augmentation,” _arXiv preprint arXiv:1906.02611_ , 2019.
  * [37] D. Laptev, N. Savinov, J. M. Buhmann, and M. Pollefeys, “Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, Conference Proceedings, pp. 289–297.
  * [38] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, _Deep learning_. MIT press Cambridge, 2016, vol. 1.
  * [39] D. Scherer, A. Müller, and S. Behnke, “Evaluation of pooling operations in convolutional architectures for object recognition,” in _International Conference on Artificial Neural Networks (ICANN)_. Springer, 2010, Conference Proceedings, pp. 92–101.
  * [40] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, “Handwritten digit recognition with a back-propagation network,” in _Advances in Neural Information Processing Systems (NIPS)_ , 1990, pp. 396–404.
  * [41] P. Burt and E. Adelson, “The laplacian pyramid as a compact image code,” _IEEE Transactions on communications_ , vol. 31, no. 4, pp. 532–540, 1983\.
  * [42] Q. Li, L. Shen, S. Guo, and Z. Lai, “Wavelet integrated cnns for noise-robust image classification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, Conference Proceedings, pp. 7245–7254.
  * [43] J. Ryu, M.-H. Yang, and J. Lim, “Dft-based transformation invariant pooling layer for visual classification,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, Conference Proceedings, pp. 84–99.
  * [44] P. Benz, C. Zhang, A. Karjauv, and I. S. Kweon, “Revisiting batch normalization for improving corruption robustness,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021.
  * [45] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015, pp. 1–9.
  * [46] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016, Conference Proceedings, pp. 770–778.
  * [47] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 4700–4708.
  * [48] P. Dayan and L. F. Abbott, _Theoretical neuroscience: computational and mathematical modeling of neural systems_. Computational Neuroscience Series, 2001.
  * [49] G. E. Dahl, T. N. Sainath, and G. E. Hinton, “Improving deep neural networks for lvcsr using rectified linear units and dropout,” in _2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2013, pp. 8609–8613.
  * [50] V. Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in _ICML_ , 2010.
  * [51] A. Krizhevsky and G. Hinton, “Convolutional deep belief networks on cifar-10,” _Technical Report, University of Toronto_ , 2010.
  * [52] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems (NIPS)_ , 2012, Conference Proceedings, pp. 1097–1105.
  * [53] A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates _et al._ , “Deep speech: Scaling up end-to-end speech recognition,” _arXiv preprint arXiv:1412.5567_ , 2014.
  * [54] K. Turkowski, “Filters for common resampling tasks,” in _Graphics gems_. Academic Press Professional, Inc., 1990, pp. 147–165.
  * [55] K. Jarrett, K. Kavukcuoglu, M. Ranzato, and Y. LeCun, “What is the best multi-stage architecture for object recognition?” in _2009 IEEE 12th international conference on computer vision (ICCV)_. IEEE, 2009, pp. 2146–2153.
  * [56] P. J. Werbos, “Backpropagation through time: what it does and how to do it,” _Proceedings of the IEEE_ , vol. 78, no. 10, pp. 1550–1560, 1990.
  * [57] V. Dumoulin and F. Visin, “A guide to convolution arithmetic for deep learning,” _arXiv preprint arXiv:1603.07285_ , 2016.
  * [58] M. T. Hossain, S. W. Teng, F. Sohel, and G. Lu, “Robust image classification using a low-pass activation function and dct augmentation,” _arXiv preprint arXiv:2007.09453_ , 2020.
  * [59] H. J. Keisler, _Elementary calculus: An infinitesimal approach_. Courier Corporation, 2013.

|  Md Tahmid Hossain is currently pursuing his Ph.D. degree at the School of
Science, Engineering and Information Technology, Federation University,
Australia. He received a Bachelor of Science (B.Sc.) degree in Computer
Science and Engineering (CSE) from the Islamic University of Technology (IUT),
Bangladesh, in 2015. He has held the position of lecturer in the department of
CSE at the same university he graduated from. His area of research interests
includes deep learning-based image classification, object detection,
generative adversarial networks and computer vision in general.
---|---
|  Shyh Wei Teng is an Associate Professor and Deputy Dean at School of
Engineering, Information Technology and Physical Sciences, Federation
University Australia. He previously held positions at Monash University after
he obtained his PhD at that university in 2004. His research interests include
Image/video processing; Machine learning; and Multimedia analytics. He has so
far published over 80 refereed research papers. He received various
competitive research funding, including three Federal Government grants on
image retrieval and analytics—one from the Australian Research Council (ARC).
He supervised 9 PhD and 1 MPhil students to completion.
---|---
|  Ferdous Sohel (M’08-SM’13) received a PhD degree from Monash University,
Australia. He is currently an Associate Professor in Information Technology at
Murdoch University, Australia. Prior to joining Murdoch, he was a Research
Fellow at the School of Computer Science and Software Engineering, The
University of Western Australia from January 2008 to 2015. His research
interests include computer vision, machine learning, pattern recognition, and
digital agriculture. He is a recipient of the best PhD thesis medal from
Monash University. He has received several best paper awards. He was a co-
presenter of a tutorial at CVPR2015. He is an Associate Editor of IEEE
Transactions on Multimedia (2020-2021) and IEEE Signal Processing Letters
(2020-2021). He served as a Technical Program Chair of DICTA2021, DICTA2019;
as an Organising Secretary of APCC2017; and as a Tutorial Chair of PSIVT2019.
He is a member of the Australian Computer Society and a senior member of IEEE.
---|---
|  Guojun Lu (Senior Member of IEEE) is a Professor at School of Engineering,
Information Technology and Physical Sciences, Federation University Australia.
He has many years’ research experience in artificial intelligence, multimedia
signal processing, and retrieval, and has worked on an ARC DP project and
supervising an ARC DECRA project. He has successfully supervised over 20 PhD
students. He has held positions at Loughborough University, National
University of Singapore, Deakin University and Monash University, after he
obtained his PhD in 1990 from Loughborough University and BEng from Nanjing
Institute of Technology (now South East University, China). He has published
over 230 refereed journal and conference papers and wrote two books(
Communication and Computing for Distributed Multimedia Systems (Artech House
1996), and Multimedia Database Management Systems (Artech House 1999)).
---|---