# Beware of Aliases – Signal Preservation is Crucial for Robust Image
Restoration

Shashank Agnihotri0000-0001-6097-8551 Corresponding Author. Email:
<EMAIL_ADDRESS>Julia Grabinski0000-0002-8371-1734  Janis
Keuper0000-0002-1327-1243  Margret Keuper0000-0002-8437-7993  University of
Mannheim  Fraunhofer ITWM, Kaiserslautern  Institute for Machine Learning and
Analytics (IMLA), Offenburg University  Max-Planck-Institute for Informatics,
Saarland Informatics Campus, Saarbrücken

###### Abstract

Image restoration networks are usually comprised of an encoder and a decoder,
responsible for aggregating image content from noisy, distorted data and to
restore clean, undistorted images, respectively. Data aggregation as well as
high-resolution image generation both usually come at the risk of involving
aliases, i.e. standard architectures put their ability to reconstruct the
model input in jeopardy to reach high PSNR values on validation data. The
price to be paid is low model robustness. In this work, we show that simply
providing alias-free paths in state-of-the-art reconstruction transformers
supports improved model robustness at low costs on the restoration
performance. We do so by proposing _BOA-Restormer_ , a transformer-based image
restoration model that executes downsampling and upsampling operations partly
in the frequency domain to ensure alias-free paths along the entire model
while potentially preserving all relevant high-frequency information.

2259

##  1 Introduction

Computer vision, an area with tremendous advances in recent year due to
improved machine learning models, is essentially a processing of 2D or 3D
signals. However, in the course of modifying machine learning methods for
vision tasks, some essential signal processing fundamentals have been
overlooked, potentially risking unexpected model behavior. Some recent works
like [18, 51, 17, 42] have explored this problem for the task of image
classification. However, the remnant spectral artifacts occurring in the
feature maps due to inadequate processing of signals are equally crucial for
pixel-wise prediction tasks like semantic segmentation, image restoration, and
others which are often involved in real-world applications that might affect
human safety. Therefore, reducing these spectral artifacts is of paramount
importance for a model’s reliability under noise and for model’s robustness
[22]. These spectral artifacts might emerge as grid artifacts or checkerboard
artifacts [34], ringing artifacts [31], or lattice artifacts [24]. Often these
are visible to the human eye. However in some independently and identically
distributed (i.i.d.) scenarios, the method might have learned shortcuts [13]
around it, to project a sample signal from the input distribution to the
target distribution. In such scenarios, these spectral artifacts, while still
existing in the learned feature representations, might not appear in the
restored images and provide a wrong sense of safety. Thus, slightly perturbing
the input distribution such that the perturbed and original input samples
still look similar to the human eye, might help in exposing these underlying
inadequately learned feature representations. Then, these would again be
visible to the human eye when performing pixel-wise prediction tasks. This can
be achieved by performing white-box adversarial attacks on the input images,
as they optimize the perturbations by utilizing the availability of a deep
neural networks (DNNs) learned weights. Therefore, we use white-box
adversarial attacks to accentuate the spectral artifacts in the restored
images and expose the vulnerabilities of the DNN’s learned feature
representations [1]. This can be seen in Fig. 1, where FSNet [9], Restormer
[46] and FLC Pooling [18] have visible artifacts under a PGD [27] attack.

FSNet [9] | Restormer [46]
---|---
|
PSNR: 16.54 | PSNR: 9.04
FLC Pooling [17] | BOA-Restormer (proposed)
|
PSNR: 16.00 | PSNR: 18.91
Figure 1: Comparing images deblurred by image restoration models after 10
iterations of PGD attack. We observe strong spectral artifacts in the restored
images are fixed by our proposed “FrequencyPreservedPooling” (downsampling)
and “FreqAvgUp” (upsampling) operations. The PSNR values reported are averaged
over the entire test set.

Some proposed methods for vision tasks, attempt to reduce spectral artifacts
using low-pass filters in the Fourier domain when downsampling feature maps
[17, 9, 19], motivated by the fact that convolution kernels widely used for
downsampling in DNNs sample inefficiently and introduce artifacts in the
learned feature representations[17]. However, depriving the DNN of the high-
frequency information might have detrimental effects on tasks such as edge
detection, image restoration, shape analysis, and others. One such task is
image deblurring, where the blurring has already deprived the DNN of some
essential high-frequency information like edges, and shapes. The restoration
DNNs is expected to learn them.

Thus, in this work, we study both the downsampling and upsampling operations
performed in a DNN for the image restoration task of image deblurring. We
propose novel and theoretically motivated sampling methods, “FrequencyPooling”
for downsampling and “FreqAvgUp” for upsampling that sample information
efficiently to ensure that the artifacts in the learned representations are
significantly reduced, as shown in Fig. 1.

This work makes the following contributions:

  * •

A novel downsampling method “FrequencyPreservedPooling”, that uses the Fourier
domain to downsample feature maps _without_ compromising the high-frequency
features and simultaneously making the network focus on low-frequency
features. This increases the stability of the learned features.

  * •

A novel upsampling method “FreqAvgUp” which uses the frequency domain, to
upsample feature maps and is symmetric to “FrequencyPreservedPooling”. We
refer to both modules used together as _Beware of Aliases (BOA)_ -Pooling.

  * •

The symmetry is important to allow for appropriate communication between model
encoder and decoder via skip connections. We demonstrate that symmetry in the
encoder and decoder helps in learning more robust representations.

  * •

We show that focusing on low-frequency information provides samples with
stable projections from the input distribution to the target distribution,
leading to significantly reduced spectral artifacts in the restored images.

##  2 Related Work

#### Adversarial Robustness

Adversarial robustness is a model’s robustness against intentionally crafted
image perturbations that aim to fool the network. These adversarial attacks
have proven to be an important tool for judging if the quality of the
representations learned by a network are reasonable[3, 38, 40, 1]. While the
first adversarial attacks focused on classification networks [5, 16, 27] these
kinds of attacks are now broadly used on different tasks like image denoising,
restoration, or semantic segmentation, to evaluate the network’s robustness
[3, 21, 39]. In our work, we show that the model’s robustness can be enhanced
by providing an aliasing-free path for the downsampling and upsampling in
image restoration models.

#### Aliasing Artifacts

Recent work analyzing the properties of robust models demonstrates that one
important aspect of a model’s robustness is the inherent robustness against
aliasing artifacts [18, 28, 48, 51]. Thereby, [48, 51] enhance a model’s
shift-invariance by blurring before downsampling to reduce aliases, [28]
include wavelet denoising against aliasing and for more robustness against
common corruptions, [23] introduce an anti-aliasing activation function in
combination with blurring before downsampling and [18] show that robust models
inherently exhibit less aliasing after downsampling. While these works
primarily focus on the encoding of images and the introduced artifacts during
downsampling they are equally important for upsampling methods [2]. Also in
the field of image generation reducing aliasing improves the quality of the
generated images [12, 25, 26].

#### Sampling Methods

As previously discussed, aliasing artifacts arise from incorrect sampling.
Thus, most work focused on the improvement of downsampling by special blurring
techniques [23, 26, 28, 48, 51]. Yet, these approaches only reduce the amount
of aliasing, but do not fully eliminate it. In contrast, [17, 19] propose a
completely aliasing-free downsampling. However, this approach only includes
the downsampling while for pixel-wise prediction tasks like image restoration
upsampling is equally important. Further, all these approaches remove aliases
by removing high-frequencies, the exact information that one would want to
restore for beblurring. Thus, we propose a pair of down- and upsampling
operations which provides the network with a completely aliasing-free path
while being able to retain high-frequency information.

Upsampling is typically done via Transpose Convolution [4, 15, 29, 33, 35, 37]
or Interpolation [6, 49, 50] which both suffer from spectral artifacts
introduced due to incorrect sampling [2]. In Transpose Convolutions, the
kernels can overlap leading to uneven contributions of the different pixels
which lead to grid-like artifacts [34]. Linear interpolation can lead to
aliasing artifacts which are caused by the limited context during
interpolation [2].

Pixel Shuffle and Unshuffle [41] increase the spatial dimensions of the
feature maps by combining $r^{2}$ channels in the Pixel Unshuffle setting and
decreasing the spatial resolution of the feature maps by $r$ in height and
width to increase the channel dimension by a factor of $r^{2}$ in the Pixel
Shuffle setting. Hence, the spatial resolution of the feature maps can easily
be increased (upsampling) by aggregating channels or decreased (downsampling)
by expanding the channel dimension while maintaining all information present
in the feature maps. Yet, Pixel Unshuffle performs downsampling with sampling
rate $r$ resulting in severe aliasing artifacts [17, 19] in each channel.

All these methods are prone to spectral artifacts like aliasing. In contrast,
we incorporate a way to learn high- and low-frequency information separately
and thus avoid aliases in one path of our down- and upsampling method, while
still incorporating the strength of Pixel Unshuffle and Pixel Shuffle in the
other path.

#### Frequency Learning in Image Restoration

Operating parts of a network in the frequency domain can enhance global
information [8, 20, 36] or improve downsampling in classification tasks [17,
28]. Yet, pixel-wise prediction tasks like image restoration also apply
upsampling. Thus prior work in this direction used discrete wavelet transforms
to replace the up- and downsampling in a Unet [45]. Furthermore, FSNet [10]
uses a multi-branch approach to select the most informative frequencies during
network training and thus enhance several image restoration tasks. While [30]
apply a ReLU function on the frequencies of the feature map to gain insightful
information on the blur pattern in image deblurring. In MRLPFNets [11] a low-
pass filter is learned to reduce high-frequency noise and the multi-scale
features from different stages of the network are fused via a wavelet-based
feature fusion.

All these approaches demonstrate that changing the perspective can enhance
image restoration tasks. However, all methods are driven by improvements over
clean tasks while our approach is inherently more robust due to the anti-
aliasing path we establish.

##  3 Method

FrequencyPreservedPooling Downsampling block | FreqAvgUp Upsampling block
---|---
(Encoder) | (Decoder)
|
Figure 2: A visual representation of the proposed sampling operations. Left,
we show the flow diagram for our proposed downsampling operation
“FrequencyPreservedPooling” used in the encoder of the image restoration
model’s architecture. Right, we show the flow diagram for our proposed
upsampling operation “FreqAvgUp” used in the decoder of the BOA architecture.
BOA-Restormer Architecture
---
Figure 3: BOA-network architecture using our up- and downsampling from Fig. 2
and skip connections between encoder and decoder.

We propose a method to preserve the high-frequency information while making
the network also focus on low-frequency information. To this effect, we split
the feature maps into their low-frequency and high-frequency components and
learn the proper mixture such as to optimize the stability of the feature maps
that the network learns, and to provide the network with an _alias-free_
information path.

Let $\eta_{\text{encode}}$ be the repeating blocks of the encoder and
$\eta_{\text{decode}}$ be the repeating blocks of the decoder, such that each
$\eta_{\text{encode}}$ is followed by a downsampling operation $\zeta_{down}$
and each decoder block is followed by an upsampling operation $\zeta_{up}$.

Most works [46, 7] use Pixel Unshuffle [41] for downsampling and Pixel Shuffle
[41] for upsampling. However, as discussed in 2, these operations can
introduce artifacts in the feature maps.

Thus in our work, we modify the $\zeta_{down}$ and $\zeta_{up}$ operations in
the encoder and decoder parts of the network, respectively, while keeping the
encoder and decoder building blocks the same.

Following we formulate these changes. Let feature $\mathrm{X}_{s}$ of shape
$[\mathrm{B},\mathrm{C},\mathrm{M},\mathrm{N}]$ consist of
${[\mathrm{x},\mathrm{y}]}_{\mathrm{M}\times\mathrm{N}}$ coordinates in the
spatial domain, where $\mathrm{B}$ is the batch size, $\mathrm{C}$ are the
number of feature channels, and the spatial resolution of the feature map is
$\mathrm{M}\times\mathrm{N}$.

###  3.1 Downsampling

A naïve implementation of FLC Pooling for downsampling would not work for
pixel-wise tasks such as image restoration due to the inherent nature of a
low-pass filter in the Fourier domain. This generates lattice-like artifacts
as shown by [24] due to Gibb’s phenomenon [14]. FLC Pooling provides good
results for image classification tasks where these spectral artifacts are
inconsequential. However, for image restoration tasks, since we need to
preserve the locality of our signal, these artifacts have to be reduced. An
ideal solution to this problem would be infinite padding, however, due to
computational reasons this is not viable. Thus, we choose a more feasible
padding size to reduce Gibb’s phenomenon, while keeping the size of spatial
dimensions odd for additional numerical stability [19].

So, first we pad $\mathrm{X}_{s}$ on all sides such that,

$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\bar{\mathrm{M}}\times\bar{\mathrm{N}}}}\xrightarrow[]{Padding}\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\frac{5}{2}\mathrm{M}+1\times\frac{5}{2}\mathrm{N}+1}}=:\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\bar{\mathrm{M}}\times\bar{\mathrm{N}}}}$
(1)

Then, we obtain and $\hat{\mathrm{X}}_{\omega}$ by performing a 2D Discrete
Fourier Transform (DFT) followed by a low pass filter on these feature maps,
i.e.,

$\hat{\mathrm{X}}_{\omega}\left[k,l\right]=\frac{1}{\bar{M}\times\bar{N}}\sum_{m=0}^{\bar{M}-1}\sum_{n=0}^{\bar{N}-1}X_{s}[x_{m},y_{n}]\cdot
e^{-j2\pi\left(\frac{k}{\bar{M}}m+\frac{l}{\bar{N}}n\right)}$ (2)

for $k,l$ in $0,\dots,\frac{\bar{N}-1}{2}$ and $0,\dots,\frac{\bar{M}-1}{2}$,
respectively. Now,
$\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}$
are the low frequency features in
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\bar{\mathrm{M}}\times\bar{\mathrm{N}}}}$.

Then, we zero pad $\hat{\mathrm{X}}_{\omega}$ to obtain
$\check{\mathrm{X}}_{\omega}$ which is of shape
$[\mathrm{B},\mathrm{C},\mathrm{\bar{M}},\mathrm{\bar{N}}]$

$\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}\xrightarrow[]{Padding}\check{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\mathrm{\bar{M}}\times\mathrm{\bar{N}}}}$
(3)

Please note, Eq. 2 and Eq. 3 together represent a low-pass filter such that
the threshold is the median of the frequencies present.

Then, we perform a full Inverse Discrete Fourier Transform on
$\hat{\mathrm{X}}_{\omega}$ to convert it to the spatial domain and obtain
$\hat{\mathrm{X}}_{s}$

$\hat{\mathrm{X}}_{s}[m,n]=\sum_{k=0}^{\frac{\bar{M}-1}{2}}\sum_{l=0}^{\frac{\bar{N}-1}{2}}\hat{\mathrm{X}}_{\omega}\left[k,l\right]\cdot
e^{j2\pi\left(\frac{k}{\bar{M}}m+\frac{l}{\bar{N}}n\right)},$ (4)

and, we perform a full Inverse DFT on $\check{\mathrm{X}}_{\omega}$ to convert
it to the spatial domain and obtain $\check{\mathrm{X}}_{s}$.

$\check{\mathrm{X}}_{s}[m,n]=\sum_{k=0}^{\bar{M}-1}\sum_{l=0}^{\bar{N}-1}\check{\mathrm{X}}_{\omega}\left[k,l\right]\cdot
e^{j2\pi\left(\frac{k}{\bar{M}}m+\frac{l}{\bar{N}}n\right)}$ (5)

To emulate the increase in the number of channels done by Pixel Unshuffle
operation, we then concatenate $\hat{\mathrm{X}}_{s}$ along the $\mathrm{C}$
dimension four times:

$\displaystyle\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{4C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}=$
(6)
$\displaystyle{\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}}\mathbin{\|}{\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}}\mathbin{\|}{\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}}\mathbin{\|}{\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\frac{\mathrm{\bar{M}}}{2}\times\frac{\mathrm{\bar{N}}}{2}}}}.$

Next, we remove the padding from Eq. 1 in $\check{\mathrm{X}}_{s}$ and we
obtain the features
$\mathrm{X^{\prime}}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{\bar{M}}\times\mathrm{\bar{N}}}}$
in the feature maps that correspond to the high frequencies in
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
using Eq. 7

$\mathrm{X^{\prime}}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}=\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}-\check{\mathrm{X}}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
(7)

We downsample
$\mathrm{X^{\prime}}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
to
$\mathrm{X^{\prime}}_{s_{\mathrm{B}\times\mathrm{4C}\times\mathrm{\frac{M}{2}}\times\mathrm{\frac{N}{2}}}}$
using Pixel Unshuffle. For PixelUnshuffle, we know the information being
removed, making the operation perfectly invertible. This property is crucial
later when we attempt to mirror the downsampling operation, to perform an
upsampling operation.

Now, we formulate the downsampling operations of the features maps, i.e.
$\zeta_{down}$ as shown in Eq. 8

$\zeta_{down}(\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}})=(1-\alpha)\cdot\check{\mathrm{X}}_{s_{\mathrm{B}\times\mathrm{4C}\times\mathrm{\frac{M}{2}}\times\mathrm{\frac{N}{2}}}}+\alpha\cdot\mathrm{X^{\prime}}_{s_{\mathrm{B}\times\mathrm{4C}\times\mathrm{\frac{M}{2}}\times\mathrm{\frac{N}{2}}}}$
(8)

where $\alpha$ is a learnable parameter. We learn a value for alpha such that
we mix the features with low and high frequencies optimally. We initialize
$\alpha$=0.3 to have a bias towards low-frequency information.

Thus, using $\zeta_{down}$ we obtain
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{4C}\times\mathrm{\frac{M}{2}}\times\mathrm{\frac{N}{2}}}}$
which is downsampled
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
as shown by Eq. 9.

$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{4C}\times\mathrm{\frac{M}{2}}\times\mathrm{\frac{N}{2}}}}=\zeta_{down}(\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}})$
(9)

We name this method FrequencyPreservedPooling (FP), because it allows the
downsampling operation to be fully signal preserving, depending on $\alpha$,
while providing an alias-free path. We represent these operations as a flow
diagram in Fig. 2 (left). For ablation, we also consider the scenario in which
we randomly drop the high frequencies with a probability of 30%, i.e. we use
only the low frequencies
$\hat{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{4C}\times\frac{\mathrm{M}}{2}\times\frac{\mathrm{N}}{2}}}$
during learning 30% of the time (batch-wise), in this scenario Eq. 8
effectively changes to

$\tilde{\zeta}_{down}(\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}})=\check{\mathrm{X}}_{s_{\mathrm{B}\times\mathrm{4C}\times\mathrm{\frac{M}{2}}\times\mathrm{\frac{N}{2}}}}.$
(10)

###  3.2 Upsampling

For upsampling the feature maps, we propose an approach symmetric to the
downsampling operation. Here the objective is to perform upsampling operation
$\zeta_{up}$ on
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
and obtain
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{\frac{C}{4}}\times\mathrm{2{M}}\times\mathrm{2{N}}}}$.

$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{\frac{C}{4}}\times\mathrm{2{M}}\times\mathrm{2{N}}}}=\zeta_{up}(\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}})$
(11)

Following we describe the operations that comprise $\zeta_{up}$.

First, we perform a DFT on the feature maps $\mathrm{X}_{s}$ that exist in the
spatial domain to obtain the feature maps in the frequency domain denoted by
$\mathrm{X}_{\omega}$ as shown by Eq. 12.

$\mathrm{X_{\omega}\left[k,l\right]}=\frac{1}{M\times
N}\sum_{m=0}^{M-1}\sum_{n=0}^{N-1}X_{s}[x_{m},y_{n}]\cdot
e^{-j2\pi\left(\frac{k}{M}m+\frac{l}{N}n\right)}$ (12)

Since in Eq. 6 we concatenated four feature maps at a time, now we calculate
$\bar{\mathrm{X}}_{\omega}$, in which replace a set of four feature maps from
$\mathrm{X}_{\omega}$ with average over those four feature maps along the
channel dimension. This is shown by Eq. 13,

$\displaystyle\bar{\mathrm{X}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}=$
(13)
$\displaystyle\frac{1}{4}\left(\sum_{c=0}^{c+4}\mathrm{X}_{\omega_{\mathrm{B}\times\mathrm{c}\times\mathrm{M}\times\mathrm{N}}}\mathbin{\|}\sum_{c=4}^{c+4}\mathrm{X}_{\omega_{\mathrm{B}\times\mathrm{c}\times\mathrm{M}\times\mathrm{N}}}\mathbin{\|}...\mathbin{\|}\sum_{c=C-4}^{C}\mathrm{X}_{\omega_{\mathrm{B}\times\mathrm{c}\times\mathrm{M}\times\mathrm{N}}}\right)$

Please note ${\mathrm{X}}_{\omega}$ contains both low-frequency and high-
frequency information, but since during downsampling, we focus on learning
low-frequency information, ${\mathrm{X}}_{\omega}$ contains more low-frequency
information and by extension same is true for $\bar{\mathrm{X}}_{\omega}$.

Next, we separate the frequencies higher than the mean to obtain
$\mathrm{X^{*}}_{\omega}$ for each four-tuple of channels

$\mathrm{X^{*}}_{\omega}=\mathrm{X}_{\omega}-\bar{\mathrm{X}}_{\omega},$ (14)

and perform IDFT on
$\mathrm{X^{*}}_{\omega_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
to convert the feature maps to
$\mathrm{X^{*}}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
in the spatial domain as shown by Eq. 15,

$\mathrm{X^{*}}_{s}[m,n]=\sum_{k=0}^{M-1}\sum_{l=0}^{N-1}\mathrm{X^{*}}_{\omega}\left[k,l\right]\cdot
e^{j2\pi\left(\frac{k}{M}m+\frac{l}{N}n\right)}.$ (15)

This operation is followed by a Pixel Shuffle operation on
$\mathrm{X^{*}}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
to upsample it to
$\mathrm{X^{*}}_{s_{\mathrm{B}\times\mathrm{\frac{C}{4}}\times\mathrm{2M}\times\mathrm{2N}}}$.

Next, we upsample
$\bar{\mathrm{X}}_{{\omega}_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$,
by first choosing the first of every four identical channels and obtain
$\bar{\mathrm{X}}_{{\omega}_{\mathrm{B}\times\mathrm{\frac{C}{4}}\times\mathrm{M}\times\mathrm{N}}}$.

In Pixel Shuffle and Unshuffle, the spatial shuffle is not given. Thus,
combining the 4 maps in the frequency domain is not trivial as they are
shuffled in space.

Next, we are adding with zero along the x-axis and y-axis of the feature maps
in the frequency domain to obtain
$\bar{\mathrm{X}}_{{\omega}_{\mathrm{B}\times\mathrm{\frac{C}{4}}\times\mathrm{2M}\times\mathrm{2N}}}$
to complete the upsampling.

This is followed by an IDFT operation on $\bar{\mathrm{X}}_{\omega}$ to obtain
$\bar{\mathrm{X}}_{s}$

$\displaystyle\bar{\mathrm{X}}_{s}[m,n]=\sum_{k=0}^{2M-1}\sum_{l=0}^{2N-1}\bar{\mathrm{X}}_{\omega}\left[k,l\right]\cdot
e^{j2\pi\left(\frac{k}{M}m+\frac{l}{N}n\right)}$

Lastly, we learn a parameter $\beta$ to mix the low-frequency information and
high-frequency information feature maps optimally. Thus, we express
$\zeta_{up}$ by Eq. 16,

$\zeta_{up}(\mathrm{X}_{s})=(1-\beta)\cdot\bar{\mathrm{X}}_{s}+\beta\cdot\mathrm{X^{*}}_{s}$
(16)

Here, similar to $\alpha$ in Sec. 3.1, we initialize $\beta$ with 0.3 to
induce a bias towards low-frequency information. We name this upsampling
method as FreqAvgUp. We represent these operations as a flow diagram in Fig. 2
(right) titled “Upsampling”.

Additionally for ablation, we explore another upsampling technique that we
name SplitUp. Here we first, upsample the feature maps
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}}$
using Pixel Shuffle to
$\mathrm{X}_{s_{\mathrm{B}\times\mathrm{\frac{C}{4}}\times\mathrm{2M}\times\mathrm{2N}}}$.

Then, we convert the feature maps $\mathrm{X}_{s}$ to the frequency domain
using DFT and obtain $\mathrm{X}_{\omega}$ similar to as shown in Eq. 12.

Next, we split the upsampled feature maps into low-frequencies
$\tilde{\hat{X}}_{\omega}$ by using a low-pass filter with the threshold being
the median frequency in the feature maps, and high-frequencies
$\tilde{X}^{*}_{\omega}$ by subtracting the low-frequencies from the entire
feature map in the frequency domain.

Then, we convert these split feature maps to the spatial domain to obtain
$\tilde{\hat{X}}_{s}$ (corresponds to low-frequencies) and
$\tilde{{X}^{*}}_{s}$ (corresponds to high-frequencies).

Lastly, we learn the parameter $\beta$ to learn and mix the low-frequency and
high-frequency information in the spatial domain as shown by Eq. 17, to get
the upsampling operation $\tilde{\zeta}_{up}$,

$\tilde{\zeta}_{up}(\mathrm{X}_{s_{\mathrm{B}\times\mathrm{C}\times\mathrm{M}\times\mathrm{N}}})=(1-\beta)\cdot\tilde{\hat{X}}_{\omega}+\beta\cdot\tilde{X}^{*}_{s}$
(17)

Inspired by [19], to have symmetry in the Fast Fourier Transform (FFT) along
the x-axis and y-axis, for every alternate downsampling and upsampling
operation, we use the transposed feature maps in the spatial domain, for the
FFT operation, and transpose them back to their original dimensions after
having finished all operations.

##  4 Experiments

###  4.1 Experimental Setup

Following we provide the experimental setup for this work. Please refer
Appendix C for additional details.

#### Downstream Task.

We focus on the image deblurring task. We make this choice as image deblurring
is a good example of a task in which preserving high-frequency information is
of paramount importance, especially to restore sharp edges and object
boundaries in an image, while attempting to reduce the spectral artifacts
infused in the image, due to incorrect sampling. Additionally, since image
deblurring is a pixel-wise task, it has an encoder and decoder in the
architecture, thus helping with the study on importance of symmetry for such
architectures.

#### Dataset.

Similar to [46, 7], for our experiments we use the GoPro image deblurring
dataset[32]. This dataset consists of 3214 real-world images, split into 2103
training images and 1111 test images, with realistic blur and their
corresponding ground truth (deblurred images) captured using a high-speed
camera.

#### Evaluation Metrics.

A higher Peak Signal-to-Noise Ration (PSNR) and a higher Structural similarity
(SSIM)[44] indicate a better quality image or an image closer to the ground
truth image.

#### Networks.

For our proposed architectural design changes, we use the SotA image
restoration architecture Restormer[46]. This is a Vision-Transformer based
UNet-like architecture. Additionally, we compare against FSNEt[9], a recently
proposed image restoration model that uses Fourier transforms in their
downsampling operations as well.

#### Adversarial Attacks.

We use PGD[27] and CosPGD[3] white-box attacks as they provide a good overview
of the quality of the features learned by the network[3]. As proposed by [27,
3], for both attacks we use $\epsilon\approx\frac{8}{255}$ and $\alpha$=0.01.
We evaluate against these adversarial attacks over the number of
iterations$\in${5, 10, 20}.

###  4.2 Results

Table 1: Comparison of performance of the proposed architectural changes
against CosPGD and PGD attacks with various attack strengths. Attack strength
increases with the number of attack iterations (itrs). Baseline Restormer uses
Pixel Unshuffle for dowsampling, and Pixel Shuffle for upsampling. Thus, we
first replace the Pixel Unshuffle with FLC Pooling (FLC), then we replace it
with our proposed Frequency Pooling (FP). Lastly, we replace the Pixel Shuffle
upsampling operation with our proposed FreqAvgUp to evaluate the full BOA-
Restormer. As shown by [47, 43] a small trade-off in the adversarial and non-
adversarial performance is expected .

Architecture | Test | CosPGD | PGD
---|---|---|---
|  | 5 attack itrs | 10 attack itrs | 20 attack itrs | 5 attack itrs | 10 attack itrs | 20 attack itrs
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
FSNet[10] | 33.27 | 0.9684 | 19.08 | 0.5184 | 16.51 | 0.3476 | 14.01 | 0.2550 | 19.10 | 0.5191 | 16.54 | 0.3485 | 14.08 | 0.2567
Restormer | 31.99 | 0.9635 | 11.36 | 0.3236 | 9.05 | 0.2242 | 7.59 | 0.1548 | 11.41 | 0.3256 | 9.04 | 0.2234 | 7.58 | 0.1543
\+ FLC | 23.85 | 0.7811 | 17.3 | 0.4725 | 16.01 | 0.4031 | 14.66 | 0.3401 | 17.3 | 0.4723 | 16.00 | 0.4030 | 14.66 | 0.3402
\+ FP | 29.95 | 0.9395 | 12.17 | 0.3024 | 10.35 | 0.2117 | 9.44 | 0.1651 | 12.17 | 0.3018 | 10.35 | 0.2112 | 9.45 | 0.1655
BOA-Restormer (ours) | 26.99 | 0.8806 | 23.9 | 0.7578 | 21.76 | 0.6875 | 21.00 | 0.6351 | 22.84 | 0.7290 | 18.91 | 0.6390 | 18.86 | 0.601

MODEL | NO ATTACK | 5 iterations | 10 iterations | 20 iterations
---|---|---|---|---
FSNet[9] |  |  |  |  |  |
Restormer |  |  |  |  |  |
Restormer |  \+ FLC |  |  |  |  |
Restormer |  \+ FP |  |  |  |  |
BOA-Restormer |  (+ FP |  \+ FreqAvgUp) |  |  |  |

Figure 4: Comparing the proposed architectural design choices qualitatively
against other baselines for image restoration, on normal blurry input images
and input images adversarially attacked using CosPGD. Symbolic notations are
same as those in Tab. 1.

In the following, we discuss the experimental results from using our proposed
BOA-Restormer, employing our downsampling method “FrequencyPreservedPooling”
and our upsampling method “FreqAvgUp” for image restoration. We report the
quantitative results in Tab. 1, and we provide qualitative results in Fig. 4.
We provide additional qualitative results with PGD attack in Fig. 7 in
Appendix B. We observe in Fig. 4, that while both Restormer and FSNet restore
blurry image reasonably well, under adversarial attack Restormer’s
restorations have significant visual artifacts. These visual artifacts, while
reduced, also exist in the images restored by FSNet and are accentuated under
strong adversarial attack (>10 attack iterations). We first ablate over
naïvely implementing FLC Pooling[17] in the Restormer encoder for downsampling
and we observe that while under attack, the spectral artifacts that exist in
baseline Restormer have reduced slightly, the restored images are far from
ideal and are significantly worse compared to FSNEt. Additionally, due to the
sinc-interpolation artifacts in FLC Pooling, we observe additional spectral
artifacts in a window in the center of the images restored when using FLC
Pooling for downsampling. These artifacts are called lattice artifacts by [24]
and occur due to the inherent nature of low-pass filters in the Fourier domain
as shown by [14]. Moreover, the images restored are still significantly
blurry. We hypothesize that this is due to the removal of high-frequency
information in the images, which hampers the image restoration model to
restore sharp edges and image boundaries. Thus, we replace the FLC Pooling in
the downsampling with FrequencyPreservedPooling (FP), and observe that the
spectral artifacts that existed in the restored blurry images have been
removed. However, the blurry images restored under adversarial attacks still
have significant spectral artifacts. Since the Restormer architecture,
Restormer with FLC Pooling for downsampling and Restormer with
FrequencyPreservedPooling for downsampling all use PixelShuffle for
upsampling, we hypothesize that the decoder is unable to interpret the
information encoded during downsampling correctly. Thus, we introduce symmetry
in the downsampling-upsampling operation by replacing the PixelShuffle in
“Restormer+FP” with FreqAvgUp for upsampling to obtain “BOA-Restormer”. This
symmetry helps the restoration model perform significantly better, and we
observe that the restored images are relatively artifacts free even under
strong adversarial attacks. We discuss the importance of this symmetry in
detail in Appendix A

From a quantitative perspective, in Tab. 1, we observe that the metrics
reflect the observations from Fig. 4. While FSNet performs significantly
better for blurry images without attack, under adversarial attacks, it lacks
stability in its restorations. A similar observation is made for the baseline
Restormer, which performs worse under adversarial attacks. Including FLC
Pooling for downsampling, does help increase the adversarial robustness of the
models. However the quality of the blurry images restored when not attacked is
severely low with previously discussed spectra artifacts visible. Replacing
FLC Pooling with FrequencyPreservedPooling, helps in increasing the
performance of the model on blurry images when not attacked. However, it is
less robust than using FLC Pooling. Lastly, we introduce symmetry in Restormer
with Frequency Pooling, by replacing the upsampling operation from
PixelShuffle to FreqAvgUp and observe an increase the adversarial robustness
with the resulting BOA-Restormer. We observe a slight trade-off in the
adversarial and non-adversarial performance, however this trade-off is
expected as shown by [47, 43].

###  4.3 Analysing Focus on Low-Frequencies

Table 2: Different downsampling techniques compared based on performance on
clean samples and performance against CosPGD and PGD attacks with various
attack strengths. Baseline Restormer uses Pixel Unshuffle for downsampling. As
discussed in Eq. 10 in Sec. 3.1, we ablate over dropping high frequencies
during downsampling with a 30% probability. We denote this with “+ Drop High”.
Additionally, we ablate over dropping high frequencies with a 30% probability
only in the first downsampling step, we denote this using “+ Drop High in
First Step”.

Architecture | Test | CosPGD | PGD
---|---|---|---
|  | 5 attack itrs | 10 attack itrs | 20 attack itrs | 5 attack itrs | 10 attack itrs | 20 attack itrs
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Restormer | 31.99 | 0.9635 | 11.36 | 0.3236 | 9.05 | 0.2242 | 7.59 | 0.1548 | 11.41 | 0.3256 | 9.04 | 0.2234 | 7.58 | 0.1543
\+ FLC | 23.85 | 0.7811 | 17.3 | 0.4725 | 16.01 | 0.4031 | 14.66 | 0.3401 | 17.3 | 0.4723 | 16.00 | 0.4030 | 14.66 | 0.3402
\+ FP | 29.95 | 0.9395 | 12.17 | 0.3024 | 10.35 | 0.2117 | 9.44 | 0.1651 | 12.17 | 0.3018 | 10.35 | 0.2112 | 9.45 | 0.1655
\+ FP + Drop High | 29.93 | 0.9395 | 12.67 | 0.3707 | 10.34 | 0.2875 | 9.25 | 0.2355 | 12.17 | 0.371 | 10.34 | 0.2878 | 9.24 | 0.235
\+ FP + Drop High in First Step | 29.96 | 0.9402 | 13.11 | 0.4458 | 11.00 | 0.3513 | 9.75 | 0.2899 | 13.10 | 0.4458 | 10.98 | 0.3502 | 9.76 | 0.2907

| NO ATTACK | 5 iterations
---|---|---
bbRestormer |  |
Restormer + FLC |  |
Restormer + FP |  |
Restormer + FP \+ Drop High |  |
Restormer + FP \+ Drop High First Step |  |

Figure 5: Different downsampling methods being compared qualitatively on
normal blurry input images and input images adversarially attacked using
CosPGD for five iterations. Symbolic notations are same as those in Tab. 2.

In the following, we ablate design decisions to encourage the focus on low-
frequency information, while providing both high and low frequencies to the
model. As discussed in Sec. 4.2, we observe that naïvely implementing FLC
Pooling for a pixel-wise task such as image deblurring does not result in
usable restored images. The better performance of “FrequencyPreservedPooling”
over FLC Pooling for image deblurring indicates that retaining high-frequency
information from the feature maps is crucial to the task. However, the extent
to which this high-frequency information needs to be retained needs further
exploration. Thus, we study “FrequencyPreservedPooling” in more detail. To
this effect, we modify “FrequencyPreservedPooling”, such that for 30% of the
function calls, we drop the high-frequency information, to effectively reduce
“FrequencyPreservedPooling” to Eq. 10 for these calls. We implement this
ablation in two possible settings, first: allowing all downsampling steps to
drop high frequencies in 30% of the calls. We denote this as “Drop High”.
Second, we allow only the first downsampling step to drop high frequencies 30%
of the time. We denote this as “Drop High in First Step”. The intuition behind
“Drop High in First Step” is that, since the image itself might have noise
that might correspond to high frequencies, reducing them to only the first
downsampling step might be enough. Further reducing high-frequencies at random
in all downsampling steps might deprive the model of some important
information as well. We observe in Tab. 2, that this intuition is validated
and indeed “Drop High in First Step” outperforms the other ablations over
“FrequencyPreservedPooling”. However, when viewed qualitatively in Fig. 5, we
observe a massive loss of information in the restored images, with
considerable areas simply being white. Thus, for the ablations with the
upsampling, we continue using “FrequencyPreservedPooling” without any explicit
loss in high frequencies and do not consider its ablated variants.

#### Proof of Focus on Low-Frequencies.

Since the proposed methods are learning a parameter for mixing the low-
frequency and high-frequency information from the feature maps, we can
investigate if the model is focusing on low-frequency information. We can do
so my observing the values of these parameters learned for each downsampling
and upsampling step. As shown in Eq. 8 and Eq. 16, a value of $\alpha$ and
$\beta$ less than 0.5 would mean the model is focusing more on low-frequency
information. In Tab. 3, we report these values learned for
“FrequencyPreservedPooling” and its variants. Similarly, in Tab. 4, we report
the values learned for “FrequencyPreservedPooling” along with the upsampling
operation variants “FreqAvgUp” and “SplitUp”. For both cases, we observe that
in all instances but one, the learned value for mixing the information is less
than 0.5 providing strong indication for the model focusing on low-
frequencies.

Table 3: Values learnt for the mixing of feature maps, $\alpha$ during
downsampling by different downsampling operations. Restormer has three
downsampling steps and thus three values of $\alpha$. The initial value of
$\alpha$ is 0.3, with the intention of inducing a bias towards low
frequencies. Here the upsampling operation used is PixelShuffle. Notations
“+Drop High” and “+Drop High in First Step” mean the same as in Tab. 2.

Model |  $\alpha$ 0 |  $\alpha$ 1 |  $\alpha$ 2
---|---|---|---
Restormer + FP | 0.0686 | 0.1703 | 0.0886
\+ FP + Drop High | 0.5239 | 0.2861 | 0.1382
\+ FP + Drop High in First Step | 0.1312 | 0.2815 | 0.1246

Table 4: Values learnt for the mixing of feature maps, $\alpha$ during
downsampling and $\beta$ during upsampling by different upsampling operations.
Restormer has three downsampling steps and three upsampling steps, thus three
values of each. The initial value of both $\alpha$ and $\beta$ is 0.3, with
the intention of inducing a bias towards low frequencies. Here the model used
is “BOA Restormer”.

Upsampling Method |  $\alpha$ 0 |  $\alpha$ 1 |  $\alpha$ 2 |  $\beta$ 0 |  $\beta$ 1 |  $\beta$ 2
---|---|---|---|---|---|---
Freq Avg Up | 0.4866 | 0.2805 | 0.2125 | 0.2517 | 0.2032 | 0.2246
Split Up | 0.4990 | 0.2915 | 0.2281 | 0.2656 | 0.3306 | 0.3035

Please note, we provide an analyze the importance of symmetry in Appendix A in
the Appendix.

##  5 Conclusion

With the increased use of machine learning in computer vision, foundational
principles of signal processing are often overlooked. These foundations help
guide computer vision tasks and can be utilized by machine learning methods to
ensure that the model is learning meaningful representations of the data.
Fortunately, for pixel-wise tasks like image deblurring, this can be validated
by the human eye by spotting visible spectral artifacts in the images
restored. For some scenarios, one might have to use adversarial attacks as a
tool to accentuate these spectral artifacts to be visible to the human eye.

In this work, we focus on the downsampling and upsampling operations being
used by some SotA architectures for image deblurring and propose more stable
solutions. We propose a novel downsampling method “FrequencyPreservedPooling”
and the upsampling method “FreqAvgUp” which are theoretically motivated
sampling operations that lead to more stable feature representation being
learned by the model without the need for adversarial training. These
operations preserve important parts of the signal while discarding noise and
significantly reducing all spectral artifacts in the restored images. Our
final model is termed _BOA-Restormer_.

#### Limitations and Future Work.

While, this work shows that focusing on low-frequency information, while still
retaining some high-frequency information, helps the image deblurring model to
restore images with significantly reduced spectral artifacts, even under
adversarial attack. Yet, the performance on blurry images when not
adversarially attacked is not ideal. Here, a point needs to be made for the
currently used evaluation metrics that do not provide a reliable evaluation of
image restoration methods, especially as they ignore spectral artifacts that
are visible to the human eye. This can be observed in Fig. 1, here FLC Pooling
visibly generate significantly more artifacts than FSNet, however both are
very similar PSNR values. However, these metrics do serve as a community-
adopted benchmark and provide some information regarding the quality of the
restorations. Thus, improving our proposed methods, to perform better on these
metrics like PSNR would be additionally helpful to the community.

The authors acknowledge support by the state of Baden-Württemberg through
bwHPC and the German Research Foundation (DFG) through grant INST 35/1597-1
FUGG. The authors additionally acknowledge funding by the DFG Research Unit
5336 - Learning to Sense.

## References

  * Agnihotri et al. [2023a] S. Agnihotri, K. V. Gandikota, J. Grabinski, P. Chandramouli, and M. Keuper.  On the unreasonable vulnerability of transformers for image restoration-and an easy fix.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3707–3717, 2023a.
  * Agnihotri et al. [2023b] S. Agnihotri, J. Grabinski, and M. Keuper.  Improving stability during upsampling – on the importance of spatial context, 2023b.
  * Agnihotri et al. [2024] S. Agnihotri, S. Jung, and M. Keuper.  CosPGD: a unified white-box adversarial attack for pixel-wise prediction tasks.  In _Proceedings of the 41th International Conference on Machine Learning, ICML 2024_ , 2024.  URL https://arxiv.org/abs/2302.02213.
  * Badrinarayanan et al. [2017] V. Badrinarayanan, A. Kendall, and R. Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  _TPAMI_ , 39(12):2481–2495, 2017.
  * Carlini and Wagner [2017] N. Carlini and D. Wagner.  Towards evaluating the robustness of neural networks.  In _2017 ieee symposium on security and privacy (sp)_ , pages 39–57. IEEE, 2017.
  * Chang and Chen [2018] J.-R. Chang and Y.-S. Chen.  Pyramid stereo matching network.  In _CVPR_ , pages 5410–5418, 2018.
  * Chen et al. [2022] L. Chen, X. Chu, X. Zhang, and J. Sun.  Simple baselines for image restoration.  In _ECCV_ , pages 17–33. Springer, 2022.
  * Chi et al. [2020] L. Chi, B. Jiang, and Y. Mu.  Fast fourier convolution.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _NeurIPS_ , volume 33, pages 4479–4488, 2020.
  * Cui et al. [2023] Y. Cui, Y. Tao, Z. Bing, W. Ren, X. Gao, X. Cao, K. Huang, and A. Knoll.  Selective frequency network for image restoration.  In _ICLR_ , 2023.
  * Cui et al. [2024] Y. Cui, W. Ren, X. Cao, and A. Knoll.  Image restoration via frequency selection.  _TPAMI_ , 46(2):1093–1108, 2024.
  * Dong et al. [2023] J. Dong, J. Pan, Z. Yang, and J. Tang.  Multi-scale residual low-pass filter network for image deblurring.  In _ICCV_ , pages 12345–12354, October 2023.
  * Durall et al. [2020] R. Durall, M. Keuper, and J. Keuper.  Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions.  In _CVPR_ , pages 7890–7899, 2020.
  * Geirhos et al. [2020] R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann.  Shortcut learning in deep neural networks.  _Nature Machine Intelligence_ , 2(11):665–673, nov 2020.
  * Gibbs [1898] J. W. Gibbs.  Fourier’s series.  _Nature_ , 59(200), December 1898.
  * Goodfellow et al. [2020] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Goodfellow et al. [2014] I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  2014\.
  * Grabinski et al. [2022a] J. Grabinski, S. Jung, J. Keuper, and M. Keuper.  Frequencylowcut pooling-plug and play against catastrophic overfitting.  In _European Conference on Computer Vision_ , pages 36–57. Springer, 2022a.
  * Grabinski et al. [2022b] J. Grabinski, J. Keuper, and M. Keuper.  Aliasing and adversarial robust generalization of cnns.  _Machine Learning_ , pages 1–27, 2022b.
  * Grabinski et al. [2023a] J. Grabinski, J. Keuper, and M. Keuper.  Fix your downsampling asap! be natively more robust via aliasing and spectral artifact free pooling, 2023a.
  * Grabinski et al. [2023b] J. Grabinski, J. Keuper, and M. Keuper.  As large as it gets: Learning infinitely large filters via neural implicit functions in the fourier domain.  2023b.
  * Gu et al. [2022] J. Gu, H. Zhao, V. Tresp, and P. H. Torr.  Segpgd: An effective and efficient adversarial attack for evaluating and boosting segmentation robustness.  In _ECCV_ , pages 308–325. Springer, 2022.
  * Hoffmann et al. [2021] J. Hoffmann, S. Agnihotri, T. Saikia, and T. Brox.  Towards improving robustness of compressed cnns.  In _ICML Workshop on Uncertainty and Robustness in Deep Learning (UDL)_ , 2021.
  * Hossain et al. [2023] M. T. Hossain, S. W. Teng, G. Lu, M. A. Rahman, and F. Sohel.  Anti-aliasing deep image classifiers using novel depth adaptive blurring and activation function.  _Neurocomputing_ , 536:164–174, 2023.
  * Hovden et al. [2015] R. Hovden, Y. Jiang, H. L. Xin, and L. F. Kourkoutis.  Periodic artifact reduction in fourier transforms of full field atomic resolution images.  _Microscopy and Microanalysis_ , 21(2):436–441, 2015.  10.1017/S1431927614014639.
  * Jung and Keuper [2021] S. Jung and M. Keuper.  Spectral distribution aware image generation.  In _AAAI_ , volume 35, pages 1734–1742, 2021.
  * Karras et al. [2021] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila.  Alias-free generative adversarial networks.  _NeurIPS_ , 34:852–863, 2021.
  * Kurakin et al. [2017] A. Kurakin, I. J. Goodfellow, and S. Bengio.  Adversarial machine learning at scale.  In _ICLR_ , 2017.
  * Li et al. [2021] Q. Li, L. Shen, S. Guo, and Z. Lai.  Wavecnet: Wavelet integrated cnns to suppress aliasing effect for noise-robust image classification.  _IEEE Transactions on Image Processing_ , 30:7074–7089, 2021.
  * Long et al. [2015] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In _CVPR_ , pages 3431–3440, 2015.
  * Mao et al. [2023] X. Mao, Y. Liu, F. Liu, Q. Li, W. Shen, and Y. Wang.  Intriguing findings of frequency selection for image deblurring.  In _AAAI_ , 2023.
  * Mosleh et al. [2014] A. Mosleh, J. M. P. Langlois, and P. Green.  Image deconvolution ringing artifact detection and removal via psf frequency analysis.  In D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, editors, _ECCV_ , pages 247–262, Cham, 2014. Springer International Publishing.
  * Nah et al. [2017] S. Nah, T. H. Kim, and K. M. Lee.  Deep multi-scale convolutional neural network for dynamic scene deblurring.  In _CVPR_ , July 2017.
  * Noh et al. [2015] H. Noh, S. Hong, and B. Han.  Learning deconvolution network for semantic segmentation.  In _ICCV_ , pages 1520–1528, 2015.
  * Odena et al. [2016] A. Odena, V. Dumoulin, and C. Olah.  Deconvolution and checkerboard artifacts.  _Distill_ , 1(10):e3, 2016.
  * Radford et al. [2015] A. Radford, L. Metz, and S. Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_ , 2015.
  * Rao et al. [2021] Y. Rao, W. Zhao, Z. Zhu, J. Lu, and J. Zhou.  Global filter networks for image classification.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, _NeurIPS_ , 2021.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , pages 234–241. Springer, 2015.
  * Schmalfuss et al. [2022a] J. Schmalfuss, L. Mehl, and A. Bruhn.  Attacking motion estimation with adversarial snow.  2022a.
  * Schmalfuss et al. [2022b] J. Schmalfuss, P. Scholze, and A. Bruhn.  A perturbation-constrained adversarial attack for evaluating the robustness of optical flow.  In _ECCV_ , pages 183–200. Springer, 2022b.
  * Schrodi et al. [2022] S. Schrodi, T. Saikia, and T. Brox.  Towards understanding adversarial robustness of optical flow networks.  In _CVPR_ , pages 8916–8924, 2022.
  * Shi et al. [2016] W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network.  In _CVPR_ , June 2016.
  * Sommerhoff et al. [2023] H. Sommerhoff, S. Agnihotri, M. Saleh, M. Moeller, M. Keuper, and A. Kolb.  Differentiable sensor layouts for end-to-end learning of task-specific camera parameters.  _arXiv preprint arXiv:2304.14736_ , 2023.
  * Tsipras et al. [2019] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry.  Robustness may be at odds with accuracy.  In _ICLR_ , 2019.
  * Wang et al. [2004] Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.
  * Yang and Fu [2019] H.-H. Yang and Y. Fu.  Wavelet u-net and the chromatic adaptation transform for single image dehazing.  In _ICIP_ , pages 2736–2740, 2019.
  * Zamir et al. [2022] S. W. Zamir, A. Arora, S. Khan, M. Hayat, F. S. Khan, and M.-H. Yang.  Restormer: Efficient transformer for high-resolution image restoration.  In _CVPR_ , 2022.
  * Zhang et al. [2019] H. Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. Jordan.  Theoretically principled trade-off between robustness and accuracy.  In _ICML_ , 2019.
  * Zhang [2019] R. Zhang.  Making convolutional networks shift-invariant again.  In _ICML_ , 2019.
  * Zhao et al. [2017] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia.  Pyramid scene parsing network.  In _CVPR_ , pages 2881–2890, 2017.
  * Zhao et al. [2018] H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia.  PSANet: Point-wise spatial attention network for scene parsing.  In _ECCV_ , 2018.
  * Zou et al. [2020] X. Zou, F. Xiao, Z. Yu, and Y. J. Lee.  Delving deeper into anti-aliasing in convnets.  In _BMVC_ , 2020.

Beware of Aliases – Signal Preservation is Crucial for Robust Image
Restoration
Paper #2259 Supplementary Material

##  Appendix A Further Analysis on Importance of Symmetry

Table 5: Different Upsampling techniques used in the decoder compared against
clean performance and against CosPGD and PGD attacks with various attack
strengths. Attack strength increases with the number of attack iterations
(itrs). The baseline Restormer uses Pixel Shuffle for upsampling, and Pixel
Unshuffle for downsampling. First we replace the downsampling operation with
FrequencyPooling (FP), while still using Pixel Shuffle for upsampling. Next,
we ablate over the choice of the upsampling operation by replacing
PixelShuffle with SplitUp and FreqAvgUp.

Architecture | Test | CosPGD | PGD
---|---|---|---
|  | 5 attack itrs | 10 attack itrs | 20 attack itrs | 5 attack itrs | 10 attack itrs | 20 attack itrs
PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
Restormer | 31.99 | 0.9635 | 11.36 | 0.3236 | 9.05 | 0.2242 | 7.59 | 0.1548 | 11.41 | 0.3256 | 9.04 | 0.2234 | 7.58 | 0.1543
Restormer + FP | 29.95 | 0.9395 | 12.17 | 0.3024 | 10.35 | 0.2117 | 9.44 | 0.1651 | 12.17 | 0.3018 | 10.35 | 0.2112 | 9.45 | 0.1655
\+ FP + SplitUp upsampling | 6.26 | 0.2370 | 6.31 | 0.1971 | 6.06 | 0.1854 | 5.79 | 0.1702 | 6.29 | 0.1964 | 6.04 | 0.1847 | 5.78 | 0.1697
\+ FP + FreqAvgUp upsampling | 26.99 | 0.8806 | 23.9 | 0.7578 | 21.76 | 0.6875 | 21.00 | 0.6351 | 22.84 | 0.7290 | 18.91 | 0.6390 | 18.86 | 0.601

MODEL | NO ATTACK | 5 iterations | 10 iterations | 20 iterations
---|---|---|---|---
Restormer |  |  |  |  |  |
Restormer |  \+ FP |  |  |  |  |
Restormer |  \+ FP |  \+ SplitUp |  |  |  |
Restormer |  \+ FP |  \+ FreqAvgUp |  |  |  |

Figure 6: Different upsampling methods being compared qualitatively on normal
blurry input images and input images adversarially attacked using CosPGD.
Symbolic notations are same as those in Tab. 5.

Following, we ablate over the symmetry introduced in the architecture, by
making the downsampling and upsampling operations mirror images to each other.
One might hypothesize, given the downsampling operation “Frequency Pooling”
treats the low and high-frequency information differently, merely treating
them differently again during upsampling might be enough, and performing an
entirely symmetric architecture is not essential. Thus, we device “SplitUp” as
an upsampling operation, that merely splits the feature maps into low and high
frequencies and learns to combine them using a learnable parameter $\beta$.
However, we observe in Fig. 6, that this leads to improper sampling, causing
severe artifacts in the restored images, but when under adversarial attack and
not under adversarial attacks. This observation is reflected in the
quantitative results reported in Tab. 5, the performance of “SplitUp” is
inadequate.

##  Appendix B Additional Qualitative Results

Following we provide qualitative results for the models considered in Fig. 4,
Fig. 6 and Fig. 5 under PGD attack and more attack iterations for Tab. 2.

MODEL | NO ATTACK | 5 iterations | 10 iterations | 20 iterations
---|---|---|---|---
FSNet[9] |  |  |  |  |  |
Restormer |  |  |  |  |  |
Restormer |  \+ FLC |  |  |  |  |
Restormer |  \+ FP |  |  |  |  |
Restormer |  \+ FP |  \+ FreqAvgUp |  |  |  |

Figure 7: Comparing the proposed architectural design choices qualitatively
against other baselines for image restoration, on normal blurry input images
and input images adversarially attacked using PGD. Symbolic notations are same
as those in Tab. 1.

MODEL | NO ATTACK | 5 iterations | 10 iterations | 20 iterations
---|---|---|---|---
Restormer |  |  |  |  |  |
Restormer |  \+ FP |  |  |  |  |
Restormer |  \+ FP |  \+ SplitUp |  |  |  |
Restormer |  \+ FP |  \+ FreqAvgUp |  |  |  |

Figure 8: Different upsampling methods being compared qualitatively on normal
blurry input images and input images adversarially attacked using PGD.
Symbolic notations are same as those in Tab. 5.

MODEL | NO ATTACK | 5 iterations | 10 iterations | 20 iterations
---|---|---|---|---
Restormer |  |  |  |  |  |
Restormer |  \+ FLC |  |  |  |  |
Restormer |  \+ FP |  |  |  |  |
Restormer |  \+ FP |  \+ Drop High |  |  |  |
Restormer |  \+ FP |  \+ Drop High First Step |  |  |  |

Figure 9: Different downsampling methods being compared qualitatively on
normal blurry input images and input images adversarially attacked using PGD.
Symbolic notations are same as those in Tab. 2.

MODEL | NO ATTACK | 5 iterations | 10 iterations | 20 iterations
---|---|---|---|---
Restormer |  |  |  |  |  |
Restormer |  \+ FLC |  |  |  |  |
Restormer |  \+ FP |  |  |  |  |
Restormer |  \+ FP |  \+ Drop High |  |  |  |
Restormer |  \+ FP |  \+ Drop High First Step |  |  |  |

Figure 10: Different downsampling methods being compared qualitatively on
normal blurry input images and input images adversarially attacked using
CosPGD. Symbolic notations are same as those in Tab. 2.

##  Appendix C Additional Experimental Setup Details

#### Evaluation Metrics.

We report the Peak Signal-to-Noise Ratio (PSNR) and Structural similarity
(SSIM)[44] scores of the deblurred images w.r.t. to the ground truth images,
averaged over all images in the test set. A higher PSNR indicates a better
quality image or an image closer to the ground truth image. A higher SSIM
score corresponds to better higher similarity between the two compared images.

#### Training Regime.

To train our proposed architectures we follow the same training regime and
hyperparameters as those used by [46] for training their proposed Restormer.

#### Compute Resources.

For training and evaluation, we used a single NVIDIA A100 GPU with 48GB VRAM
for each run except runs using FSNet[10]. Due to the excessive memory
requirements of FSNet, for these experiments we used single NVIDIA A100 GPU
with 80GB VRAM.